Dimensionality crossover to a two-dimensional vestigial nematic state from a three-dimensional antiferromagnet in a honeycomb van der Waals magnet

The effects of fluctuations and disorder, which are substantially enhanced in reduced dimensionalities, can play a crucial role in producing non-trivial phases of matter such as vestigial orders characterized by a composite order parameter. However, fluctuation-driven magnetic phases in low dimensions have remained relatively unexplored. Here we demonstrate a phase transition from the zigzag antiferromagnetic order in the three-dimensional bulk to a Z3 vestigial Potts nematicity in two-dimensional few-layer samples of van der Waals magnet NiPS3. Our spin relaxometry and optical spectroscopy measurements reveal that the spin fluctuations are enhanced over the gigahertz to terahertz range as the layer number of NiPS3 reduces. Monte Carlo simulations corroborate the experimental finding of threefold rotational symmetry breaking but show that the translational symmetry is restored in thin layers of NiPS3. Therefore, our results show that strong quantum fluctuations can stabilize an unconventional magnetic phase after destroying a more conventional one. Magnetic phases that are stabilized by quantum fluctuations in low dimensions are rare. A thickness-dependent crossover from three-dimensional antiferromagnetism to a two-dimensional vestigial nematic state that is driven by fluctuations has now been observed.