Long wavelength interdomain phonons and instability of dislocations in small-angle twisted bilayers

Abstract

We develop a theory for long wavelength phonons originating at dislocations separating domains in small-angle twisted homobilayers of 2D materials such as graphene and MX$_2$ transition metal dichalcogenides (M=Mo,W; X=S,Se). We find that both partial and perfect dislocations, forming due to lattice relaxation in the twisted bilayers with parallel and anti-parallel alignment of unit cells of the constituent layers, respectively, support several one-dimensional subbands of the {\it interdomain} phonons. We show that spectrum of the lowest gapless subband is characterized by imaginary frequencies, for wave-numbers below a critical value, dependent on the dislocation orientation, which indicates an instability for long enough straight partial and perfect dislocations. The other subbands are gapped, with subband bottoms lying below the frequency of interlayer shear mode in domains, which facilitates their detection with the help of optical and magnetotransport techniques.