Reversible data hiding in encrypted domain (RDH-ED) can perform encryption and data embedding to simultaneously fulfill the privacy protection and access control. The key distribution in current RDH-ED primarily follows a symmetric mechanism, resulting in limitations in key management and distribution. Therefore, public-key embedding (PKE) mechanism in RDH-ED is proposed to address the limitations, where embedding permission is open to the public while extracting is under control. Then a two-stage RDH-ED scheme with PKE mechanism is designed based on learning with errors (LWE) for images. The algorithm of the first stage is redundancy recoding in LWE encrypted domain (RR-LWE) for the ciphertext encrypted from any a pixel bit. Public embedding key is specially constructed. bits data could be embedded per ciphertext. The algorithm of the second stage is difference expansion in LWE encrypted domain (DE-LWE) for the ciphertext of the entire image following RR-LWE. It transfers the bit operations of DE from spatial domain into LWE encrypted domain. We theoretically deduce the necessary conditions for embedding correctness and security. Experimental results demonstrate the outperformed effects in security and efficiency of the proposed algorithms. RR-LWE achieves an embedding capacity up to 24 bits per pixel (bpp) and DE-LWE further enhances that by approximately 0.5 bpp.