Achieving precise robotic assembly is paramount for safety and performance in industrial settings. Conventional assembly methods require tedious manual adjustment of many parameters, and it is challenging to meet the assembly requirements with tight clearance. Robot learning has been at the forefront of research, showing potential for automation and intelligence in robotic operations. However, intricate information such as force feedback is indispensable for high-precision robot assembly tasks, and current learning methods grapple with processing this complex observation data due to discontinuous force changes during operation and the inherent noise from the force sensor. To enhance learning efficiency amidst challenging observations, this paper proposes introducing the concept of latent causal factors that drive sensor observations and hold paramount significance in assembly tasks. This paper analyses the impact of discontinuous and noisy observations and offers two ways to infer causal factors. Based on the implied latent factors, we propose a method that learns high-precision assembly policies from interacting with the environment. The algorithm’s performance is evaluated in simulated and real-world nut insertion environments, demonstrating significant improvements over the previous methods. This research also underscores the promise of causal inference in addressing industrial challenges.