In-memory computing has carried out calculations in situ within each memory unit and its main power consumption comes from data writing and erasing. Further improvements in the energy efficiency of in-memory computing require memory devices with sub-femto-Joule energy consumption. Floating gate memory devices based on two-dimensional (2D) material heterostructures have outstanding characteristics such as non-volatility, multi-bit storage, and low operation energy, suitable for application in in-memory computing chips. Here, we report a floating gate memory device based on a WSe2/h-BN/Multilayer-graphene/h-BN heterostructure, the energy consumption of which is in sub-femto Joule (0.6 fJ) per operation for program/erase, and the read power consumption is in the tens of femto Watt (60 fW) range. We show a Hopfield neural network composed of WSe2/h-BN/Multilayer-graphene/h-BN heterostructure floating gate memory devices, which can recall the original patterns from incorrect patterns. These results shed light on the development of future compact and energy-efficient hardware for in-memory computing systems.