This study investigates the synthesis and characterization of a thioglycolic acid (TGA) functionalized magnetic graphene oxide nanocomposite (TGA/Fe₃O₄/GO) with enhanced nonlinear optical (NLO) properties for advanced optical applications. Comprehensive characterization confirmed the successful preparation of graphene oxide (GO) nanosheets, the incorporation of Fe₃O₄ nanoparticles onto the GO surface, and the effective attachment of a TGA ligand. Field-emission scanning electron microscopy (FESEM) images confirmed the formation of well-defined nanostructures, At the same time, X-ray diffraction (XRD) patterns verified the presence of the face-centered cubic (fcc) spinel phase of Fe₃O₄, with an average nanoparticle crystal size of approximately 10 nm. Raman spectroscopy indicated an increased ID/IG ratio (in both Fe₃O₄/GO and TGA/Fe₃O₄/GO) compared to GO affirming increased defect density induced by Fe₃O₄ decoration and subsequent TGA functionalization. The measured surface area and the saturation magnetization values of TGA/Fe₃O₄/GO nanocomposite were 57.24 m²/g and 11.90 emu/g, respectively. Furthermore, NLO characterization of the TGA/Fe₃O₄/GO nanocomposite using the Z-scan technique under continuous-wave (CW) 532 nm excitation reveals strong third-order nonlinear behavior. The open-aperture (OA) traces exhibit pronounced two photon absorption (TPA), reflected in nonlinear absorption (NLA) coefficients, (:beta:) on the order of (:{10}^{-3}:cm/W). Closed-aperture (CA) measurements show a clear peak–valley signature that increases with both power and concentration, confirming a positive nonlinear refractive (NLR) response and the occurrence of self-focusing, with (:{n}_{2}) values in the (:{10}^{-7}{cm}^{2}/W) range. The corresponding third-order susceptibilities (:left|{chi:}^{left(3right)}right|) in the range of (:{10}^{-6}left(esuright))indicate a highly responsive nonlinear system. The figures of merit ((:W>:1:)and (:T<:1)) further verify that the nanocomposite meets the criteria required for effective CW optical limiting and switching.