Effective thermal management in compact heat exchangers is crucial for enhancing energy efficiency and operational reliability. While vortex generators and hybrid nanofluids are recognized as effective passive enhancement techniques, their combined application remains underexplored. This study proposes a new deformation-based vortex generator with adjustable curvature governed by a dimensionless parameter (N) and evaluates its effect on the hydrothermal and exergy performance of a double-pipe heat exchanger. The k–ε turbulence model was employed for numerical simulations at Reynolds numbers ranging from 12,000 to 48,000. The nanofluid flow was considered as a two-phase mixture, and the mixture model was adopted for its simulation. To address the pressure–velocity coupling problem, the SIMPLC algorithm was utilized. Four vortex generator curvatures (N = 0, 1.5, 2.5, and 3.5) and three nanoparticle volume fractions (0%, 1.5%, and 3.5%) of a hybrid nanofluid containing single-walled carbon nanotubes and copper oxide nanoparticles in Syltherm 800 were tested under turbulent conditions. The hybrid nanofluid in the inner tube was modeled using the two-phase mixture approach, while hot water in the outer tube was modeled as a single-phase fluid. Results show that greater vortex generator curvature, nanoparticle concentration, and Reynolds number enhance the Nusselt number, achieving up to 71.25% improvement over the base fluid without a vortex generator. For the Syltherm 800 base fluid with a VG of geometric case N = 3.5, (Nu) increases by 69.91% compared to the case without a VG. Although these modifications increase pressure drop, the performance evaluation criterion remains above unity, indicating an optimal trade-off between heat transfer and hydraulic loss. Exergy efficiency also improves with higher curvature and nanoparticle content.