Battery energy storage systems (BESSs) have recently been utilized in power systems for various purposes. Integrating these devices into power systems can enhance the damping capability of subsynchronous oscillations. The interaction between the control modes of the BESS and synchronous machines, as well as the control parameters of the BESS, reduces subsynchronous oscillations. To damp oscillations and improve dynamic stability, this work develops a linear model of a power system integrated with a BESS to investigate small-signal stability. The gain tuning of the BESS controller is formulated as an optimization problem and is solved using a fuzzy-based algorithm. The efficacy of the proposed method is evaluated under various operating conditions. Furthermore, the proposed method is compared with a power system stabilizer (PSS) damping controller, and the results demonstrate the superiority of the BESS damping method in mitigating subsynchronous oscillations and enhancing the dynamic stability of power systems. Lastly, eigenvalue analysis is employed to determine the permissible ranges of BESS parameters for stable power system operation.