The linear stability analysis of a viscoelastic Navier-Stokes-Voigt fluid flow, or the Kelvin-Voigt fluid of zero order in a Brinkman porous medium, is investigated using both modal and non-modal analysis. The numerical solution is obtained using the Chebyshev collocation method. The combined effects of the medium's porosity, represented by the porous parameter, the fluid viscosity, represented by the ratio of effective viscosity to fluid viscosity, and the fluid elasticity, represented by the Kelvin-Voigt parameter are investigated using both modal and non-modal analysis. The modal analysis describes the long-term behavior of the system, obtained through plotting the eigenspectrum, eigenfunctions, growth rate curves, neutral stability curves, and streamline plots, along with accurate values of critical triplets. In non-modal analysis, the pseudospectrum of the Orr-Sommerfeld operator, transient energy growth curves, and regions of stability, instability, and potential instability are depicted. The results obtained from modal analysis indicate that the porous parameter, Kelvin-Voigt parameter, and the ratio of effective viscosity to fluid viscosity act as stabilizing agents. However, using non-modal analysis, it is observed that while the porous parameter and the ratio of effective viscosity to fluid viscosity act as stabilizing agents, the Kelvin-Voigt parameter acts as a destabilizing agent over shorter periods.