In the aftermath of an earthquake, damage detection and performance evaluation of structural components are imperative for assessing the residual seismic capacity of a building. In this study, an integrated image processing and deep learning approach was developed to evaluate the degradation in strength and stiffness (i.e., strength reduction and stiffness reduction) of reinforced concrete (RC) shear walls. The approach comprised two main tasks: detecting and localizing visible seismic damage from photographs and evaluating strength and stiffness degradation based on this information. The semantic segmentation network, Damage-Net, was used for damage detection and localization. A novel crack morphological processing layer and a patch feature extraction layer were developed for damage feature extraction and compression. A lightweight deep convolutional neural network named DegradeEval-Net_v2, featuring the upgraded dilated and separable convolution block and multi-layer perception, was developed to link the damage feature with strength and stiffness degradation. A database comprising test data and photographs of 14 RC shear wall specimens with a flexural-dominated behavior mode and high to intermediate ductility was constructed to train and test the DegradeEval-Net_v2 network. The results indicate that DegradeEval-Net_v2 substantially improved the performance assessment accuracy of damaged RC shear walls, with a 35% smaller root mean square error (RMSE) for stiffness degradation evaluation and 75% smaller RMSE for strength degradation evaluation, compared with the provisions specified in JBDPA and FEMA guidelines. Moreover, evaluation results on test sets demonstrate that introducing the damage feature extraction and compression layers effectively preserved local crack information and improved the accuracy with which stiffness reduction was evaluated. In addition, DegradeEval-Net_v2 outperformed ResNet18 and MobileNet V3 in terms of balanced efficiency and accuracy. Interpretability analysis demonstrates that the model learned the distinct contribution patterns of various visible damage indexes to stiffness and strength degradation across different loading levels.