Based on the modeling of optical edge selection, studies were conducted to optimize the parameters of spatial filters. The parameters of the ring and vortex spatial filters were optimized for optical edge selection in different images using the Fourier-correlator. The result of the research is the development of an algorithm that helps to find the optimal parameters of ring and vortex spatial filters for a given image. The optimality criterion is determined using Canny’s digital algorithm, therefore, the developed algorithm is a hybrid algorithm. The developed algorithm for finding optimal filter parameters is based on minimizing the standard deviation from the samples formed by the Canny operator. The purpose of the developed algorithm consists of adjusting the Spatial light modulator to the input image in a real experiment. The fact that the vortex filter with optimal parameters is slightly inferior to the Canny’s operator was established. It has been shown that the vortex filter makes it possible to selectively highlight object details.