An impartial framework to investigate demosaicking input embedding options

Abstract

Convolutional Neural Networks (CNNs) have proven highly effective for demosaicking, transforming raw Color Filter Array (CFA) sensor samples into standard RGB images. Directly applying convolution to the CFA tensor can lead to misinterpretation of the color context, so existing demosaicking networks typically embed the CFA tensor into the Euclidean space before convolution. The most prevalent embedding options are Reordering and Pre-interpolation. However, it remains unclear which option is more advantageous for demosaicking. Moreover, no existing demosaicking network is suitable for conducting a fair comparison. As a result, in practice, the selection of these two embedding options is often based on intuition and heuristic approaches. This paper addresses the non-comparability between the two options and investigates whether pre-interpolation contributes additional knowledge to the demosaicking network. Based on rigorous mathematical derivation, we design pairs of end-to-end fully convolutional evaluation networks, ensuring that the performance difference between each pair of networks can be solely attributed to their differing CFA embedding strategies. Under strictly fair comparison conditions, we measure the performance contrast between the two embedding options across various scenarios. Our comprehensive evaluation reveals that the prior knowledge introduced by pre-interpolation benefits lightweight models. Additionally, pre-interpolation enhances the robustness to imaging artifacts for larger models. Our findings offer practical guidelines for designing imaging software or Image Signal Processors (ISPs) for RGB cameras.