CPFormer: End-to-End Multi-Person Human Pose Estimation From Raw Radar Cubes With Transformers

Abstract

It is challenging to reconstruct human pose in multi-person scenes using a single commercial millimeter-wave (mmWave) radar due to its limited resolution and susceptibility to noise. On the other hand, the signal processing process may cause the loss of detailed features of the raw radar signals or introduce errors, making it difficult to perform detailed analysis of radar signals in multi-person scenes. To address these issues, a cube pose transformer (CPFormer) is proposed, an end-to-end method for multi-person pose estimation from raw radar cubes. Specifically, the CPFormer consists of a learnable 3-D-discrete Fourier transform (DFT) module and Transformer-based networks. The learnable 3-D-DFT module extracts features from raw radar cubes and adaptively learns the time-to-frequency domain transformation for each dimension, replacing the traditional DFT. The Transformer includes the dual-stream hierarchical encoder (DHE) and the multi-person pose decoder (MPD). First, the proposed spatiotemporal fusion tokenizer (SFT) captures the spatiotemporal cues of adjacent frames and represents the radar cubes as token embeddings. Then, the DHE uses window attention and global cross-view attention (GCVA) to learn the local, global, and cross-view dependencies from the radar cubes of the horizontal and vertical views, to extract fine-grained sensing cues. The MPD directly predicts multi-person poses based on features extracted by the encoder, without separating signals of different targets in the low-resolution radar data. Evaluated on a multi-person human pose dataset collected with a TI AWR1843 Boost mmWave radar in two different environments, the CPFormer achieves the lowest pose estimation error.