Individuals with cancer are recommended to engage in regular physical activity (PA) even during cancer therapy. The aim of this study was to explore how patient-reported physician PA counseling influences their PA intention and behavior in addition to psycho-cognitive determinants derived from the theory of planned behavior (TPB). A longitudinal study during cancer treatment was conducted among N = 115 patients with breast, prostate, or colorectal cancer (Mage = 58.0, SD = 11.5; 55.7% female). The median time since diagnosis was 2 months, and 19.1% were diagnosed with metastases. Participants provided information on PA counseling by their physicians and on psycho-cognitive variables of the TPB at three measurement points. Additionally, they wore accelerometers for seven days at baseline and three months later. Nearly half of participants (48%) reported basic PA counseling and 30% reported in-depth PA counseling. Patients in poorer health and with lower education reported significantly less in-depth counseling. In addition to patient self-efficacy in performing PA, only in-depth physician PA counseling, but not basic physician counseling, predicted intention for PA four weeks later. Patients' PA three months after baseline was predicted by patients' PA at baseline and their intention for PA. Overall, the PA level at baseline was identified as the most important predictor of PA three months later. Nevertheless, physicians seem to have the ability to increase their cancer patients' intention for PA by in-depth counseling.