The immune system is a key component of malignant cell control and it is also involved in the elimination of pathogens that threaten the host. Despite our body is permanently exposed to a myriad of pathogens, the interference of such infections with the immune responses against cancer has been poorly investigated. Through a mathematical model, we show that the frequency, the duration and the action (positive or negative) of immune challenges may significantly impact tumor proliferation. First, we observe that a long immunosuppressive challenge increases accumulation of cancerous cells only if it occurs 14 years after the beginning of immunosenescence. However, short immune challenges result in an even greater accumulation of cancerous cells for the same total duration of immunosuppression. Finally, we show that short challenges of immune activation could lead to a slightly decrease in cancerous cell accumulation compared to a long one. Our results predict that frequent and acute immune challenges could have a different and in some extent higher impact on cancer risk than persistent ones even they have been much less studied in cancer epidemiology. These results are discussed regarding the existing empirical evidences and we suggest potential novel indirect role of infectious diseases on cancer incidence which should be investigated to improve prevention strategies against cancer.