Glycolipids constitute a major part of the cell envelope of Mycobacterium tuberculosis (Mtb). They are potent immunomodulatory molecules recognized by several immune receptors like pattern recognition receptors such as TLR2, DC-SIGN and Dectin-2 on antigen-presenting cells and by T cell receptors on T lymphocytes. The Mtb glycolipids lipoarabinomannan (LAM) and its biosynthetic relatives, phosphatidylinositol mannosides (PIMs) and lipomannan (LM), as well as other Mtb glycolipids, such as phenolic glycolipids and sulfoglycolipids have the ability to modulate the immune response, stimulating or inhibiting a pro-inflammatory response. We explore here the downmodulating effect of Mtb glycolipids. A great proportion of the studies used in vitro approaches although in vivo infection with Mtb might also lead to a dampening of myeloid cell and T cell responses to Mtb glycolipids. This dampened response has been explored ex vivo with immune cells from peripheral blood from Mtb-infected individuals and in mouse models of infection. In addition to the dampening of the immune response caused by Mtb glycolipids, we discuss the hyporesponse to Mtb glycolipids caused by prolonged Mtb infection and/or exposure to Mtb antigens. Hyporesponse to LAM has been observed in myeloid cells from individuals with active and latent tuberculosis (TB). For some myeloid subsets, this effect is stronger in latent versus active TB. Since the immune response in individuals with latent TB represents a more protective profile compared to the one in patients with active TB, this suggests that downmodulation of myeloid cell functions by Mtb glycolipids may be beneficial for the host and protect against active TB disease. The mechanisms of this downmodulation, including tolerance through epigenetic modifications, are only partly explored.