To identify areas at highest risk of infectious disease transmission in Africa, we develop a physical distancing index (PDI) based on the share of households without access to private toilets, water, space, transportation, and communication technology and weight it with population density. Our results highlight that in addition to improving health systems, countries across Africa, especially in the western part of Africa, need to address the lack of essential domestic infrastructure. Missing infrastructure prevents societies from limiting the spread of communicable diseases by undermining the effectiveness of governmental regulations on physical distancing. We also provide high-resolution risk maps that show which regions are most limited in protecting themselves. We find considerable spatial heterogeneity of the PDI within countries and show that it is highly correlated with detected COVID-19 cases. Governments could pay specific attention to these areas to target limited resources more precisely to prevent disease transmission.