Caves in coastal cliffs in the Apostle Islands were analyzed for factors controlling their origin and evolution. Even though the Bayfield Group quartz sandstones were found to be of relatively low tensile strength, the traditional explanation attributing speleogenesis to localized zones of lithological and tectonic weakness does not adequately explain all geomorphic features of the caves. The caves were found to be formed by coalescence of smaller arcade-shaped cavities serially developed along subhorizontal discontinuities. In concordance with the previously formulated “arcade principle”, positions and shapes of the cavities are defined by areas of reduced stress within the gravity-induced stress field. Evacuation of material from the rock massif can be attributed to wave action including the effect of water-pressurized air, and frost action. Compound cavities above the discontinuities tend to form “higher-order” arcade-like cavities/caves while those beneath more resistant strata rather develop into elongated overhangs similar to ordinary coastal notches. Propagation of the largest cavities, formed along steeply dipping joints and faults, is attributed to mass wasting and roof collapses. Similar examples of discontinuity-related cavities in coastal settings, suspectedly also controlled by gravity-induced stress distribution, can be found worldwide, being mostly concentrated to areas covered by ice sheets in the Pleistocene. The interplay between post-glacial isostatic uplift and Holocene lake-level rise is responsible for the development of mature forms of coastal caves in the Apostle Islands where present cliffs are estimated to have been exposed to the effects of wave action for the last 3–5 ky.