Caves form where wave erosion hollows out a cavity in a headland or sea cliff. If the opposite side of the headland is open, then continued erosion may eventually form an arch.
Images
Fingal's Cave, a sea cut cave in a large basaltic lava flow, Scotland. Original picture from the Ward Collection