Sediment is highly mobile under the combined effects of waves, currents, tides, and winds in coastal systems. This frequent sediment movement, reconfiguration, and sorting can lead to high spatial and temporal variability in the surface-most sediment layer. To advance the ability to better map these complex patterns in surface sediment composition over broad spatio-temporal scales, we develop a 4 year observational record of spatially extensive grain size variability at the U.S. Army Engineer Research and Development Center (ERDC) Field Research Facility (FRF) in Duck, NC, USA by leveraging Argus camera observations to train a deep learning model on visual differences between quartz sand and coarse shell gravel. Tests of model performance against both labeled imagery and 3 years of weekly grain size sampling demonstrates that machine learning approaches have skill in mapping out complex observed patterns in coastal bed sediments.