Scaffolds, also called sidewalk sheds, are intended to be temporary structures to protect pedestrians from construction and repair hazards. However, some sidewalk sheds are left up for years; this trend recently motivated New York City’s (NYC) mayor to issue a ‘Get Sheds Down’ plan that notably took down a scaffold in Harlem that had been up for 21 years. Today, there are over 9000 active permits for scaffolds in NYC. This research uses computer vision to develop a longitudinal map of scaffolding. Using a dataset of 29,156,833 images recorded from vehicle-attached dashcams between August 2023 and January 2024 in NYC, we develop an algorithm to track scaffolding presence over time and match detected scaffolds to reported permit locations. We identify 850,766 images of scaffolding, tagging 5156 active sidewalk sheds and estimating 529 unpermitted sheds. We discuss implications for urban tech, governmental inspection processes, and out-of-distribution evaluations outside NYC.