Serverless computing has emerged as a powerful paradigm for distributed workflows, offering fine-grained, low-latency resource provisioning to precisely meet job demands. While workflows often utilize hundreds of concurrent CPUs, existing serverless benchmarking suites frequently concentrate on small-scale parallelism and microservice workloads. Furthermore, these benchmarks typically consider only Function-as-a-Service (FaaS) backends, overlooking their natural cloud successors, Container-as-a-Service (CaaS). These limitations have created a significant gap in evaluating a crucial feature of serverless platforms: their ability to accurately handle sudden changes in resource allocation, also known as elasticity. To address this, we introduce gumeter, a new benchmarking suite specifically designed to evaluate the elasticity of serverless platforms (both FaaS and CaaS) for highly parallel distributed workflows. gumeter facilitates a thorough assessment of an underlying platform using a “fire-and-forget” execution model. It leverages a set of comprehensive pipelines that sample various typical scaling behaviors, providing an in-depth analysis of elasticity, execution time, cost, and efficiency. We apply gumeter to evaluate the elasticity of three popular serverless platforms: AWS Lambda, Google Cloud Run, and IBM Code Engine. Our results reveal significant differences in elasticity across these platforms, showing that FaaS offerings still outperform CaaS in elasticity by a factor of up to 6.5x. However, CaaS can be up to 64.5% less expensive than FaaS, thereby unveiling an interesting optimization space for cloud workflows.