Empowering Statistical Analysis in FaaS Environments Through the Orama Framework

The Function-as-a-Service cloud computing model has aroused great interest in the community. In view of this, the importance of tools that help in the execution of benchmarks in environments with this architecture has grown. The Orama framework has positioned itself as an aid tool in this type of evaluation, allowing the comparison of environments of this category through statistical analyses with factorial design and t-test, as well as a direct comparison between the evaluated levels of concurrence. In this work, the Orama framework is used for comparative execution of benchmarks between four large cloud providers (AWS, GCP, Azure and Alibaba) in two different use case scenarios. The results showed that Alibaba maintained an average time lower than its rivals, while Azure suffered from a high penalty on the first request due to the cold start. AWS and GCP presented similar performances, including failures in the face of high levels of concurrence.