A Public Benchmark of REST APIs

In software engineering, benchmarks are widely used to evaluate and compare the performance, functionality, and reliability of analysis tools. Despite the prevalence of benchmarks in areas such as databases, machine learning, and programming languages, there is a notable absence of publicly available benchmarks for REST APIs, a cornerstone of modern web-based systems. While existing research papers occasionally employ similar REST APIs in their evaluations, opportunistic API selection hampers comparison. Moreover, these studies often rely on API documentation and structural characteristics. Without a reliable benchmark, API data used in evaluations may be outdated or inaccurate, compromising reliability and reproducibility. Hence, this paper addresses a gap in the literature by providing a comprehensive and Public REST API Benchmark (PRAB), to be utilized by researchers in their evaluations. The benchmark contains documentation and structural characteristics of 60 publicly available REST APIs. First, we conduct a systematic mapping study to discover the available and public REST APIs that are utilized in the academic literature. Then, by analyzing the resulting APIs, we report their structural characteristics (e.g., routes, query parameters, HTTP methods, authentication). Finally, we provide their documentation (i.e., OpenAPI Specification, Postman Collection) in a publicly available GitHub repository, to help with future evaluations of REST API studies.

Conference paper