Multi-Agent LLM-based Metamorphic Testing for REST APIs
基于多智能体LLM的REST API蜕变测试
Shehroz Khan, Abdullah Mughees, Gaadha Sudheerbabu, Tanwir Ahmad, Dragos Truscan
AI总结 提出ARMeta方法,利用基于LLM的多智能体工作流自动识别蜕变测试场景并生成可执行测试,以解决REST API测试中的预言问题。
详情
- Comments
- Author submitted version accepted for publication the IEEE Conference on Computers, Software, and Applications (COMPSAC2026), July 7-11, 2026, Madrid Spain
随着REST API在软件系统中日益重要,其验证也变得更为关键。因此,测试和发现潜在问题对于提高软件质量至关重要。然而,测试REST API的主要挑战在于难以评估API调用的输出是否正确,即测试预言问题。蜕变测试是一种基于规约的测试方法,适用于正确输出未知或未明确指定的情况。为了检查系统的正确性,需要指定不同输出之间的关系。我们提出了ARMeta,一种支持工具的方法,利用基于LLM的多智能体工作流来支持使用OpenAPI文档化的REST API的蜕变测试。该智能体工作流用于识别蜕变测试场景,并以Given-When-Then格式进行规约。这些场景自动实现为可执行测试,并针对被测系统执行。我们在两个公开的暴露REST接口的Web应用程序上评估了ARMeta,并将其性能与基于场景的测试基线进行了比较。结果表明,ARMeta探索的行为可作为现有基于场景的测试方法的补充。
As REST APIs become an increasingly significant part of software systems, their validation is becoming more critical. Hence, testing and uncovering underlying issues are of utmost importance for improving software quality. However, testing REST APIs is challenging mainly due to the difficulty of assessing whether the output of an API call is correct, i.e., the test oracle problem. Metamorphic testing is a specification-based testing approach for situations where correct outputs are unknown or not specified explicitly. To check the correctness of a system, relations between the different outputs are specified. We present ARMeta, a tool-supported approach that uses an LLM-based multi-agent workflow to support metamorphic testing of REST APIs documented with OpenAPI. The agentic workflow is used to identify metamorphic test scenarios and specify them in the Given-When-Then format. These scenarios are automatically implemented as executable tests and executed against the system under test. We evaluate ARMeta on two publicly available web applications that expose REST interfaces and compare its performance with a scenario-based testing baseline. The results show that ARMeta explores behaviors that serve as a complement to existing scenario-based testing approaches.