-
Notifications
You must be signed in to change notification settings - Fork 33
Package Mutation
The RESTest Mutation Package is a tool designed to enhance the quality of software testing in applications based on OpenAPI specifications. Its goal is to generate mutations in specifications and test cases, allowing for the assessment of the robustness of existing tests and the discovery of potential defects or vulnerabilities.
Mutation is an essential technique in black-box testing and plays a key role in evaluating the effectiveness of existing tests. With the RESTest Mutation Package, teams can explore broader scenarios, improve defect detection capabilities, and ensure the reliability and security of the final software.
The package includes various mutation operators that address different aspects of specifications and test cases. Its use helps teams continually improve test quality and, ultimately, develop more robust and reliable software.
In the following sections, detailed information will be provided on how to use the RESTest Mutation Package, its available mutation operators, practical examples, and guidelines for effective use.
This package contains the core classes related to mutation in the RESTest package. Here, you will find the main classes that enable mutations in OpenAPI specifications and test cases.
The TestCaseMutation class in RESTest is a critical component of the mutation testing technique. Its main purpose is to enable the controlled application of faults or mutations to valid test cases. These mutations are essential to assess the effectiveness of existing test cases and discover possible areas for improvement.
-
The mutate method takes as input a valid test case and an OpenAPI-related operation (specifically an Operation object).
-
Its primary objective is to transform the valid test case into a defective or invalid test case by applying a series of predefined mutation operators. The available operators include:
-
INVALID_VALUE: Changes the value of a parameter in the valid test case to a value that does not comply with the parameter's constraints.
-
REMOVE_REQUIRED_PARAMETER: Removes a required parameter from the valid test case, turning it into an incomplete test case.
-
-
The method applies these operators in a random order to ensure an unbiased evaluation of test case quality.
-
If a mutation operator is successfully applied, the method returns a message indicating which operator was applied. If no mutation is applied, it returns an empty string.
The mutate method is capable of applying different types of mutations to a valid test case, including:
-
Remove Required Parameter: This mutation involves removing a parameter that is necessary for the test case to be valid, making it a defective test case.
-
Change Parameter Value Type: This mutation consists of changing the data type of a parameter's value, such as using a string instead of an integer. This can lead the test case to an invalid state.
-
Violation of Parameter Constraints: This mutation involves introducing values that violate the specified constraints for a parameter, such as using an integer value greater than the maximum allowed value.
The SchemaMutation class in RESTest is a fundamental component of the RESTest mutation toolkit. Its primary purpose is to facilitate the controlled application of mutations to valid OpenAPI specifications, allowing for the assessment of specification robustness and the discovery of potential defects or vulnerabilities.
-
The mutate method takes as input a valid schema and a related OpenAPI specification (specifically, an OpenAPI object).
-
Its main objective is to transform the valid schema into a faulty schema by applying a series of predefined mutation operators. The available operators include:
-
DUPLICATE: Duplicates a random property in the schema, potentially leading to a schema with redundant properties.
-
DROP_SELECT_TYPE: Applies a specific mutation flow that includes the DropRule, SelectRule, and TypeRule on the OpenAPI specification schema.
-
-
The method applies these operators in a random order to ensure an unbiased assessment of schema quality.
-
If a mutation operator is successfully applied, the method returns a message indicating which operator was applied. If no mutation is applied, it returns an empty string.
This package is responsible for managing mutation flows or "pipelines." The classes in this package define and execute specific sequences of mutation that involve the application of multiple mutation rules.
The DropSelectTypePipeline is a prominent component of this package. It acts as a mutation flow designed to apply a combination of mutation rules to an OpenAPI specification schema. These rules include the 'DropRule,' which removes certain properties, the 'SelectRule,' which selectively retains or removes properties, and the 'TypeRule,' which modifies data types within the schema.
The DropSelectTypePipeline provides a powerful tool for systematically altering OpenAPI specification schemas, facilitating comprehensive assessments of their robustness and adaptability in various scenarios.
Within this package, you will find classes dedicated to implementing individual mutation rules utilized within the RESTest Mutation Package.
Overview
The DropRule class is a fundamental component of the RESTest Mutation Package. Its purpose is to remove a randomly selected property from an object schema within an OpenAPI specification. This mutation rule contributes to the systematic mutation of specifications, enabling comprehensive evaluations of their robustness.
Methods
- applyNodeFuzzingRule(Schema<?> schema, String objectChild): This method applies the mutation rule by removing a randomly selected property from the input schema. It ensures that at least one property remains in the schema after the operation.
Overview
The DuplicateRule class is another key element of the RESTest Mutation Package. Its primary function is to duplicate a randomly selected property within an object schema in an OpenAPI specification. This mutation rule supports the generation of diverse specification scenarios for robustness evaluation.
Methods
- applyNodeFuzzingRule(Schema<?> schema, OpenAPI spec): This method implements the mutation rule by duplicating a randomly selected property from the input schema. It also generates a new schema for the duplicated property and adds it to the schema with a "-duplicated" suffix.
Overview
The PathRule class is a foundational element within the RESTest Mutation Package. It serves as an abstract class providing the basic structure for other mutation rules in the package. This class defines how mutation operations are applied to an OpenAPI schema. It handles scenarios involving both object and array data types within the schema.
Methods
-
apply(Schema<?> schema, OpenAPI spec): This method applies mutation operations to the input schema based on its type (object or array). It also handles nested object schemas by recursively applying the mutation operations. The specific mutation logic for altering the schema is implemented in subclasses.
-
applyNodeFuzzingRule(Schema<?> schema, String objectChild): This abstract method, which is implemented in subclasses, defines the core mutation operation for changing the schema.
Overview
The SelectRule class is a versatile mutation rule in the RESTest Mutation Package. It selectively retains or removes properties from an object schema within an OpenAPI specification. This rule supports fine-grained mutation, allowing for focused testing of various specification aspects.
Methods
- applyNodeFuzzingRule(Schema<?> schema, String objectChild): This method applies the mutation rule by selectively retaining a single property in the schema while removing all other properties. It ensures that at least one property remains in the schema after the operation.
Overview
The SingleRule class is an abstract class within the RESTest Mutation Package. It serves as a base class for mutation rules that focus on altering individual properties or aspects of an OpenAPI schema. Subclasses of this rule implement specific mutation operations for different aspects of the schema.
Methods
- apply(Schema<?> schema, boolean internalNode, OpenAPI spec): This method applies mutation operations to the input schema, considering whether it is an internal node or not. If the schema is of type "array," it applies the operation to its items. If the schema contains properties, it processes them based on whether they are nested or not. Subclasses implement the core mutation logic for altering the schema.
Overview
The TypeRule class is a comprehensive mutation rule within the RESTest Mutation Package. Its primary function is to change the data type of a schema. This rule allows for the alteration of data types such as converting a string property into an integer property or vice versa.
Methods
- applyNodeFuzzingRule(Schema<?> schema, OpenAPI spec): This method implements the mutation rule by changing the data type of the input schema to a randomly selected data type, ensuring diversity in the generated specifications. It also clears the example value, promoting consistency in testing.
Within this package, you'll find classes that implement specific mutation operators designed to apply mutations to valid test cases.
The AbstractMutationOperator class plays a crucial role in the mutation operators package. Its necessity lies in providing a common structure and a set of methods that must be implemented by all concrete mutation operators. Essentially, this class acts as a base template for other mutation operators and defines the operations that should be available in each concrete operator.
The primary functionality of the AbstractMutationOperator is to ensure that all mutation operators in the package adhere to a standardized interface. This facilitates the integration of new mutation operators and allows them to function consistently throughout the RESTest system.
The AbstractMutationOperator class defines three abstract methods, each intended for a specific type of mutation:
-
mutate(TestCase tc, Operation specOperation): This method is used when a mutation needs to be applied to a valid test case, and information about the related OpenAPI specification operation is required.
-
mutate(TestCase tc): This method is used when you want to apply a mutation to a valid test case but do not require additional information about the specification operation.
-
mutate(TestCase tc, OpenAPIParameter param): This method is used when mutation is applied to a valid test case, and specific information about an OpenAPI parameter is needed.
In summary, the AbstractMutationOperator ensures that all concrete mutation operators in the package have a consistent structure and offer methods that cover various mutation application scenarios. This simplifies the development and maintenance of mutation operators within the RESTest framework.
The RemoveRequiredParameter class has the primary functionality of applying mutations to test cases by removing a required parameter.
It features a method, mutate(TestCase tc, Operation specOperation) , which is responsible for removing a required parameter from a test case. Parameters are essential for a valid request, so this mutation aids in evaluating how the system responds when a critical parameter is missing. To perform the mutation, the following is required:
-
tc: The test case to be mutated. It's important to note that if the mutation is applied, the original test case object is not preserved. Therefore, it is recommended to clone the test case before calling this method.
-
specOperation: The Swagger operation related to the test case. This information is necessary to extract all parameters that can be modified to have an invalid value.
The method returns a message indicating the required parameter that was removed, such as "Removed required parameter paramName." If the mutation cannot be applied because there are no required parameters that can be removed, it returns an empty string.
The RemoveRequiredParameter class is crucial for evaluating how a system responds when required parameters are missing from requests, ensuring comprehensive testing and a robustness assessment of the system.
This package contains specialized classes for generating invalid values for parameters for testing purposes.
This class, part of the package es.us.isa.restest.mutation.operators.invalidvalue, is designed to insert invalid values into parameters during software testing. It allows introducing values that do not adhere to parameter constraints to evaluate how the system behaves under such adverse conditions.
Methods:
-
mutate(TestCase tc, Operation specOperation): This method attempts to insert an invalid value into a parameter of a test case. For example, it can insert a string value into an integer parameter. The parameters include:
-
tc: The test case to be mutated. Please note that if the mutation is applied, the original tc object will be modified. Therefore, it's recommended to clone the test case before calling this method.
-
specOperation: The OpenAPI operation related to the test case. This is necessary to extract all the required parameters.
-
Returns: A description of the applied mutation or an empty string if no mutation is applied.
-
The InvalidParameterValue class is essential for evaluating how a system responds when its parameters receive invalid values. It supports comprehensive testing and robustness assessment of the system under test. Developers can use this class to insert various types of invalid values into parameters, facilitating thorough testing of the system's behavior in adverse scenarios.
The BooleanToInvalid class has the functionality to mutate a boolean parameter by assigning it an invalid value, which can be a string, an integer, or a double.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param): This method allows mutating a test case by assigning an invalid value to a boolean parameter. The parameters are as follows:
-
tc: The test case to mutate. It's important to note that if the mutation is applied, the original tc object will not be preserved, so it is recommended to clone the test case before calling this method.
-
param: The boolean parameter to mutate.
-
Returns: A description of the applied mutation, or an empty string if no mutation is applied.
-
Constants:
- REPLACE_WITH_INT: Constant representing the mutation of replacing the value with an integer.
- REPLACE_WITH_NUMBER: Constant representing the mutation of replacing the value with a number.
- REPLACE_WITH_STRING: Constant representing the mutation of replacing the value with a string.
The BooleanToInvalid class is essential for evaluating how a system responds when a boolean parameter receives an invalid value, ensuring comprehensive testing and a robustness assessment of the system.
The EnumToInvalid class has the functionality to mutate an enum parameter by assigning it an out-of-range value, which can be a string, a boolean, or an integer.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param): This method allows mutating a test case by assigning an invalid value to an enum parameter. The parameters are as follows:
-
tc: The test case to mutate. It's important to note that if the mutation is applied, the original tc object will not be preserved, so it is recommended to clone the test case before calling this method.
-
param: The enum parameter to mutate.
-
Returns: A description of the applied mutation, or an empty string if no mutation is applied.
-
Constants:
- REPLACE_WITH_INT: Constant representing the mutation of replacing the value with an integer.
- REPLACE_WITH_BOOL: Constant representing the mutation of replacing the value with a boolean.
- REPLACE_WITH_STRING: Constant representing the mutation of replacing the value with a string.
- REPLACE_WITH_NUMBER: Constant representing the mutation of replacing the value with a number.
The EnumToInvalid class is essential for evaluating how a system responds when an enum parameter receives an out-of-range value, ensuring comprehensive testing and a robustness assessment of the system.
The IntegerToInvalid class has the functionality to mutate an integer-type parameter by assigning it a value of string-type (string), boolean-type (boolean), double-type (double), or by violating the maximum or minimum value constraints.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param): This method allows mutating a test case by assigning an invalid value to an integer-type parameter. The parameters are as follows:
-
tc: The test case to mutate. It's important to note that if the mutation is applied, the original tc object will not be preserved, so it is recommended to clone the test case before calling this method.
-
param: The integer-type parameter to mutate.
-
Returns: A description of the applied mutation, or an empty string if no mutation is applied.
-
Constants:
- VIOLATE_MAX_CONSTRAINT: Constant representing the mutation of violating the maximum constraint of the integer parameter.
- VIOLATE_MIN_CONSTRAINT: Constant representing the mutation of violating the minimum constraint of the integer parameter.
- REPLACE_WITH_NUMBER: Constant representing the mutation of replacing the value with a number.
- REPLACE_WITH_STRING: Constant representing the mutation of replacing the value with a string.
- REPLACE_WITH_BOOL: Constant representing the mutation of replacing the value with a boolean.
The **IntegerToInvalid **class is fundamental for evaluating how a system responds when an integer-type parameter receives invalid values, ensuring comprehensive testing and a robustness assessment of the system.
The NumberToInvalid class has the functionality to mutate a number-type parameter by assigning it a string-type (string), boolean-type (boolean), or violating the maximum or minimum value constraints.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param): This method allows mutating a test case by assigning an invalid value to a number-type parameter. The parameters are as follows:
-
tc: The test case to mutate. It's important to note that if the mutation is applied, the original tc object will not be preserved, so it is recommended to clone the test case before calling this method.
-
param: The number-type parameter to mutate.
-
Returns: A description of the applied mutation, or an empty string if no mutation is applied.
-
Constants:
- VIOLATE_MAX_CONSTRAINT: Constant representing the mutation of violating the maximum constraint of the number parameter.
- VIOLATE_MIN_CONSTRAINT: Constant representing the mutation of violating the minimum constraint of the number parameter.
- REPLACE_WITH_STRING: Constant representing the mutation of replacing the value with a string.
- REPLACE_WITH_BOOL: Constant representing the mutation of replacing the value with a boolean.
The NumberToInvalid class is crucial for evaluating how a system responds when a number-type parameter receives invalid values, ensuring comprehensive testing and a robustness assessment of the system.
The StringToInvalid class has the functionality to mutate a string-type parameter by assigning it another string-type value that violates format, maximum length, or minimum length constraints.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param): This method allows mutating a test case by assigning an invalid value to a string-type parameter. The parameters are as follows:
-
tc: The test case to mutate. It's important to note that if the mutation is applied, the original tc object will not be preserved, so it is recommended to clone the test case before calling this method.
-
param: The string-type parameter to mutate.
-
Returns: A description of the applied mutation, or an empty string if no mutation is applied.
-
Constants:
- VIOLATE_FORMAT_CONSTRAINT: Constant representing the mutation of violating the format constraint of the string parameter.
- VIOLATE_MAX_LENGTH_CONSTRAINT: Constant representing the mutation of violating the maximum length constraint of the string parameter.
- VIOLATE_MIN_LENGTH_CONSTRAINT: Constant representing the mutation of violating the minimum length constraint of the string parameter.
The StringToInvalid class is essential for evaluating how a system responds when a string-type parameter receives invalid values, ensuring comprehensive testing and a robustness assessment of the system.
This class is a core component for mutating parameters during software testing. It allows for the introduction of invalid values into parameters to assess system behavior under adverse conditions.
Methods:
-
mutate(TestCase tc, OpenAPIParameter param, String[] mutations): This method facilitates test case mutation by introducing invalid values into a parameter. Parameters include:
-
tc: The test case to be mutated. It is important to note that if the mutation is applied, the original tc object will be altered. Therefore, it is advisable to clone the test case before calling this method.
-
param: The parameter intended for mutation.
-
mutations: An array of strings representing the types of mutations to apply to the parameter.
-
Returns: A description of the applied mutation. Returns an empty string if no mutation is applied.
-
-
private static String getMutationMessage(String mutation, OpenAPIParameter param, TestCase tc, String newValue): This internal method generates a descriptive message about the applied mutation. It is used by the mutate method to provide clear information about the performed mutations. Parameters include:
-
mutation: A string indicating the type of mutation.
-
param: The parameter being mutated.
-
tc: The test case being mutated.
-
newValue: The new value assigned to the parameter (may be null in certain cases).
-
The AbstractToInvalidOperator class is fundamental for assessing how a system responds when parameters receive invalid values. It supports comprehensive testing and robustness evaluation of the system under test. Developers can extend this class to implement specific mutation operators tailored to their testing requirements.
Each class and package plays a significant role in the RESTest mutation process, enabling mutations in OpenAPI specifications and test cases, thus enhancing the robustness and reliability of software tests based on OpenAPI.