Regression testing

Regression testing (rarely, non-regression testing[1]) is re-running functional and non-functional tests to ensure that previously developed and tested software still performs as expected after a change.[2] If not, that would be called a regression.

Changes that may require regression testing include bug fixes, software enhancements, configuration changes, and even substitution of electronic components (hardware).[3] As regression test suites tend to grow with each found defect, test automation is frequently involved. The evident exception is the GUIs regression testing, which normally must be executed manually. Sometimes a change impact analysis is performed to determine an appropriate subset of tests (non-regression analysis[4]).

Background

As software is updated or changed, or reused on a modified target, emergence of new faults and/or re-emergence of old faults is quite common.

Sometimes re-emergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area.

Finally, it may happen that, when some feature is redesigned, some of the same mistakes that were made in the original implementation of the feature are made in the redesign. Therefore, in most software development situations, it is considered good coding practice, when a bug is located and fixed, to record a test that exposes the bug and re-run that test regularly after subsequent changes to the program.[5]

Although this may be done through manual testing procedures using programming techniques, it is often done using automated testing tools.[6] Such a test suite contains software tools that allow the testing environment to execute all the regression test cases automatically; some projects even set up automated systems to re-run all regression tests at specified intervals and report any failures (which could imply a regression or an out-of-date test).[7]

Common strategies are to run such a system after every successful compile (for small projects), every night, or once a week. Those strategies can be automated by an external tool.

Regression testing is an integral part of the extreme programming software development method. In this method, design documents are replaced by extensive, repeatable, and automated testing of the entire software package throughout each stage of the software development process. Regression testing is done after functional testing has concluded, to verify that the other functionalities are working.

In the corporate world, regression testing has traditionally been performed by a software quality assurance team after the development team has completed work. However, defects found at this stage are the most costly to fix. This problem is being addressed by the rise of unit testing. Although developers have always written test cases as part of the development cycle, these test cases have generally been either functional tests or unit tests that verify only intended outcomes. Developer testing compels a developer to focus on unit testing and to include both positive and negative test cases.[8]

Techniques

The various regression testing techniques are:

Retest all

This technique checks all the test cases on the current program to check its integrity. Though it is expensive as it needs to re-run all the cases, it ensures that there are no errors because of the modified code.[9]

Regression test selection

Unlike Retest all, this technique runs a part of the test suite (owing to the cost of retest all) if the cost of selecting the part of the test suite is less than the Retest all technique.[9]

Test case prioritization

Prioritize the test cases so as to increase a test suite's rate of fault detection. Test case prioritization techniques schedule test cases so that the test cases that are higher in priority are executed before the test cases that have a lower priority.[9]

Types of test case prioritization

  • General prioritization – Prioritize test cases that will be beneficial on subsequent versions
  • Version-specific prioritization – Prioritize test cases with respect to a particular version of the software.

Hybrid

This technique is a hybrid of regression test selection and test case prioritization.[9]

Benefits and drawbacks

Regression testing is performed when changes are made to the existing functionality of the software or if there is a bug fix in the software. Regression testing can be achieved through multiple approaches; if a test all approach is followed, it provides certainty that the changes made to the software have not affected the existing functionalities, which are unaltered.[10]

In agile software development—where the software development life cycles are very short, resources are scarce, and changes to the software are very frequent—regression testing might introduce a lot of unnecessary overhead.[10]

In a software development environment which tends to use black box components from a third party, performing regression testing can be tricky, as any change in the third-party component may interfere with the rest of the system (and performing regression testing on a third-party component is difficult, because it is an unknown entity).[10]

Uses

Regression testing can be used not only for testing the correctness of a program but often also for tracking the quality of its output.[11] For instance, in the design of a compiler, regression testing could track the code size and the time it takes to compile and execute the test suite cases.

Also as a consequence of the introduction of new bugs, program maintenance requires far more system testing per statement written than any other programming. Theoretically, after each fix, one must run the entire batch of test cases previously run against the system to ensure that it has not been damaged in an obscure way. In practice, such regression testing must indeed approximate this theoretical idea, and it is very costly.

Regression tests can be broadly categorized as functional tests or unit tests. Functional tests exercise the complete program with various inputs. Unit tests exercise individual functions, subroutines, or object methods. Both functional testing tools and unit-testing tools tend to be automated and are often third-party products that are not part of the compiler suite. A functional test may be a scripted series of program inputs, possibly even involving an automated mechanism for controlling mouse movements and clicks. A unit test may be a set of separate functions within the code itself or a driver layer that links to the code without altering the code being tested.

The role of AI in Regression Testing

Regression testing is a crucial phase in software development, ensuring that new code changes do not negatively impact existing functionalities. Over time, the role of Artificial Intelligence (AI) in regression testing has significantly evolved, offering several advantages to streamline and enhance the testing process.[12] This section discusses the key contributions of AI in the context of regression testing.

Automated Test Creation

One of the primary roles of AI in regression testing tools is the ability to autonomously generate and automate test cases. AI algorithms can analyze the software's codebase and identify potential areas where regression tests should be conducted. This automation significantly reduces the manual effort required for test case creation, making the testing process more efficient.

Test Maintenance

AI-powered regression testing tools are equipped with capabilities to efficiently manage and update test suites. These tools can detect changes in the application's codebase and automatically adjust test cases accordingly, reducing the need for extensive manual test maintenance. This leads to a more agile and responsive testing process.

Enhanced Test Coverage

AI-driven regression testing tools can identify and prioritize test scenarios based on historical data and code changes. This ensures that critical test cases are executed more frequently, thereby enhancing test coverage and increasing the chances of identifying regressions early in the development cycle.

Efficient Bug Detection

AI algorithms can be trained to identify patterns associated with common regression issues. This enables these tools to detect potential regressions more effectively and with greater accuracy. AI can also perform exploratory testing, helping testers uncover hidden bugs and improve overall software quality.

Automated Reporting

AI can automate the generation of detailed test reports. These reports provide actionable insights into the state of the software, including information on test results, code changes, and potential regressions. Automated reporting saves time for quality assurance teams and allows them to focus on analyzing results rather than report generation.

AI Oversight

In regression testing, AI can work in conjunction with human testers. QA experts can monitor the performance of AI algorithms, provide evaluations, and offer feedback for continuous improvement. This collaborative approach ensures that AI adapts to evolving testing needs.

Product-Centric Approach

AI-powered regression testing tools enable development teams to concentrate on product development rather than managing testing frameworks, libraries, and scripts. This shift towards a product-centric approach enhances the overall efficiency of the software development process.

Incorporating AI into regression testing tools has revolutionized the way software testing is conducted. It has led to faster, more reliable, and cost-effective testing processes, ultimately contributing to higher software quality. As technology continues to advance, the role of AI in regression testing tools is expected to evolve further, offering even more sophisticated capabilities for ensuring software reliability and stability.

See also

References

  1. Pezzè, Mauro; Young, Michal (2008). Software testing and analysis: process, principles, and techniques. Wiley. Testing activities that focus on regression problems are called (non) regression testing. Usually "non" is omitted
  2. Basu, Anirban (2015). Software Quality Assurance, Testing and Metrics. PHI Learning. ISBN 978-81-203-5068-7.
  3. National Research Council Committee on Aging Avionics in Military Aircraft: Aging Avionics in Military Aircraft. The National Academies Press, 2001, page 2: ″Each technology-refresh cycle requires regression testing.″
  4. Boulanger, Jean-Louis (2015). CENELEC 50128 and IEC 62279 Standards. Wiley. ISBN 978-1119122487.
  5. Kolawa, Adam; Huizinga, Dorota (2007). Automated Defect Prevention: Best Practices in Software Management. Wiley-IEEE Computer Society Press. p. 73. ISBN 978-0-470-04212-0.
  6. Automate Regression Tests When Feasible, Automated Testing: Selected Best Practices, Elfriede Dustin, Safari Books Online
  7. daVeiga, Nada (2008-02-06). "Change Code Without Fear: Utilize a Regression Safety Net". Dr. Dobb's Journal.
  8. Dudney, Bill (2004-12-08). "Developer Testing Is 'In': An interview with Alberto Savoia and Kent Beck". Retrieved 2007-11-29.
  9. Duggal, Gaurav; Suri, Bharti (2008-03-29). Understanding Regression Testing Techniques. National Conference on Challenges and Opportunities. Mandi Gobindgarh, Punjab, India. CiteSeerX 10.1.1.460.5875.
  10. Yoo, S.; Harman, M. (2010). "Regression testing minimization, selection and prioritization: a survey". Software Testing, Verification and Reliability. 22 (2): 67–120. doi:10.1002/stvr.430.
  11. Kolawa, Adam. "Regression Testing, Programmer to Programmer". Wrox.
  12. "MuukTest's AI-Powered Test Automation Platform Releases New Features". 2023-09-01. Retrieved 2023-10-06.
This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.