Regression·Field Guide
DARK
SE401 · Software Testing and Quality · Chapter 06

You fixed one thing.
Did you break another?

Software never stops changing — bug fixes, new features, refactors, a new server OS. Every one of those changes can quietly break code that used to work fine. Regression testing is the discipline of re-running your test suite after every change so "it worked yesterday" doesn't turn into "it's broken today" without anyone noticing.

Definition & cycle
Cost problem
Selection vs prioritization
Exam-tested
REGRESSION CYCLE BUILD TEST FIX BUGS WRITE REG. TESTS RE-RUN SHIP no code untested
§ 01 — SOFTWARE EVOLUTION

Change is constant — and it doesn't stay where you put it

This is the whole justification for regression testing existing as a discipline: change happens everywhere, all the time, and it doesn't respect boundaries.

🌊
Memory hook
"Change happens before and after delivery — everywhere, forever."
Change happens throughout the whole SDLC, can happen to every aspect of software, and can even affect areas nobody touched.
⏳

Happens throughout the SDLC

Before delivery (still in development) and after delivery (in maintenance) — change never fully stops.

🧩

Can happen to every aspect

Requirements, code, design, configuration, environment — nothing is off-limits.

💥

Can affect unchanged areas

A change in one place can break code, introduce new bugs, uncover previously unknown bugs, or reintroduce old bugs — all in code nobody edited.

§ 02 — DEFINITION

What regression testing actually is

Regression Test

re-verify after modification

Testing of a previously tested program following modification, to ensure that new defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It should be performed whenever the software or its environment changes, and it applies to testing at all levels.

🔁
Memory hook
"Yesterday it worked, today it doesn't."
"I was fixing X, and accidentally broke Y" / "That bug was fixed, but now it's back." Both are regressions — the whole reason tests get re-run.
In one sentence

A type of software testing that intends to ensure that changes — enhancements or defect fixes — to the software have not adversely affected it. It reruns both functional and non-functional tests, and it's a type of change-related testing: it detects whether defects have been introduced or uncovered in unchanged areas.

The working loop
Keep a test suite → use it after every change → compare output with previous tests, understanding all changes → if new tests are needed, add them to the suite.
Build Run Tests Find & Resolve Bugs Ship Write Regression Tests Run Regression Tests

The cycle repeats: build → run tests → find/resolve bugs → write regression tests for the fix → run regression tests → ship — then it all starts again on the next change.

Regression vs. System vs. Acceptance

The instructor's slides place regression testing side by side with the two other test types students already know, to sharpen what makes it different.

SystemAcceptanceRegression
Test for…Correctness, completionUsefulness, satisfactionAccidental changes
Test by…Development test groupTest group with usersDevelopment test group
CategoryVerificationValidationVerification
§ 03 — REASONS

Why regression testing is needed

Conducted to make sure that fixing one thing has not broken another. The need can arise from any of these changes.

🔧
Memory hook — chunk it into 3 buckets
"Fix it, Build it, Move it."
Fix it — defect fix. Build it — new feature, changed feature, code refactoring, design/architecture change. Move it — configuration/environment change (hardware, software, network).
🐞

Defect fix

A bug was patched — did the patch touch anything else?

✨

New feature

New code paths can interact with existing ones in unexpected ways.

🔀

Change in existing feature

Behavior shifted — every dependent path needs re-checking.

🧱

Code refactoring

Internals restructured, external behavior should be unchanged — prove it.

🏛️

Design / architecture change

A structural shift can ripple through every subsystem.

🖥️

Configuration / environment change

New hardware, software, or network — same code, different world.

§ 04 — BASIC PROBLEMS

The two problems that make regression testing expensive

Maintaining the test suite

upkeep

If I change feature X, how many test cases must be revised because they use feature X? Which test cases should be removed or replaced? Which should be added?

Cost of re-testing

execution cost

Often proportional to the size of the whole product, not the size of the change. A big problem if testing requires manual effort.

⚖️
Memory hook
"Keep it lean, keep it cheap."
A well-maintained (lean) suite is what makes re-testing (cheap) actually affordable — the two problems feed each other.
§ 05 — TEST CASE MAINTENANCE

Some maintenance is inevitable. Some should be avoided.

Inevitable
If feature X has changed, test cases for feature X will require updating — there's no way around this.
Should be avoided
Example: trivial changes to a user interface or file format should not invalidate large numbers of test cases.
🧩
Memory hook
"Test suites should be modular!"
Avoid unnecessary dependence between test cases and implementation details. Generating concrete test cases from test case specifications can help keep that modularity.
§ 06 — OBSOLETE & REDUNDANT

Two reasons a test case might not deserve to stay in the suite

🗑️
Memory hook
"Obsolete is dead. Redundant is just a twin."
Obsolete = no longer valid, must go. Redundant = still valid, just overlaps with another — may or may not go, it's a cost decision.
TermMeaningWhat to do
Obsolete A test case that is no longer valid — it tests features that have been modified, substituted, or removed. Should be removed from the test suite.
Redundant A test case that does not differ significantly from others — unlikely to find a fault missed by similar test cases. Has real cost in re-execution and (maybe more) in human effort to maintain — may or may not be removed, depending on those costs.
§ 07 — SELECTING & PRIORITIZING

Should you re-run the whole suite? In what order?

Maybe you don't care — if you can re-run everything automatically over lunch break, just do it. But sometimes you do care.

🚦
If-then decision rule
If test cases are expensive to execute → Selection matters.
If a very large suite can't be executed every day → Prioritization matters.
Selection decides whether a test case runs at all. Prioritization decides what order it runs in. They solve different cost problems — don't conflate them.

Code-Based Regression Test Selection

coverage-driven

Observation: a test case can't find a fault in code it doesn't execute. In a large system, many parts of the code are untouched by many test cases. So: only execute test cases that execute changed or new code.

How to picture it

A codebase map where most regions are shaded "executed by test case" — but only the test cases whose shaded region overlaps the small "new or changed" patch get selected to re-run.

Requires
Coverage instrumentation that knows, per test case, exactly which lines/modules it exercises — so you can trace changed code back to the tests that touch it.

Specification-Based Regression Test Selection

functionality-driven

Like code-based selection: pick test cases that test new and changed functionality. Difference: no guarantee of independence — a test case that isn't "for" a changed or added feature X might still find a bug in feature X anyway.

Typical approach
Specification-based prioritization: execute all test cases, but start with the ones related to changed and added features.
Why not just exclude the rest?

Because there's no coverage guarantee like in code-based selection — an "unrelated" test might be the one that catches the bug, so it's safer to reorder than to drop.

Prioritized Rotating Selection

everything, eventually

Basic idea: execute all test cases, eventually — but execute some sooner than others.

🎡
Acronym — R·T·S
"Round, Track, Structure"
3 possible priority schemes to pick who goes first.
SchemePriority rule
Round RobinPriority to least-recently-run test cases.
Track recordPriority to test cases that have detected faults before.
StructuralPriority for executing elements that have not been recently executed.
§ 08 — CHEAT SHEET

The whole chapter in one scannable table

ConceptKey ideaNote / example
Regression testingRe-test a previously-tested program after modification, to catch new defects in unchanged areasApplies at all test levels
Why it's neededDefect fix, new/changed feature, refactor, design/architecture change, config/environment change"Fixing one thing broke another"
Maintaining the suiteWhich tests need revising, removing, or adding as features changeSome maintenance is inevitable; trivial changes shouldn't cascade
Cost of re-testingProportional to product size, not change sizeWorse if manual
Obsolete test caseNo longer valid — features modified/substituted/removedShould be removed
Redundant test caseDoesn't differ meaningfully from othersMay or may not be removed — a cost tradeoff
Selection matters whenTest cases are expensive to executeOtherwise, just re-run everything
Prioritization matters whenA very large suite can't run every dayOrder execution instead of skipping tests
Code-based selectionOnly run tests that execute changed/new codeA test can't find a fault in code it doesn't touch
Spec-based selectionPick tests tied to changed/new functionality — no independence guaranteeTypical: run everything, prioritize changed-feature tests
Prioritized rotatingRound Robin · Track record · StructuralRun all eventually, some sooner
§ 09 — COMMON MISTAKES

Where students actually lose marks

Each of these is a genuinely confusable point straight from the slides — drill the RIGHT side until it's automatic.

1. Regression testing "checks correctness" like system testing

✗ WRONG — treating regression testing as just another correctness/completion check, same as system testing.
✓ RIGHT — regression testing specifically targets accidental changes caused by a modification. It's verification, done by the development test group, but its target is "did the change break something," not "is the feature complete."

2. "Redundant" and "Obsolete" test cases are treated the same

✗ WRONG — assuming any overlapping/redundant test case should be deleted just like an obsolete one.
✓ RIGHT — obsolete test cases (invalid, feature changed/removed) should be removed. Redundant test cases are still valid; whether to remove them depends on the cost of re-execution vs. the cost of maintaining them.

3. Assuming re-testing cost scales with the size of the change

✗ WRONG — "it was a small fix, so regression testing should be cheap."
✓ RIGHT — cost is often proportional to the size of the whole product, not the size of the change. A one-line fix can still require re-running a suite sized to the entire system.

4. Confusing selection with prioritization

✗ WRONG — using the two terms interchangeably.
✓ RIGHT — selection decides whether a test case runs at all (matters when execution is expensive). Prioritization decides the order tests run in (matters when a huge suite can't run in the available time).

5. Assuming spec-based selection is as safe as code-based selection

✗ WRONG — dropping test cases not "for" a changed feature because spec-based selection looks like code-based selection.
✓ RIGHT — code-based selection has a coverage guarantee (a test can't catch a bug in code it never executes). Spec-based selection has no such guarantee, so the safe approach is to prioritize, not exclude — run everything, changed-feature tests first.

6. Believing you should always minimize which tests you run

✗ WRONG — always trying to trim the regression suite down.
✓ RIGHT — if you can re-run the entire suite automatically (e.g. overnight or over a lunch break) at no real cost, selection doesn't matter at all. It only becomes a problem when execution is expensive.

7. Letting a trivial change invalidate a huge chunk of the suite

✗ WRONG — a cosmetic UI or file-format tweak triggers mass revision of unrelated test cases.
✓ RIGHT — this is exactly the "maintenance that should be avoided." Well-modularized test suites avoid unnecessary dependence, so a trivial change only touches the tests genuinely tied to it.
§ 10 — POP QUIZ

Quick self-check

Click a question to reveal the answer.

Multiple choice
1. What does regression testing specifically test for, according to the System / Acceptance / Regression comparison?
Accidental changes. (System tests for correctness/completion; Acceptance tests for usefulness/satisfaction; Regression tests for accidental changes caused by a modification.)
Short answer
2. Name at least three of the six reasons the need for regression testing could arise.
Any three of: defect fix, new feature, change in an existing feature, code refactoring, change in technical design/architecture, change in configuration/environment (hardware, software, network).
True / False
3. The cost of re-testing is usually proportional to the size of the change that was made.
False. It's often proportional to the size of the whole product, not the size of the change — and worse if testing requires manual effort.
Short answer
4. What's the difference between an obsolete test case and a redundant test case?
Obsolete = no longer valid (tests a modified/removed feature) and should be removed. Redundant = still valid but doesn't differ significantly from other test cases; may or may not be removed depending on the cost of re-execution vs. maintenance.
Multiple choice
5. When does test case selection matter most? A) Always  B) When test cases are expensive to execute  C) Never  D) Only for new features
B. If you can automatically re-run everything cheaply, selection doesn't matter at all.
Multiple choice
6. When does prioritization matter most?
When a very large test suite cannot be executed every day — so you need to decide what runs first.
Short answer
7. Name the three prioritized rotating selection schemes.
Round Robin (least-recently-run first), Track record (priority to tests that found faults before), Structural (priority to elements not recently executed).
Multiple choice
8. In code-based regression test selection, what's the guiding rule?
Only execute test cases that execute changed or new code — because a test case can't find a fault in code it doesn't execute.
§ 11 — PRACTICE LAB

Scenario questions, worked through

Exam-style scenarios grounded directly in the chapter's concepts.

1. A team fixes a bug in the payment module. Two days after release, a support ticket reports the previously-working discount code is now broken. What kind of failure is this, and what should have caught it before release?
→ A classic regression
Why: exactly "I was fixing X, and accidentally broke Y." A regression test suite re-run after the payment-module fix — covering unchanged areas like the discount code — should have caught it before ship.
2. Your regression suite has 4,000 test cases, manual execution is prohibitively expensive, but you have coverage instrumentation showing exactly which tests exercise which modules. Which selection approach fits, and why?
→ Code-based regression test selection
Why: with coverage data available, only run test cases that execute the changed or new code — a test can't find a fault in code it doesn't touch, so untouched regions can be safely skipped.
3. Your regression suite can be executed in full, automatically, overnight with zero manual effort. Should you bother with selection or prioritization?
→ No
Why: selection and prioritization only matter when execution is expensive or the full suite can't fit in the available time. If you can re-rerun everything automatically for free, just do it — "maybe you don't care."
4. A UI test suite has hundreds of tests for a login screen. The team fixes a purely cosmetic CSS spacing bug on that screen. Should all login tests be revised?
→ No
Why: trivial UI changes shouldn't invalidate large numbers of test cases — this is exactly the "maintenance that should be avoided." A well-modularized suite limits the blast radius to the tests actually tied to the changed behavior.