Your code compiles.
That was never the hard part.
Testing exists because "it runs" and "it's correct" are two completely different claims. This chapter is the foundation for the entire course: why bugs are expensive (sometimes fatal), the exact vocabulary that separates an error from a defect from a failure, the seven principles every tester internalizes, and the process that turns "let's test it" into something repeatable.
- Midterm: listing-type questions are capped around 2 points — but you're expected to understand everything, not just list it.
- No hand-drawn diagrams needed in exam answers.
- Be ready to give a real example / case study on request (not just recite one).
- The whole chapter is really answering four questions: Why are we doing testing? What happens if we didn't? What's a test plan? What's a test case?
Software is everywhere, and unfinished testing has a body count
Software now runs phones, smart homes, satellites, trucks, farm equipment, and aircraft. When it fails, the failure isn't abstract.
A software bug delivered lethal radiation doses
Malfunctioned due to a software bug, leaving 3 people dead and critically injuring 3 others.
Crashed due to a software bug
Killed 264 innocent lives.
Self-destructed 39 seconds after liftoff
Deep-dive case study below — this one gets its own section because the instructor spent real time on it.
Failed because of a software bug
Called out in the deck as the costliest accident in history.
A software bug credited 823 customers $920 million
Not every failure kills someone — some just quietly move a billion dollars that was never supposed to move.
Deep dive — Ariane 5: what actually happened
the exam's favorite case studyEuropean expendable launch system, built to carry double the payload of its predecessor Ariane 4 (which had 113 successful launches and 3 failures).
Inside SRI 2 (inertial reference system): the value BH (Bias Horizontal) exceeds 2¹⁵. The routine
convert_double_to_int(BH) fails — the value
is too big to fit. An exception is thrown, crashing SRI 2, then SRI 1 too.Result: the onboard computer (OBC) receives diagnostic data and misreads it as real flight data — the rocket becomes disoriented, swings past a 20° angle, hits huge aerodynamic stress, and the boosters start separating.
T0+39s: self-destruction. Cost: €500M.
Not a design error either — it was justified against Ariane 4's flight trajectory and real-time constraints.
A problem with integration testing — theoretically detectable, but the test space was huge versus limited resources, and the SRI software was actually useless at that stage of flight anyway.
Reuse of a component with a hidden constraint — precondition was
abs(BH) < 32768.0, valid for Ariane 4's flight profile but no longer
valid for the more powerful Ariane 5.
Per the Cost of Poor Software Quality in the US, 2020 report (CISQ — Consortium for Information & Software Quality): the total Cost of Poor Software Quality (CPSQ) in the US is $2.08 trillion — $1.56 trillion from operational failures and $520 billion from legacy systems. We want software systems to be reliable; testing is how, in most cases, we find out if they are.
CoC: Prevention (quality planning, tools, training) + Appraisal (testing, inspection).
CoNC: Internal failures (rework) + External failures (liability, loss of property, loss of lives).
Testing adds to CoC — but it must directly reduce CoNC, or it wasn't worth doing.
A defect caught early is cheap. Caught late, it's 16× the price.
"Your code is complete. It compiles. It runs... your program fails. How can this be?" There's a defect in the code; when executed, it causes bad behavior that later becomes visible as a failure. Before a program can be debugged, it must be set up so it can be tested — executed with the intent to make it fail. The first step in debugging is to reproduce the problem: first to bring it under control so it can be observed, second to later verify the fix actually worked.
Testing has an important role in all stages of a product's life cycle: Planning, Development, Maintenance, Operations.
Creates confidence in the software's quality, if properly tested with a minimum of defects found.
Teaches lessons for future projects — understanding root causes lets you improve process and prevent recurrence.
A systematic, destructive search for defects
"Are we meeting the requirements?" is the plain-language version. The formal version is more precise — and more testable on an exam.
Definition
know this verbatimSoftware testing is a systematic process used to identify the correctness, completeness, and quality of developed software. It includes a set of activities conducted with the intent of finding errors so they can be corrected before the product reaches end users. In simple words: testing is an activity that ensures the software system is defect free. It can be done either manually or with automated tools.
The process of testing all SW life-cycle activities — both static and dynamic — concerned with planning, preparation, and evaluation of software products and related work products, to: determine they satisfy specified requirements, demonstrate they are fit for purpose, and detect defects.
Confirming requirements are met · Causing as many failures as possible · Checking no defects were introduced during changes · Assessing quality (with no intention of finding defects) · Finding defects (reduces the probability of undiscovered ones) · Creating confidence in the level of quality · Providing information for decision-making · Preventing defects.
Four myths, busted with the exact fact from the slide
Myth #1: BUSTED
"Testing shows there are no errors." Fact: the main objective is to discover defects. Testing is a destructive activity — you're trying to break the system, not prove it's fine.
Myth #2: BUSTED
"Testing ensures the software does what it's supposed to." Only partly true — testing must also ensure the software does not do what it's not supposed to do.
Myth #3: BUSTED
"Testing is easier than design/implementation." Fact: you must consider every possible scenario, implied and unstated requirements and threats, and be imaginative and creative.
Myth #4: CONFIRMED
"Testing is an extremely creative and intellectually challenging task." This one's actually true.
The exact exercise the deck uses to prove Myth #3 wrong
A program reads 3 integers representing triangle side lengths and prints whether it's Equilateral (all 3 equal), Isosceles (exactly 2 equal), or Scalene (all different). Write test cases that would adequately test it. How many are actually needed?
| Category | Sample Input | Expected Result |
|---|---|---|
| Enter no / 1 / 2 values | nil · 12 · 12, 13 | Error — 3 integers required |
| Happy path | 5,5,5 / 3,3,4 / 2,3,4 | Equilateral / Isosceles / Scalene |
| Wrong format / non-integers / spaces / other chars | 2.5, 7.5, 8.7 · a, %, Z | Error — 3 integers required |
| Boundary — at least one is 0 | 0, 0, 0 | Error — all values must be > 0 |
| Boundary — at least one is > max | 27, 27, 27 | Error — all values must be ≤ max |
| Invalid triangle (fails triangle inequality) | 2, 2, 5 · 2, 3, 25 | Error — not a triangle |
- Smoke test ("Hello World!"): test major functions first — if there are bugs, it's an early indication that everything else has bugs too.
- Check handling of inputs: illegal inputs (text instead of integers), impossible inputs (floating vs. integer), outrageous values (infinities, max/min), out-of-domain values (negatives, outside spec), input errors (mis-spellings).
- Stress test: multiple inputs without restart, or run the program for long periods of time.
Error, defect, failure — three words that are NOT synonyms
This is the single most exam-tested distinction in the whole chapter.
A human action
Human actions that result in a fault or defect in the system. Concerned with the underlying cause of the defect. Can occur in design, coding, requirements — even testing itself.
A flaw sitting in the system
Flaws in a system that can cause the system to fail to perform its required function — e.g. an incorrect condition or statement. Concerned with specific parts/components of the system. Can occur in requirements, design, or program code.
Observed bad behavior
Deviation of the observed behavior of a system from its specification (its expected behavior). Can only be determined with respect to the specification. If a defect in code is executed, a failure may occur — but not all defects cause failures all the time.
- "Is bug the same as defect? — Yes. But failure is different."
- "Bugs and faults ≠ failures... but a bug/fault might cause a failure."
square(int x)
return x*2; itself — it
sits there whether or not you ever call the function.square(3) → returns 6, expected 9. A
failure — the defect executed and produced observably wrong behavior.square(2) → returns 4, which is also what 2×2
equals. Not a failure — correct result by coincidence, even though the exact same
defect is still sitting in the code, unexecuted-in-a-way-that-shows.
The vocabulary you need before you can write a real test suite
Eight terms, chunked by what role they play — write it, run it, or judge it.
| Group | Term | What it actually is |
|---|---|---|
| ✍️ What you write | Test Case | An execution of the software with a given test input — includes input values, sometimes execution steps, and expected outputs. Consists of inputs, steps/actions, and a pass/fail criterion. |
| Test Oracle | The expected outputs of software for a given input — a part of every test case. Generating oracles automatically is considered the hardest problem in auto-testing. | |
| 🔧 What you build to run it | Test Scaffolding | Additional code needed to execute a unit or subsystem in isolation — replacements for parts of the program you're omitting from the test. Not useful in production code; needs to be removed. |
| Test Fixture | A fixed state of the software under test used as a baseline for running tests (aka the "test context") — e.g. loading a database with a known dataset, or preparing input data and mock/fake objects. | |
| Test Driver | A software framework that loads a collection of test cases or a test suite; can also handle configuration and comparison between expected and actual outputs. | |
| 📦 How it's organized | Test Suite | A collection of test cases, usually sharing similar prerequisites and configuration, usually run together in sequence. Different suites exist for different purposes (certain platforms, certain features, performance...). |
| Test Script | A script that runs a sequence of test cases or a whole test suite automatically. | |
| 🛑 When to stop | Test Adequacy | We can't use every possible input — so which do we use, and when do we stop? An adequacy criterion is a rule that lets us judge the sufficiency of a set of test data. Example: test coverage (statement coverage, branch coverage...) — a measurement of what % of the code was exercised, e.g. via the Cobertura tool. |
Pass: execution completed, and the function performed as expected.
Fail: execution completed, but the function did not perform as expected.
Error: execution was not completed at all, due to an unexpected event, exception, or improper setup of the test case.
Here, 3 is the test
oracle (the expected output) baked directly into the assertion.
Four zoom levels, and testing's place inside quality assurance
Two things testing is explicitly NOT
The purpose of testing is to find defects — to discover every conceivable weakness in a software product. The deck highlights this in a bright yellow callout box, verbatim:
2. Software testing ≠ Quality assurance.
Software testing is a planned process used to identify the correctness, completeness, security, and quality of software. Quality Assurance (QA) is a planned and systematic way to evaluate the quality of the process used to produce a quality product. QA's goal is to assure a product meets the customer's quality expectations.
P1 through P7 — starred by the instructor as "memorize? understand all"
This is the one slide the instructor personally flagged with a star. Given the midterm note up top ("listing capped, but understand everything"), these seven are the highest-yield thing in the whole chapter.
P1 — Presence of defects
Testing can show defects are present, but cannot prove there are none. It reduces the probability of undiscovered defects — but even zero found is not proof of correctness.
P2 — Exhaustive testing is impossible
Testing every combination of input and precondition isn't feasible except in specific cases. We use risks and priorities to focus test effort.
P3 — Early testing
Testing activities should start as early as possible in the development life cycle, and should be focused on defined objectives.
P4 — Defect clustering
A small number of modules contain most of the defects discovered during pre-release testing.
P5 — Pesticide paradox
If the same set of tests is repeated over and over, it will stop finding new bugs — the defects it can catch are already gone, just like pests develop resistance.
P6 — Context dependent
Testing is done differently in different contexts — safety-critical software is tested very differently from an e-commerce site.
P7 — Absence-of-errors fallacy
Finding and fixing defects doesn't help if the software system is unusable or doesn't meet the user's expectations.
Seven steps from "who/what/why/when/where" to a closed test cycle
Test suite: a group of test cases, usually organized around some principle (e.g. a smoke test suite).
Test plan: a document that specifies how a system will be tested, including criteria for success — the exit criteria.
Testing process: developing test plans, designing test cases, running the test cases, and evaluating the results.
1–2. Planning, monitoring & control
who / what / why / when / whereA plan encompasses what, how, when, and by whom: scope, objectives, and risk analyses; the test levels and types to be applied; documentation to be produced; resources assigned to each test activity; and the schedule for implementation, execution, and evaluation. Control and adjust the plan to reflect new information and new project challenges.
3–4. Analysis & design
objectives → conditions → casesFirst, review the test basis: requirements, product architecture, product design, interfaces, and the risk analysis report.
General test objectives are transformed into concrete test conditions.
Produces test cases, test environments, test data, and creates traceability back to requirements.
5–6. Implementation & execution
build it, then run itGroup tests into scripts, prioritize the scripts, prepare test oracles, write automated test scenarios.
Run the tests and compare results with oracles, report incidents, repeat test activities for each corrected discrepancy, and log the outcome of test execution.
7. Test completion
close the loopAssess test execution against the defined objectives. Check if more tests are needed, or if exit criteria should change.
Report: write or extract a test summary report for stakeholders. Test closure activities: the activities that make test assets available for later reuse.
Testers need a different mindset than developers
A certain degree of independence is often more effective at finding defects and failures. However, a developer can also very efficiently find bugs in their own code. The right level of independence depends on the objective of testing.
Tests designed by the same person who wrote the code
Designed by another person, same team, same organization
Designed by a person from a separate testing team, same organization
Designed by a person from an outside organization / company (outsourced)
What testing can never promise you
Testing vs. Verification vs. Debugging
three different jobsBoth testing and verification attempt to exhibit new failures. Debugging is a separate, systematic process that finds and eliminates the defect that led to an observed failure. Programs without any known failures may still contain defects — either because they haven't been verified yet, or because they have been verified but the failure in question simply isn't covered by the specification.
Consider a simple program with a loop bounded at 20 iterations and a handful of branches: it has 10¹⁴ possible paths. At one test per millisecond, testing every path would take 3,170 years. Exhaustive testing is impossible — full stop.
Testing reduces the probability of undiscovered defects remaining in the software, but even if no defects are found, it is not proof of the absence of defects.
It depends on the level of risk (technical risks, business risks) and on project constraints (time, budget). It's impossible to test everything — testing should provide sufficient information for stakeholders to make informed decisions about releasing the software or planning next development steps.
One of the hardest problems in testing is not knowing when to stop. You can use metrics as a guess — keep track of the defect rate, and when it trends toward zero, treat that as an indicator. But even if you do find the very last bug, you'll never actually know it was the last one.
The wrong answers this chapter's exam questions are designed to catch
Every one of these is a distinction the deck explicitly draws — which means it's fair game to mix up under exam pressure.
1. "A defect is the same thing as a failure"
square(2) proves the same defect can run without failing.2. "Testing and debugging are basically the same activity"
3. "Testing and Quality Assurance are the same thing"
4. "If testing finds zero defects, the software is bug-free"
5. Mixing up test verdicts: Pass, Fail, and Error
6. "A great regression suite keeps finding new bugs forever"
7. Confusing P1 (Presence of Defects) with P7 (Absence-of-Errors Fallacy)
8. "Test scaffolding should ship with the production code"
Click a question to reveal the answer
Separate from the Practice Lab below — these are quick recall checks, not full exam-style questions.
convert_double_to_int) throwing an exception because a reused Ariane 4 component's precondition (abs(BH) < 32768.0) no longer held for the more powerful Ariane 5. Root cause: reuse of a component with a hidden constraint, combined with an integration-testing gap.The whole chapter, one scannable table
| Concept | One-line definition |
|---|---|
| Error (mistake) | A human action that leads to a defect. |
| Defect (fault, bug) | A flaw in the system that CAN cause it to fail — a static property of the code. |
| Failure | Observed deviation from the specification — only occurs if a defect is executed. |
| Test case | Inputs + steps/actions + expected results (pass/fail criterion). |
| Test oracle | The expected output for a given input. |
| Test scaffolding | Temporary code that lets you run a unit in isolation; removed before production. |
| Test fixture | A known baseline state for the software under test (data, mocks, DB setup). |
| Test suite / script / driver | A grouped collection of test cases / a script that runs them automatically / the framework that loads and evaluates them. |
| Test adequacy | A rule for judging when a set of test data is "enough" (e.g. code coverage %). |
| Unit → Integration → System → Acceptance | Single module → interactions between modules → whole system by developers → validated by customers against requirements. |
| Testing vs. QA vs. Debugging | Testing finds defects in the product · QA improves the process that builds the product · Debugging fixes the defect that caused an observed failure. |
| P1–P7 | Presence-not-absence · Exhaustive-impossible · Early · Clustering · Pesticide paradox · Context-dependent · Absence-of-errors fallacy. |
| 7-step test process | Planning → Monitoring & control → Analysis → Design → Implementation → Execution → Completion. |
| Verdicts | Pass (completed, correct) · Fail (completed, incorrect) · Error (didn't complete). |
Apply the vocabulary to concrete scenarios
Same shape of reasoning the instructor's own examples (square(), Ariane 5, the triangle program) use — worth drilling until automatic.
int square(int x) { return x*2; } (should be
x*x): is the incorrect line itself a defect, a failure, both, or neither?
What about calling square(3)?square() function — is calling
square(2) (which returns 4) a failure?2*2 also equals 4, so the wrong formula happens to produce
the correct result for this specific input. The defect is still there — it's just not
"showing" this time. This is exactly why P1 says testing shows presence, not absence, of
defects: you can run many inputs and see no failures while a defect still lurks.0, 0, 0, and the program correctly prints "Error: all values must be >
0." What test-design idea does this input specifically target?