Working isn't the bar.
Quality is the bar.
A system that runs is not the same as a system that's good. This chapter uses one very expensive, very public disaster — the 2013 launch of HealthCare.gov — to show what happens when "we'll test it after we build it" meets 50,000 concurrent users. Then it gives you the vocabulary to actually measure quality: reliability metrics, the full spectrum of software qualities, the cost curve of a defect left too long, and the V-model of verification & validation that should have caught all of it.
Software has a bad track record — and it's not a coincidence
Before the "how to fix it," the deck opens with "how bad is it," because the numbers explain why this course exists.
break in a switch statement) · 1996
Ariane 5 rocket exploded 37s after liftoff · 2001 NYSE trading floor shut down over
an hour · 2017 British Airways disrupted 75,000 passengers · 2020 Heathrow
disruption, 100+ flights.| Type | Definition | 2012 |
|---|---|---|
| 1 — Successful | On-time, on-budget, all features/functions as specified | 39% |
| 2 — Challenged | Completed & operational, but over-budget, over-time, fewer features | 43% |
| 3 — Impaired | Canceled at some point during development | 18% |
The initial launch of HealthCare.gov (2013)
This is the deck's centerpiece: a real, government-scale system that failed publicly because of decisions any of the "quality" concepts below would have caught. Treat it as the worked example for the whole chapter.
ACA signed; development contracts awarded
Affordable Care Act signed into law. HealthCare.gov: a healthcare exchange website — "one-stop shopping for health insurance." CBO forecast: 7 million users in year one. Contracts awarded September 2011 as no-bid, cost-plus deals to pre-certified private contractors — lead contractor CGI Group, with at least 47 companies involved including QSSI, Equifax, Serco, coordinated by CMS (Centers for Medicare and Medicaid Services). Total budget: $293 million.
Evolving, rolling requirements
Regulations and policies were still in flux when contracts were signed and stayed in flux until summer 2013 — repeated delays to specs, repeated design changes. CGI didn't start coding until spring 2013. The team followed a "write-down-all-the-requirements-then-build-to-those-requirements" waterfall approach, never adopted agile, and committed to an all-or-nothing launch date anyway.
Inadequate testing, compressed to nothing
"This system just wasn't tested enough." — CMS. Full testing didn't begin until 2 weeks before launch. The final "pre-flight checklist" at T-1 week found 41 of 91 functions failing. No end-to-end test existed as late as T-4 days. Stress tests at T-1 day showed performance degrading with only 1,100 concurrent users — against an expected 50,000–60,000. Final top-to-bottom security tests were never finished. No integration test. No beta test.
The site goes live and breaks immediately
Landing-page response time exceeded 8 seconds ("maddeningly long wait times"). Broken navigation UI. Availability around 43%, intermittent crashes. Incorrect and incomplete data returned to users. Per-page error rate around 6%. System capacity capped under 1,100 concurrent users. Enrollment completion rate under 30%.
Ideal process vs. what actually happened
McKinsey's own diagram contrasted the two: the ideal large-scale program has clear requirements, minimal third-party dependency, sequential requirements→design→build→test with iteration between phases, full end-to-end testing, and a limited/gradual initial launch. HealthCare.gov's actual characteristics: evolving requirements, multiple definitions of success, heavy dependency on external contractors, all phases "stacked" in parallel instead of sequential, insufficient time and scope for end-to-end testing, and launch at full volume with no gradual ramp-up.
Contracts over delivery, staffing over substance
"Management expertise is in getting contracts, not delivering projects." Project quality was sacrificed for appearances. Three months before launch, only 10 developers were working on a crucial part of the site, and only one was "at a high enough skill level." Coordination among the 47+ contractors was poor — unclear responsibilities, fragmented authority. Underlying policy problem: "the firms that typically get contracts are the firms that are good at getting contracts, not typically good at executing on them" — a symptom of outdated, cost-plus, no-bid government IT procurement.
Turning "it broke" into a number
HealthCare.gov's postmortem is full of numbers — 8s response time, 43% availability, 6% error rate. These are the formal metrics behind those numbers.
| Term | Definition |
|---|---|
| Performance | Ability to complete requested functions/services within the time span users expect. E.g. average response time for a task. |
| Scalability | Capacity of a system to handle increasing load or demand. E.g. # of concurrent users, transactions/sec, requests/sec. |
Error rate vs. completion rate
not the same as successReliability depends on the unit of operation you measure — and an operation may consist of multiple steps, so reliability ≠ completion rate.
The fraction of pages (the unit of operation, here) that time out or fail. HealthCare.gov: ≈6% at launch, <1% by December.
The fraction of all attempted operations that eventually complete. Completion ≠ success — a user can finish an enrollment full of wrong data. HealthCare.gov: <30% at launch, ≈80% by December.
Quality isn't one thing — it's a whole checklist
"Quality" has competing definitions before it even gets to software: Crosby says zero defects, ISO says fitness to satisfy specified/implied needs, Juran says fitness for purpose, the dictionary just says "degree of excellence." Software breaks quality down into concrete, individually measurable attributes.
Correctness
Whether the system is consistent with its specification.
externalAvailability
Fraction of time the system is up and usable.
externalReliability
Probability the system operates without failure.
externalPerformance
Completes requested functions within expected time.
externalScalability
Handles increasing load or demand.
externalEfficiency
Makes maximum, efficient use of system resources.
internalSafety
Prevents certain undesirable behaviors — hazards.
externalUsability
Users can use all features without special effort.
externalSecurity
Maintains integrity of operation and data.
externalRobustness
Fails or degrades gracefully outside normal parameters.
externalMaintainability
Can be changed, enhanced, adapted, evolved over time.
internalReusability
Parts of the system can be reused in other projects.
internalPortability
Can be ported to a different platform or OS.
internalInteroperability
Works together with other systems.
internalCorrectness vs. Reliability
expected behaviorWhether a system is consistent with its specification. It's an all-or-nothing proposition — a program cannot be "mostly correct" or "30% correct." It is correct on all possible behaviors, or it is not correct. (Ironically, it's trivially easy to be "correct" against a very bad spec.) Correctness is a goal to aim for, but is rarely provably achieved.
Software is reliable if the user can
depend on it — formally, the probability the software operates as expected over a
specified time interval, relative to its specification and a usage profile.
Reliability is a statistical approximation to correctness:
100% reliable ≈ correct.
Safety vs. Robustness
exceptional behaviorThe ability of a software system to prevent certain undesirable behaviors, i.e. hazards.
The ability to fail or degrade gracefully outside normal operating parameters — acceptable (degraded) behavior under extreme conditions. Correctness and reliability assume normal operating conditions; software that's "correct" can still fail when those assumptions are violated — how it fails is what robustness measures.
How the four qualities overlap
Venn relationshipReliable and Robust are two overlapping circles; Correct sits inside Reliable, Safe sits inside Robust.
| Region | Meaning |
|---|---|
| Reliable, but not correct | Failures occur — just rarely. |
| Robust, but not safe | Catastrophic failures can still occur. |
| Correct, but not safe/robust | The specification itself is inadequate. |
| Safe, but not correct | Annoying (non-catastrophic) failures can occur. |
Conformance to customers' requirements
The working definition
from the slides"Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software." In the narrowest sense, quality is commonly recognized as the lack of "bugs" in the product — but too many functional defects means the basic requirement of providing the desired function isn't met either.
| Quality of design | Encompasses requirements, specifications, and the design of the system. |
|---|---|
| Quality of conformance | Focused primarily on implementation — does the build match the design? |
| Prevention costs | Quality planning, formal technical reviews, test equipment, training. |
|---|---|
| Internal failure costs | Rework, repair, failure mode analysis. |
| External failure costs | Complaint resolution, product return & replacement, help-line support, warranty work. |
The earlier you catch it, the cheaper it is
This is the number that justifies "test early" as more than a slogan.
Cost-to-correct multiplier table
time introduced × time detectedA defect's cost depends on both when it was introduced and when it was detected. Fixing it in the same phase it was introduced costs "1×" — but let it slip further downstream and the multiplier explodes.
| Introduced ↓ / Detected → | Spec | Design | Code | Test | Post-release |
|---|---|---|---|---|---|
| Spec | 1× | 3× | 5–10× | 15× | 30–100× |
| Design | — | 1× | 10× | 20× | 30–100× |
| Code | — | — | 1× | 10× | 20–50× |
| Introduced (%) | Total |
|---|---|
| Specification / design | 70% |
| Code / unit test | 20% |
| Integration / system test | 10% |
| Detected (%) | Total |
|---|---|
| Spec / design | 3.5% |
| Code / unit test | 16.5% |
| Integration / system test | 50.5% |
| Beta test | 9% |
| Post-release | 20.5% |
Two questions that sound similar and aren't
Does the system meet the requirements specification?
Checked internally against the spec document. Includes: testing (mostly), inspections, static analysis. Answers: "done correctly?" Test-level examples: unit test, integration test, automated testing.
Does the system meet the user's real needs?
Checked externally against actual requirements. Includes: usability testing, user feedback. Answers: "done the right thing?" Release-level examples: prototype release, customer release.
Actual Requirements → SW Specs →
System. Validation runs against actual requirements at the front; verification
runs against the SW specs feeding into the system. System test and
qualification measures sit in the overlap — both a verification and a validation
activity.
| Verification testing | Finds defects by executing the program in a test or simulated environment. E.g. functional test, integration test. |
|---|---|
| Validation testing | Finds defects by executing the program in a real environment or with real users. E.g. usability test, beta test. |
Testing is not a phase. It's a lifestyle.
The chapter's closing argument: quality cannot be bolted on at the end — it results from a set of inter-dependent activities integrated throughout the process.
The Quality Process
a framework, not a checklistA set of activities and responsibilities, focused primarily on ensuring adequate dependability, but also concerned with project schedule and product usability. It provides a framework for selecting and arranging activities and considering interactions/trade-offs with other important goals. For every development activity there is a corresponding testing activity, each level has objectives specific to that level, and test design should start as early as possible — applicable to both waterfall and agile development models.
The V-Model of Validation & Verification
what tests against whatDevelopment moves down the left side of the V (Specs → Subsystem Design → Unit Specs → Components); testing climbs back up the right side, each level checked against its matching left-side document, until the whole thing is checked against the actual needs and constraints that started it.
| Test level | Checked against | V&V |
|---|---|---|
| Unit Test | Unit/Component Specs | Verification |
| Integration Test | Subsystem Design/Specs | Verification |
| System Test | System Specifications | Verification |
| User Acceptance (Alpha, Beta) | Actual Needs and Constraints | Validation |
Testing of an individual software unit/module/component — a.k.a. module or component testing. Focus: functionality, correctness, accuracy of that one unit. Usually carried out by the developers of the unit. Basis: component specifications, detailed design and code.
Exposes defects in the interfaces and the interactions between integrated sub-systems — focus is on the seams between modules, not the modules themselves. Usually carried out by the developers of the sub-systems involved. Basis: system design/architecture, subsystem & interface specification.
Tests an integrated system to determine whether it meets the specification, in full — a.k.a. the end-to-end test. Verifies functional and non-functional requirements. Carried out by developers and independent testers. Basis: software requirement spec, functional spec. HealthCare.gov never ran this until 4 days before launch.
Used whenever a large amount of testing is needed and small changes can affect many parts of the system. Compiler-development example: collect examples that exercise each part → add new examples whenever a bug is found → run the whole collection and capture output → repeat after every code change → compare against the baseline.
Acceptance Testing
the user's verdictFormal testing with respect to user needs, requirements, and business processes, conducted to determine whether a system satisfies the acceptance criteria — the exit criteria a component or system must satisfy to be accepted by a user, customer, or other authorized entity. Focus: customer acceptance. Carried out by independent testers and the customers. Basis: system/user requirements, use cases, business processes, risk analysis.
Random test: test cases selected randomly, possibly via a pseudo-random number generation algorithm, to match an operation/usage profile — not the same thing as ad hoc testing.
| Alpha test | Simulated operational testing, performed by personnel acting as potential users/customers, carried out in a controlled environment, observed by the development organization. |
|---|---|
| Beta test | Operational testing by real users in their own environment, performing actual tasks without interference or close monitoring. One form of acceptance testing. |
The whole chapter, one table
| Concept | Definition / formula | Watch for |
|---|---|---|
| CHAOS outcomes | Success 39% / Challenged 43% / Impaired 18% (2012) | Impaired = canceled, not "all failures" |
| MTBF | MTBF = MTTF + MTTR | MTTF ≠ MTBF |
| Availability | up time / total time = MTTF/(MTTF+MTTR) × 100% | Not the same as reliability |
| Reliability | # successful ops / # attempted ops | ≠ completion rate |
| Correctness | Consistency with spec — all-or-nothing | Rarely provably achieved |
| Safety vs. Robustness | Prevent hazards vs. degrade gracefully | Robust ≠ safe |
| Verification | "Building the software right" — vs. spec | ≠ validation |
| Validation | "Building the right software" — vs. real needs | ≠ verification |
| Defect cost multiplier | 1× at origin phase → up to 100× post-release | Earlier is always cheaper |
| Cost by phase | 67% of lifecycle cost is Maintenance | Not "coding" |
| Unit test | vs. unit/component specs | One module only |
| Integration test | vs. subsystem design/interface spec | The seams, not the units |
| System test | vs. system specification; "end-to-end" | HealthCare.gov skipped this until T-4 days |
| Acceptance test | vs. actual needs; alpha/beta/random | HealthCare.gov: no beta test |
| Alpha test | Simulated users, controlled environment | ≠ beta test |
| Beta test | Real users, real environment, no interference | ≠ alpha test |
Where students actually lose points
Eight confusable pairs the deck sets up almost on purpose — expect at least one of these on an exam.
1. Verification vs. Validation
2. MTBF vs. MTTF
3. Correctness vs. Reliability
4. Robustness vs. Safety
5. Completion rate vs. Reliability/Success
6. Quality of Design vs. Quality of Conformance
7. Integration Testing vs. System Testing
8. Alpha Testing vs. Beta Testing
Click a question to reveal the answer
Self-check before the practice lab — no reload needed.
What was the single biggest software-engineering failure in the HealthCare.gov launch? Reveal
If MTTF = 480 hours and MTTR = 20 hours, what is MTBF, and what is availability? Reveal
Which best defines Validation? (a) "Are we building the software right?" (b) "Are we building the right software?" (c) Checking code against unit specs (d) Checking interfaces between modules Reveal
Per the cost-multiplier table, a defect introduced during spec but not caught until post-release costs how much more to fix than if caught in spec? Reveal
A team hires temp staff to pose as customers and test a new banking app in a controlled lab before release. This is: (a) Beta test (b) Alpha test (c) Regression test (d) Unit test Reveal
Name the three outcomes in the CHAOS Report classification, with their 2012 percentages. Reveal
Why can a system be "correct" and still fail catastrophically in production? Reveal
67% of total software lifecycle cost typically goes to: (a) Coding (b) Requirements (c) Maintenance (d) Unit testing Reveal
Apply it to a scenario
Exam-shaped questions grounded directly in the chapter's HealthCare.gov case study and the V&V / metrics material.