Metrics·& Tools
DARK
SE401 · Software Quality Assurance and Testing · Chapter 10

You can't fix
what you don't measure.

Testing produces two kinds of output: bugs, and numbers about the bugs. This chapter is about the numbers — the base and calculated metrics that turn "I think testing went okay" into "87% requirement coverage, 4.2% critical defect rate" — and the tools that collect, generate, and report those numbers so nobody has to do it by hand.

12 base metrics
5 calculated-metric families
6 testing-tool categories
Interview-ready Q&A
TEST METRICS coverage · effort · effectiveness quality · efficiency
§ 01 — FOUNDATIONS

What is a test metric, and why bother?

Get the vocabulary locked in first — everything else in this chapter builds on it.

📏
Memory hook
"We cannot improve what we cannot measure."
A metric is a "standard of measurement" — a unit for describing an attribute. Test metrics are the means by which software quality can be measured: they give visibility into how ready the product is, and how complete and how good the testing was.

Software testing metrics (a.k.a. software test measurement) are the quantitative indication of extent, capacity, dimension, amount, or size of some attribute of a process or product — e.g. the total number of defects. The goal: improve the efficiency and effectiveness of the testing process, and support better decisions with reliable data.

Why we need metrics
You cannot improve what you cannot measure. You cannot control what you cannot measure.

Without measurement it's impossible to tell whether a process is improving. Metrics help you decide the next phase of activities, understand what kind of improvement is needed, and decide on process or technology change.
Questions test metrics answer
How long/how much will testing take? How bad are the bugs? How many bugs found were fixed, reopened, closed, deferred? How many bugs did the test team miss? How much of the software was tested? Will testing finish on time? Were we running low-value test cases? Could we have fit more testing into this release?

Three types of testing metrics

process · product · project
⚙️

Process Metrics

Used to improve the process efficiency of the SDLC (Software Development Life Cycle).

📦

Product Metrics

Deals with the quality of the software product itself.

👥

Project Metrics

Measures the efficiency of a project team, or of any testing tools the team is using.

Identifying the right metrics — a 4-step checklist: (1) fix the target audience for the metric, (2) define the goal of the metric, (3) introduce all relevant metrics based on project needs, (4) analyze the cost-benefit of each metric and which project-lifecycle phase it gives the maximum output in.
§ 02 — MANUAL TEST METRICS

Base Metrics: the raw data everything else is built from

Manual test metrics split into two classes — get this split right before the formulas, because every "calculated metric" later is just base metrics plugged into a ratio.

🧱
Memory hook
"Base is what you count. Calculated is what you compute."
Base metrics = raw data collected by the Test Analyst during test case development and execution (e.g. # of test cases executed, # of test cases). Calculated metrics = derived from base-metric data, usually followed by the test manager for reporting (e.g. % Complete, % Test Coverage).
Base Metrics

Raw, collected counts

Gathered directly during testing — nobody has computed anything yet. A great starting point that later gets used to produce calculated metrics.

Calculated Metrics

Ratios and percentages, computed from base metrics

What test managers actually report upward: coverage %, effectiveness %, pass/fail %, and more — every one of them is arithmetic on the base metrics to the left.

The 12 Major Base Metrics

raw counts
#Base Metric#Base Metric
1Total number of test cases7Number of defects rejected
2Number of test cases passed8Number of defects deferred
3Number of test cases failed9Number of critical defects
4Number of test cases blocked10Number of planned test hours
5Number of defects found11Number of actual test hours
6Number of defects accepted12Number of bugs found after shipping
§ 03 — CALCULATED METRICS

Five formula families — and what each one is really asking

Every calculated metric answers one plain-English question. Learn the question first; the formula follows naturally.

🧠
Exam tip, straight from the slide annotations
"Know what you're calculating — and know what the number you got represents."
Exams don't just ask you to plug numbers into a formula; they ask you to say what a 92% vs. a 40% means. Always finish a calculation by stating whether higher is better and why.

1 · Test Coverage Metrics — "How much of the application was tested?"

1.1
Test Execution Coverage %
Number of tests runTotal number of tests to be run
× 100
Total tests executed compared to total tests planned to run. Presented as a percentage.
1.2
Requirements Coverage
Number of requirements coveredTotal number of requirements
× 100
The percentage of requirements that have been covered by testing, compared to total requirements.

2 · Test Effectiveness Metrics — "How good were the tests? Are we running high-value test cases?"

2.1
Test Effectiveness %
Bugs found in TestTotal bugs found (in test + after shipping)
× 100
A measure of the bug-finding ability and quality of a test set — not the same thing as "efficiency."
Example: 80% test effectiveness → 20% of defects got away from the test team.
The higher the test effectiveness %, the better the test set — and the lesser the test case maintenance effort will be in the long term.

3 · Test Effort Metrics — "How long, how many, how much?"

Great for establishing baselines for future test planning.

3.1
Number of tests run per time period
Number of tests runTotal time
E.g. 30 tests per one day. Watch the unit given, and convert if needed.
3.2
Bug find rate (defects per test hour)
Total number of defectsTotal number of test hours
The rate of finding defects, shown as detected defects per test hour.
3.3
Number of bugs per test
Total number of defectsTotal number of tests
An estimate for the number of defects found in one test.
3.4
Average time to test a bug fix
Total time between defect fix & retest, all defectsTotal number of defects
Worked example: D1 = 1 day, D2 = 1 day, D3 = 2 days → total = 4 days over 3 defects = 4 / 3 ≈ 1.3 days per defect on average.

4 · Test Tracking & Quality Metrics — "What does the application's quality look like?"

4.1
Passed Test Cases %
Number of Passed TestsTotal number of tests executed
× 100
Indicates the quality of the tested application.
4.2
Failed Test Cases %
Number of Failed TestsTotal number of tests executed
× 100
Also gives an indication of the effectiveness of the conducted tests.
4.3
Critical Defects %
Critical DefectsTotal defects reported
× 100
Tracks how severe the found defects are, relative to all reported defects.
4.4
Fixed Defects %
Defects FixedDefects Reported
× 100
Gives an indication of the efficiency of testing/fixing.

5 · Test Efficiency Metrics — "How efficient is repairing defects?"

5.1
Average time to repair defects
Total time taken for bug fixesNumber of bugs
The total time to fix all bugs, divided by the number of bugs — an indicator of how efficient the repair process is.
§ 04 — COVERAGE & QUALITY METRICS

Two extra angles: code coverage, and overall product quality

Test Coverage Metrics (expanded)

how much of the code / features
Code Coverage — "How much of the code is being exercised?"
Segment coverage (% of segments hit): every executable statement is in some segment; a segment corresponds to an edge in the program's directed graph. Especially useful during unit and integration testing, and it's cumulative. A goal of 85% is a practical value.

Call-pair coverage (% of call pairs hit): an interface whereby one module invokes another. A goal of 100% is a practical value.
Requirements Coverage — "Are all the product's features being tested?"

The percentage of requirements that are covered by at least one test.

Cumulative Test Time: the total amount of time actually spent testing the product, measured in test hours. Provides an indication of product quality, and feeds into computing software reliability growth — the improvement in reliability that results from correcting faults.

Quality Metrics

defect removal & detection
Defect removal percentage
"What percentage of known defects is fixed at release?"
[Bugs fixed prior to release / Known bugs prior to release] × 100
Defects reported in each baseline

Used to help decide on process improvements, additional regression testing, and the ultimate release of the software.

Defect detection efficiency
"How well are we performing testing?"
[Unique defects we find / (Unique defects we find + Unique defects reported by customers)] × 100

Used to help decide on release of the final product, and how closely your testing resembles actual customer use.

§ 05 — TESTING TOOLS

Six families of tool support for testing

A test tool can be used for one or more activities that support testing: used directly in testing, used to manage the testing process, used in exploration, or "any tool that aids in testing."

🗺️
Memory hook
"My Team Executes, Performs, Specifies, Statically"
Management · Test specification · Execution & logging · Performance & monitoring · Specific application areas · Static testing — the six hexagons around "Tool Support for Testing."

🗂️ Management

  • Test
  • Requirement
  • Incident
  • Configuration

📝 Test specification

  • Test design
  • Test data preparation

⚡ Performance & monitoring

  • Dynamic analysis
  • Monitoring
  • Performance / load / stress

🎯 Specific application areas

  • Localization, security, usability…
  • Tools tied to a needs area

▶️ Execution & logging

  • Execution
  • Unit testing
  • Comparators
  • Coverage
  • Security

🔍 Static testing

  • Review
  • Static analysis
  • Modelling
Purposes a tool can serve
Improve efficiency (automate repetitive tasks) · automate activities requiring significant manual resources (e.g. static testing) · automate activities that cannot be done manually (e.g. large-scale performance testing) · increase reliability (automating large data comparisons or simulating complex behavior).
Classification, the probe effect & "(D)"
Tools are classified by the testing activity they support — one activity, or several (falling under the main activity). Some tools are intrusive — the tool itself can affect the test outcome (e.g. timing measurements differ depending on the performance tool used). This is called the probe effect. Tools marked "(D)" in this chapter offer support more appropriate for developers.
§ 06 — MANAGEMENT TOOLS

Managing the testing process itself

Management of testing & tests

traceability · reporting · metrics

Characteristics: support for managing tests and testing activities; support for traceability of tests, test results, and incidents back to source documents like requirements specifications; generation of progress reports; logging test results (monitoring); offering info on metrics related to the tests.

The tool family
Test mgmt / ALM Requirements mgmt Defect mgmt Configuration mgmt Continuous integration (D)
What each does
ALM: application lifecycle management, umbrella for test management. Requirements management tools: store requirements, check consistency and find undefined (missing) requirements, allow prioritization, enable individual tests to be traceable to requirements. Configuration management tools: necessary to track different versions/builds of software and tests — useful when developing on more than one HW/SW configuration.
§ 07 — STATIC TESTING TOOLS

Improving the work product without executing it

Review, Static Analysis & Modeling Tools

code untouched, quality up

Static testing tools aid in improving the code or work product without executing it. Three categories: Review Process Tools, Static Analysis Tools, Modeling Tools.

Review process tools

A common reference for review processes, keeping track of all information from the review, storing/communicating review comments, and monitoring review status (passed / passed with corrections / requires re-review). Best suited to formal review processes and geographically dispersed teams.

Static analysis tools (D)

Mostly used by developers for component (unit) testing — the tool is executed, the code under review is not. Source code is input data. An extension of compiler technology: Source Code → Code Transition → Transformed Code → Code Analysis → Results. Purpose: better understand the code and find ways to improve it. Common features: calculate metrics (complexity, nesting levels → identify risk areas), enforce coding standards, analyze code structures and dependencies.

Real tool examples (D)
SpotBugs — looks for bugs in Java code; checks for 400+ bug patterns.

SourceMonitor — collects metrics from source code files; displays and prints them in tables and charts (including Kiviat diagrams).

JDepend — generates design-quality metrics for each Java package; measures design quality in terms of extensibility, reusability, and maintainability.
Modeling tools (D)

Validate models of the system/software to better aid design: identify inconsistencies and defects within models, identify and prioritize risk areas, predict system response and behavior under various situations. Example: StarUML — a UML tool covering Class/Domain, Use Case, and Sequence diagrams.

Why this pays off: the major benefit of static testing and modeling tools is the cost-effectiveness of finding more defects earlier in development — the process accelerates and improves because there's less rework.
§ 08 — TEST DESIGN & SPECIFICATION TOOLS

Generating the inputs — and the tests themselves

Design, Model-Based & Data-Prep Tools

TDD · ATDD · BDD
Test design tools

Generate test input values from requirements, test conditions, design models (state, data, or object), code, or graphical user interfaces — and generate expected results too, if an oracle is available to the tool.

Model-based testing

Generates test input values or test cases from stored information describing a model of the system, e.g. a state transition model.

Test data preparation tools

Generate an extensive range or volume of data when needed, and manipulate databases or files to set up test data used during test execution.

TDD (D)
Part of Extreme Programming; used in Agile development.
ATDD & BDD
Acceptance test-driven development (ATDD) and behavior-driven development (BDD) use natural-language syntax:

Given <some condition>, When <something is done>, Then <result should happen>

As a <role, e.g. customer>
In order to <achieve something, e.g. product>
I want <do something>
🎬
Chunking aid — GWT structure
Given / When / Then = Setup / Action / Outcome
Read Given–When–Then like a mini story: Given sets the scene, When is the thing the user does, Then is what must be true afterward. It's the same three-beat structure whether you're writing ATDD or BDD scenarios.
§ 09 — TEST EXECUTION & LOGGING TOOLS

Running the tests, and proving the result was checked

Execution, Coverage, Harnesses & Comparators

drivers · stubs · comparators
Test execution tools

Enable tests to be executed automatically using stored inputs and expected results.

Coverage tools

Identify coverage items (instrumenting the code), calculate the % of coverage items exercised by a test suite, report items not yet exercised, identify test inputs to exercise as-yet-uncovered items, and generate stubs and drivers if part of a unit test framework.

Test comparators

Used when an executed test generates a lot of output — testing is more than providing inputs, you also need to check the software produces the correct result by comparing actual outcomes to expected results. Two ways to compare: dynamic comparison (done during test execution) vs. post-execution comparison (performed after the test finished, when the software under test is no longer running).

Test harnesses (D) & unit test framework tools (D)

Similar to each other — both support testing individual components/units. Both: supply inputs to the software being tested, receive outputs generated by it, execute a set of tests, record pass/fail results, store tests, measure coverage at code level, and support debugging. A harness uses stubs and drivers (small programs that interact with the software); unit test framework tools add support for object-oriented software. Used during test execution and logging.

Driver: calls the component being tested — "a component that calls the Tested Unit."

Stub: called from the component being tested — "a component the Tested Unit depends on." A partial implementation, returning fake values.
§ 10 — PERFORMANCE & MONITORING TOOLS

Watching the system under realistic load

Load Data, Monitoring & Dynamic Analysis

real · load · maintenance
Test data for performance testing
Real: test data obtained from actual users.
Load: large amounts of test data can be produced.
Maintenance: test data from the production environment.

Tests should reflect realistic scenarios — systems are often required to handle significant load and interactions, and inadequate/insufficient testing compromises system quality. Setting up this test data is significant effort (extensive range/volume needed, resource consuming to create), which is exactly what test data preparation tools help manage.

Monitoring tools

Identify problems and send an alert message to the administrator (e.g. network administrator): logging real-time and historical information, finding optimal settings, monitoring the number of users on a network, monitoring network traffic (real time, or over a period with analysis done afterward). Example: Windows Performance Monitor.

Dynamic analysis tools (D)

Features/characteristics include support for detecting memory leaks, identifying pointer arithmetic errors such as null pointers, and identifying time dependencies. Example: Windows Task Manager (process/memory/CPU inspection).

§ 11 — SPECIFIC APPLICATION AREAS

Tools built for one job, and "hidden" testing tools

Niche tools & the tools that don't look like testing tools

specialized
Specialized tool areas
Data quality assessment Data conversion & migration Usability & accessibility Localization Security Portability
Examples: performance testing tools built specifically for web-based applications; dynamic analysis tools built specifically for testing security aspects. Targeted-area example: embedded systems.
Tools you might not think of as "testing tools"

Testers may use a word processor or spreadsheet as a testing tool — often to store test designs, test scripts, and test data. Testers may also use SQL to set up and query databases containing test data. Tools developers use for debugging — to localize defects and check fixes — are also testing tools. It's worth looking at any tool available to you for ways it could support testing activities.

§ 12 — EFFECTIVE USE, BENEFITS & RISKS

Tools are not magic

Simply purchasing or leasing a tool does not guarantee success with it — each type of tool requires additional effort to achieve real, lasting benefit.

Potential benefits
Reduce repetitive work — running regression tests, re-entering the same test data, etc.

Greater consistency & repeatability — people do the same task a little differently each time (distractions, multitasking, interruptions, fatigue, external pressure); a tool reproduces the exact same procedure every time.

Objective assessment — humans are prone to error and subjective, preconceived bias toward verification; tools give repeatable, consistently calculated results (cyclomatic complexity, nesting levels, coverage, system behavior, incident statistics).

Ease of access to information — charts and graphs are easier for the human mind to parse than a long list of numbers; special-purpose tools provide statistics, incident rates, and performance data directly.
Potential risks
Unrealistic expectations for the tool's functionality and ease of use.

Underestimating the time, cost, and effort to introduce a tool (training, external expertise).

Underestimating the time/effort needed to achieve significant, continuing benefit.

Underestimating the effort required to maintain the test assets the tool generates.

Over-reliance on the tool — replacing testing where manual testing would actually be better.
Division of labor: the tester concentrates on what should be tested, what the test cases should be, and how to prioritize testing. The tool user concentrates on how best to get the tool to do its job effectively, and how to get increasing benefit from tool use.

Special considerations by tool type

▶️

Test execution tools

Capture/replay (recording a manual tester) seems attractive but doesn't scale and can be unstable. Data-driven: separates test inputs from a generic script. Keyword-driven: a spreadsheet holds keywords/action words plus test data; testers define tests using the keywords.

⚡

Performance testing tools

Requires care around the design of the load to be generated, timing aspects / the probe effect, and how to interpret the gathered information. Needs a tester with performance testing expertise.

🔍

Static analysis tools

Risk that changes made to conform old code to a new standard introduce unexpected side effects; applying the tool to existing code may generate many messages. A gradual rollout with initial filters is more effective.

🗂️

Test management tools

Need to interface with other tools or spreadsheets, to produce information in the best format for the organization's current needs.

§ 13 — QA, QC & CORE DEFINITIONS

Interview-question territory

These definitions get asked constantly in interviews and on exams — precise wording matters.

🎯
Fundamental questions in testing
Stop? Test? Correct? How well? Who?
"When can we stop testing?" → test coverage. "What should we test?" → test generation. "Is the observed output correct?" → test oracle. "How well did we do?" → test efficiency. "Who should test your program?" → independent V&V.
QA vs. QC vs. Software Testing
TermWhat it is
Quality Assurance (QA)The planned, systematic way of monitoring the quality of the process used to produce a quality product. QA tracks outcomes and adjusts the process. QA is not just testing.
Quality Control (QC)Concerned with the quality of the product. Finds defects and suggests improvements. The process QA sets is implemented by QC — QC is the tester's responsibility.
Software TestingThe process of ensuring a developed product meets user requirements — the motive is to find bugs and get them fixed.
Verification vs. Validation
Verification: evaluating the work-products of a development phase to determine whether they meet the specified requirements for that phase.

Validation: evaluating the software during or at the end of development to determine whether it meets the specified requirements.
Branch Coverage vs. Decision Coverage
Branch coverage: testing that ensures every branch of the software is executed at least once (uses the Control Flow Graph).

Decision coverage: testing that ensures every decision-taking statement is executed at least once.

Both exist so the tester knows no branch or decision statement will lead to failure.
Branch Coverage = Tested Decision Outcomes / Total Decision Outcomes
What is Static Analysis?

A collection of algorithms and techniques used to analyze source code in order to automatically find bugs — similar in spirit to compiler warnings, but taken further to find bugs traditionally found via run-time debugging/testing. Static analysis bug-finding tools have evolved from basic syntactic checkers to tools that find deep bugs by reasoning about the semantics of the code.

Defect Costs

rises exponentially by phase

The cost of fixing a defect rises exponentially by lifecycle phase — a defect caught in requirements is far cheaper to fix than the same defect caught after release.

But this is simplistic — the real questions are: when were the defects injected? Are all defects treated the same? Do we reduce costs more by getting better at fixing, or at preventing?
"QA" & Testing — the whole map in one glance
Testing "phases": Unit → Integration → System → User Acceptance Testing.
Testing types: Black-box, White-box.
Static vs. Dynamic Testing.
Automated Testing: has pros and cons.
Integration, 2 types: Top-down, Bottom-up.
QA & SQA definitions
Quality Assurance (QA): the process of verifying whether a product meets required specifications and customer expectations — process-driven, defines goals for design, development, and production; primary goal is tracking and resolving deficiencies prior to release. The QA concept was popularized during World War II.

Software Quality Assurance (SQA): an ongoing process within the SDLC that routinely checks developed software meets desired quality measures. Breaks down into: quality of planning, formal technical reviews, testing, and training.
§ 14 — PROFESSIONAL ETHICS

What it means to be a professional tester

⚖️
Memory hook
"If you can't test it, don't build it."
Put quality first — even if you lose the argument, you gain respect. Begin test activities early. Decouple: designs should be independent of language, programs independent of environment — couplings are weaknesses in the software. Don't take shortcuts: document your objections, vote with your feet, and don't be afraid to be right.

ACM & IEEE Code of Ethics for Software Engineers

8 principles
1PUBLIC — Software testers shall act consistently with the public interest.
2CLIENT AND EMPLOYER — shall act in a manner that is in the best interests of their client and employer, consistent with the public interest.
3PRODUCT — shall ensure that the deliverables they provide (on the products/systems they test) meet the highest professional standards possible.
4JUDGMENT — shall maintain integrity and independence in their professional judgment.
5MANAGEMENT — test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing.
6PROFESSION — shall advance the integrity and reputation of the profession, consistent with the public interest.
7COLLEAGUES — shall be fair to and supportive of colleagues, and promote cooperation with software developers.
8SELF — shall participate in lifelong learning regarding the practice of their profession, and shall promote an ethical approach to the practice of the profession.
§ 15 — COMMON MISTAKES

Where students actually lose marks

Every one of these is a real confusion this chapter sets up on purpose — drill them until the "wrong" option stops sounding reasonable.

Test Effectiveness vs. Test Efficiency

✗ WRONG"Effectiveness" and "efficiency" metrics are the same thing — both just mean "how good was testing."
✓ RIGHTTest Effectiveness asks "how good were the tests / bug-finding ability?" (Bugs found in test ÷ total bugs found). Test Efficiency is a specific different formula: average time to repair defects (Total time for bug fixes ÷ Number of bugs). One measures bug-catching power; the other measures repair speed.
The slide deck itself flags this with a handwritten note: "≠ efficiency" — it's a deliberately placed trap.

Base Metrics vs. Calculated Metrics

✗ WRONG"Number of defects found" and "% Test Coverage" are both examples of the same category of metric.
✓ RIGHT"Number of defects found" is a base metric — raw, collected data. "% Test Coverage" is a calculated metric — derived by computing a ratio from base-metric data.
If a metric is a plain count with no division/percentage sign, it's base. If it involves a ratio or percentage, it's calculated.

QA vs. QC vs. Testing

✗ WRONGQA is just another word for testing.
✓ RIGHTQA is process-focused (planned, systematic monitoring of the process to produce a quality product) — and explicitly "not just testing." QC is product-focused (finds defects, is the tester's responsibility). Software testing is the specific activity of finding bugs against user requirements.
Remember the direction: QA sets/monitors the process → QC (testing) executes within that process and reports back.

Branch Coverage vs. Decision Coverage

✗ WRONGBranch coverage and decision coverage are unrelated concepts measuring completely different things.
✓ RIGHTThey're closely related and often taught together: branch coverage ensures every branch (edge in the control-flow graph) executes at least once; decision coverage ensures every decision-taking statement executes at least once. Both exist for the same reason — proving no branch/decision leads to failure.
Know the formula cold: Branch Coverage = Tested Decision Outcomes / Total Decision Outcomes.

Driver vs. Stub

✗ WRONGA "driver" and a "stub" are interchangeable terms for the same helper component.
✓ RIGHTA driver calls the component being tested (a component that calls the Tested Unit). A stub is called from the component being tested (a component the Tested Unit depends on) and returns fake/partial values.
Picture the diagram: Driver → Tested Unit → Stub. The direction of the arrow tells you which is which.

Verification vs. Validation

✗ WRONGVerification and validation both mean "checking the final software works."
✓ RIGHTVerification checks a development phase's work-products against that phase's requirements ("did we build it right?"). Validation checks the software during/at the end of development against the overall specified requirements ("did we build the right thing?").
Verification is per-phase and internal; validation is end-to-end and requirements-facing.

Static Analysis Tools vs. Dynamic Analysis Tools

✗ WRONGStatic analysis tools run the program to find bugs, the same as dynamic analysis tools.
✓ RIGHTStatic analysis: the tool is executed, the code is not — source code is just input data (extension of compiler technology). Dynamic analysis tools (memory leaks, null pointers, time dependencies) require the program to actually run.
"Static" = without executing the code under review. Don't let the fact that the analysis tool itself runs confuse you.

Segment Coverage Goal vs. Call-Pair Coverage Goal

✗ WRONGBoth segment coverage and call-pair coverage target 100% as the practical goal.
✓ RIGHTSegment coverage's practical goal is 85%. Call-pair coverage's practical goal is 100% — they are different numbers for different reasons (interfaces where one module invokes another are expected to be fully exercised).
This is a plain memorization trap — the deck states both numbers explicitly, and exams love swapping them.
§ 16 — CHEAT SHEET

The whole chapter, one scannable grid

CategoryAnswers the questionKey formula / fact
Test Execution CoverageHow much did we run?Tests run / Tests to be run × 100
Requirements CoverageHow much did we cover?Requirements covered / Total requirements × 100
Test EffectivenessHow good were the tests?Bugs found in test / Total bugs found × 100
Test Effort — tests/timeHow fast are we running tests?Tests run / Total time
Test Effort — bug find rateHow fast are we finding bugs?Total defects / Total test hours
Test Effort — bugs per testHow buggy is each test?Total defects / Total tests
Test Effort — avg fix-retest timeHow long to verify a fix?Total fix-to-retest time / Total defects
Passed / Failed Test Cases %What's overall quality?Passed (or Failed) / Total executed × 100
Critical Defects %How severe are the bugs?Critical defects / Total defects reported × 100
Fixed Defects %How efficient is fixing?Defects fixed / Defects reported × 100
Test EfficiencyHow fast do we repair?Total repair time / Number of bugs
Defect Removal %What % of known bugs got fixed?Bugs fixed prior to release / Known bugs prior to release × 100
Defect Detection EfficiencyDid we catch bugs before customers did?Unique defects we find / (we find + customers report) × 100
Segment CoverageCode coverage (unit/integration)Practical goal: 85%
Call-Pair CoverageInterfaces between modules coveredPractical goal: 100%
Branch CoverageEvery branch executed?Tested decision outcomes / Total decision outcomes
Management toolsTraceability, reporting, metricsTest/ALM, requirements, defect, config mgmt, CI
Static testing toolsImprove code without running itReview, static analysis (D), modeling (D)
Design & spec toolsGenerate inputs / testsTest design, model-based, data prep, TDD, ATDD/BDD
Execution & logging toolsRun tests, log resultsExecution, coverage, harness/framework, comparators
Performance & monitoringLoad testing, watch the systemTest data (real/load/maintenance), monitoring, dynamic analysis (D)
QAProcess quality, not just testingPlanned, systematic; popularized WWII
QCProduct quality, finds defectsTester's responsibility
§ 17 — POP QUIZ

Click a question to reveal the answer

No page reload — just click to check yourself.

1. Is "Number of test cases blocked" a base metric or a calculated metric? Reveal
Base metric. It's a raw count collected directly by the Test Analyst during test execution — no ratio or percentage computed from it.
2. A test set finds 80 bugs during testing, and 20 more are reported after shipping. What's the Test Effectiveness %, and what does it mean? Reveal
80%. (80 / (80+20)) × 100 = 80%. It means 20% of the defects got away from the test team and reached customers — the same shape as the worked example in the deck.
3. Multiple choice: which coverage type has a practical goal of 100%, not 85%? (a) Segment coverage (b) Call-pair coverage (c) Requirements coverage Reveal
(b) Call-pair coverage. Segment coverage's practical goal is 85%; call-pair coverage (interfaces where one module invokes another) targets 100%.
4. Short answer: what's the difference between a driver and a stub? Reveal
A driver calls the component under test (it drives the Tested Unit). A stub is called FROM the component under test — it's a fake/partial implementation of something the Tested Unit depends on.
5. Multiple choice: which tool type is "an extension of compiler technology" where the tool executes but the code doesn't? (a) Dynamic analysis tool (b) Static analysis tool (c) Monitoring tool Reveal
(b) Static analysis tool. Source code is input data to the tool; the tool runs, the code under review does not.
6. Short answer: name the three "test data for performance testing" categories from the slides. Reveal
Real (data from actual users), Load (large volumes of test data), and Maintenance (data from the production environment).
7. True or false: QA is essentially the same thing as software testing. Reveal
False. The slides state this explicitly: "QA is not just testing." QA is the planned, systematic monitoring of the process used to produce a quality product; QC (which includes testing) is the tester's responsibility and is concerned with product quality.
8. Short answer: what is the "probe effect"? Reveal
The consequence of using an intrusive test tool — one where the tool itself affects the outcome of the test. Example given: timing measurements may come out different depending on which performance tool you use to measure them.
§ 18 — PRACTICE LAB

Worked, exam-style problems

Same shape as the calculations this chapter's slides worked through — drill these until the arithmetic and the interpretation are both automatic.

1. A team planned 250 test cases for a release and has executed 180 of them so far. What is the Test Execution Coverage percentage?
→ 72%
Why: Test Execution Coverage = (Number of tests run / Total number of tests to be run) × 100 = (180 / 250) × 100 = 72%.
2. Three defects were fixed and retested: D1 took 2 days between fix and retest, D2 took 1 day, D3 took 3 days. What's the average time to test a bug fix?
→ 2 days per defect
Why: Average = Total time between fix and retest / Total number of defects = (2 + 1 + 3) / 3 = 6 / 3 = 2 days — same structure as the deck's D1/D2/D3 example.
3. Of 400 defects reported this release, 36 were classified as critical. What is the Critical Defects percentage, and is a high or low value what you want?
→ 9%; lower is generally better
Why: Critical Defects % = (Critical Defects / Total defects reported) × 100 = (36 / 400) × 100 = 9%. Since this tracks the severity mix of reported bugs, a lower percentage means fewer of the found bugs are severe.
4. A code-review tool flags 400+ possible bug patterns in Java source but never actually runs the program. Which category of testing tool is this, and can you name an example from the slides?
→ Static analysis tool (D) — e.g. SpotBugs
Why: the tool is executed, the code is not — it's an extension of compiler technology used mostly by developers for component (unit) testing. SpotBugs is the named example that checks 400+ bug patterns in Java code.
5. A team scripted their tests by recording a manual tester's clicks. It broke the first time the UI changed slightly. What alternative approach does the chapter recommend, and why?
→ Data-driven or keyword-driven scripting
Why: capture/replay scripts don't scale to large numbers of automated tests and are unstable under unexpected events. A data-driven approach separates the test inputs from a generic, reusable script; a keyword-driven approach lets testers define tests from a spreadsheet of keywords/action words plus test data.