SE401 · Software Testing and Quality · Chapter 05

Quality isn't tested in.
It's built in, then verified.

Testing doesn't create quality — it measures it, like a scale measures weight without changing it. Real quality comes from good process, planning, and discipline before testing ever starts. This chapter walks through what quality actually means, how QA and QC split the work, why some projects succeed while others quietly implode, and how you turn all of that into numbers you can actually track.

QA vs. QC

Project success factors

Measurement & metrics

Exam-tested definitions

§ 01 — SOFTWARE QUALITY & TESTING

What "quality" actually means, and what testing does about it

Three ingredients define quality. Testing only checks whether you have them — it never adds them.

🧱

Memory hook

"Say it, Set it, Sense it."

Say it — explicit requirements are the foundation quality is measured against. Set it — specified standards define the development criteria. Sense it — implicit requirements are never written down, but a user still expects them (e.g. good maintainability).

1. Explicit requirements

The foundation quality is measured from

Software requirements are the foundation from which quality is measured. Lack of conformance to a requirement is lack of quality — full stop.

2. Specified standards

The rules that guide how it's built

Specified standards define a set of development criteria that guide the manner in which software is engineered. If the criteria aren't followed, lack of quality will almost surely result.

3. Implicit requirements

The ones nobody wrote down

There is a set of implicit requirements that often goes unmentioned (e.g. good maintainability). If software conforms to its explicit requirements but fails to meet implicit ones, software quality is suspect.

Software testing — the real purpose

The purpose of software testing is to assess and evaluate the quality of work performed at each step of the software development process.

It is NOT to use up all the remaining budget or schedule resources at the end of a development effort — even though it sometimes seems that way.

The goal of testing is to ensure the software performs as intended, and to improve software quality, reliability and maintainability.

Quality and testing — how they relate

"Quality cannot be tested into a product." A good development process, tools, methods, and people go far in providing quality products. Testing is one aspect of assuring software quality — it is a measure of quality, it does not deliver quality.

"Software testing is a full-life-cycle assessment of quality."

Software Quality Assurance includes:
· Software engineering process improvement — prevent the insertion of defects
· Fault tolerant software design — tolerate the existence of defects
· All aspects of software verification and validation — including testing

§ 02 — QUALITY ASSURANCE

QA: build the process that keeps defects out

QA is proactive and process-focused — it's the framework everything else operates inside of.

Goal of Quality Assurance

defect prevention

Quality assurance (QA) activities strive to ensure:

Few, if any, defects remain in the software system when it is delivered.

Remaining defects will cause minimal disruptions or damages.

Planning Quality

before the work starts

The following need to be considered when planning quality: Scope, Stakeholders, Risks, Internal and External Environmental Factors, Process.

Project-specific standards & procedures

· Based on quality standards for each deliverable
· Includes how PM activities themselves should be done
· Plans/Project must comply with external standards (CISG, ISO 9000, OSHA, etc.)
· Plans/Project must comply with organizational standards
· Plans/Project must meet the customer's quality standards
· Tracking / proof may be needed (metrics, measurements, etc.)

Four standards that must all be satisfied

External (ISO 9000, OSHA…) Organizational Customer's Project-specific

A plan that satisfies the org's internal standard but ignores the customer's quality bar has not planned quality correctly — all four layers have to be reconciled.

The Software Quality Assurance Group

an independent watchdog

The SQA group should prepare an SQA plan for a project.

📋

What the SQA plan identifies

"Evaluate, Audit, Standardize, Report, Change, Document, Feed back."

Evaluations to be performed · audits and reviews to be performed · standards applicable to the project · procedures for error reporting and tracking · procedures for change management · documents to be produced by the SQA group · amount of feedback provided to the software project team.

What the SQA group actually does

· Participates in developing the project's software process description
· Reviews software engineering activities to verify compliance with the defined process
· Audits designated software work products to verify compliance with those defined as part of the process
· Ensures deviations in software work/work products are documented and handled per a documented procedure
· Records any noncompliance and reports it to senior management

Why this matters on exam questions

Notice the last bullet: the SQA group doesn't just quietly fix things — a recorded noncompliance gets escalated to senior management. That's the detail examiners like to test.

§ 03 — QUALITY CONTROL

QC: watch the actual results, catch the actual problems

If QA is the rulebook, QC is the referee on the field, checking specific results against it.

🔍

Memory hook

"QA writes the recipe. QC tastes the soup."

Quality Assurance builds the process/framework (proactive, whole-project). Quality Control monitors specific project results for compliance with quality standards (reactive, per-deliverable) — and QA is the process that provides the framework of activities and standards for performing QC.

Perform Quality Control

monitor & correct

Perform quality control is concerned with monitoring specific project results for compliance with quality standards. It is:

Performed throughout the project — not just at the end.

May also include taking control actions to correct the causes of quality problems.

Perform quality assurance is the process that provides the framework of activities and standards for performing quality control. QA sets the rules; QC applies them to real, specific results.

§ 04 — PROJECT SUCCESS

Habits that keep a project alive

A grab-bag of practical management habits the slides call out by name — each one shows up as a specific term you should recognize.

Think Small

· Keep requirements tight & focused
· One milestone at a time
· Smaller, incremental chunks
· As simple as possible, but no simpler

The Process Spectrum

"Too much medicine can kill the patient." Process sits on a spectrum between Chaos (too little process) and Bureaucracy (too much process). Balance is crucial — neither extreme works.

🧊

Two ways over-process breaks a team

Analysis Paralysis vs. Paralysis Paranoia

Analysis Paralysis — over-process, nothing gets finished (65% of software professionals have experienced this). Paralysis Paranoia — fear of over-process, which leads to process avoidance altogether. One is too much process; the other is the overcorrection away from it.

Be Disciplined & Professional

· Learn to say "No" — be polite but firm
· The value of versions — "we will put that in phase 2"
· An ounce of prevention — cheaper than a pound of cure

Other manager habits worth naming

· MBWA (Management By Walk-About) — shows you're actually involved day-to-day, people say more 1-on-1, allows spontaneity, finds personnel problems sooner
· Delegate — don't be a "control freak"; be the hub, not everything
· Project Home Page — give the project an intranet page as a central repository for status, documents, and other resources

Success Metrics

the four that get quoted, and the one that's real

📅

1. On schedule

Requires good plan, estimation, and control.

💰

2. Within budget

Again: planning, estimation & control.

📄

3. According to requirements

Importance of good requirements; perception & negotiation are critical.

✨

4. High quality

May or may not be the same thing as item 3.

Only real measure: is the customer happy? Customer satisfaction is the metric that actually counts — a project can hit schedule, budget, and requirements and still fail this one.

§ 05 — WHY PROJECTS SUCCEED OR FAIL

The same factors, viewed from both sides

Every "successful projects do X" fact in these slides has a mirror-image "unsuccessful projects do the opposite" fact. Learn the pairs, not just one side.

🏆

Memory hook — why projects succeed

"Every User Expects Clear, Minimal, Standard, Firm, Formal, Reliable projects."

Executive support · User involvement · Experienced project manager · Clear business objectives · Minimized scope · Standard software infrastructure · Firm basic requirements · Formal methodology · Reliable estimates.

Why Executive Support?

the #1 success factor

Top management can help to: secure adequate resources, get approval for unique project needs in a timely manner, receive cooperation from people throughout the organization, and provide leadership guidance.

STATE OF THE PRACTICE IN SOFTWARE MANAGEMENT

Two categories of factors influence success or failure: Social Factors and Technology. Below, each is shown red (unsuccessful) vs. green (successful) — same axis, opposite outcome.

Social factors — unsuccessful projects

✗

Excessive schedule pressure

✗

Executive rejection of estimates

✗

Severe friction with clients

✗

Divisive corporate politics

✗

Poor team communications

✗

Naïve senior executives

✗

Project management malpractice

✗

Unqualified technical staff

✗

Generalists used for critical tasks: QA, testing, planning, estimating

Social factors — successful projects

✓

Realistic schedule pressure

✓

Executive understanding of estimates

✓

Cooperation with clients

✓

Congruent management goals

✓

Excellent team communications

✓

Experienced senior executives

✓

Capable project management

✓

Capable technical staff

✓

Specialists used for critical tasks: QA, testing, planning, estimating

Technology — unsuccessful projects

✗

No historical software measurement data

✗

Failure to use automated estimating / planning tools

✗

Failure to monitor progress or milestones

✗

Failure to use effective architecture / development methods

✗

Failure to use design reviews / code inspections

✗

Failure to include formal risk management

✗

Informal, inadequate testing

✗

Manual design and specification

✗

More than 30% creep in user requirements

Technology — successful projects

✓

Accurate software measurement, early & continuous tool use

✓

Formal progress reporting

✓

Formal architecture planning & development methods

✓

Formal design reviews & code inspections

✓

Formal risk management

✓

Formal testing methods

✓

Automated design, specification & configuration control

✓

Less than 10% creep in requirements

How To Ensure a Project Fails

a satirical checklist, taken straight

A deliberately sarcastic list — read each line as the mistake to avoid:

· Do the same things you did on the last project, over and over
· Don't listen to your experts — the last project worked out okay (mostly)
· Don't measure progress with metrics — the only thing that counts is "did you meet the delivery date?"
· Set delivery dates with the customer but not with the developers — "developers can meet any schedule we ask for"
· Don't use new tools — keep using the ones from ten, twenty years ago
· Spend your time making sure people do it your way
· Office politics and vendettas are more important than the project

§ 06 — MEASUREMENT & METRICS

Turning quality into a number you can track

"Not everything that can be counted counts, and not everything that counts can be counted." — Albert Einstein

Why measurement and metrics?

Software measurement is concerned with deriving a quantitative (numeric) value for an attribute of a software product or process (which is largely qualitative). This allows for objective comparisons between techniques and processes. Although some companies have introduced measurement programs, systematic use of measurement is still uncommon — and there are few standards in this area.

Three core definitions

Measure	Provides a quantitative indication of the size of some product or process attribute.
Measurement	The act of obtaining a measure.
Metric	A quantitative measure of the degree to which a system, component, or process possesses a given attribute.

📏

Chunking the vocabulary

Measure → Measurement → Metric

The measure is the thing (a number). Measurement is the verb — the act of getting that number. A metric is a measure specifically applied to a system/component/process attribute you care about. A software metric is any type of measurement relating to a software system, process, or related documentation (e.g. lines of code, person-days to build a component).

Categories of Metrics

product vs. process & project

Product metrics

Assess the quality of the design and construction of the software product being built. A quality metric should be a predictor of product quality.

Dynamic metrics — collected by measurements made of a program in execution. Help assess efficiency and reliability.

Static metrics — collected by measurements made of the system representations (design docs, code as text, not running). Help assess complexity, understandability, and maintainability.

Process & project metrics

Quantitative measures that let software engineers gain insight into the efficiency of the software process and the projects run inside that process framework.

Private process metrics (e.g. defect rates by individual or module) — known only to the individual or team concerned.

Public process metrics — enable organizations to make strategic changes to improve the software process.

Metrics should never be used to evaluate the performance of individuals.

Project metrics are used to avoid development schedule delays, mitigate potential risks, and assess product quality on an ongoing basis. Every project should measure its inputs (resources), outputs (deliverables), and results (effectiveness of deliverables).

Metrics assumptions

· A software property can be measured
· A relationship exists between what we can measure and what we actually want to know
· That relationship has been formalized and validated
· It may still be difficult to relate what can be measured to the desirable quality attributes we actually care about

Integrating metrics with process

Many software developers do not collect measures. Without measurement it is impossible to determine whether a process is improving or not. Baseline metrics data should be collected from a large, representative sampling of past projects — but getting that historic data is very difficult if previous developers didn't collect it on an ongoing basis.

Software Measurement, End to End

direct vs. indirect

Direct measures of the software engineering process: cost, effort.

Direct measures of the product: lines of code (LOC), execution speed, memory size, defects reported over a time period.

Indirect product measures examine the quality of the software product itself: functionality, complexity, efficiency, reliability, maintainability.

One activity, three outputs

A single measurement activity draws from process and product inputs, and feeds three outputs: process metrics, project metrics, and product metrics.

The Measurement Process

A software measurement process may be part of a quality control process. Data collected during this process should be maintained as an organizational resource. Once a measurement database has been established, comparisons across projects become possible.

Product Measurement Process (5 steps): Choose measurements to be made → Select components to be assessed → Measure component characteristics → Identify anomalous measurements → Analyse anomalous components.

Data Collection

A metrics program should be based on a set of product and process data. Data should be collected immediately (not in retrospect) and, if possible, automatically.

🤖

Three types of automatic data collection

"Static, Dynamic, Process."

Static product analysis · Dynamic product analysis · Process data collation.

Data Accuracy

· Don't collect unnecessary data — decide the questions to be answered in advance, and identify the required data
· Tell people why the data is being collected — it should not be part of personnel evaluation
· Don't rely on memory — collect data when it is generated, not after a project has finished

What Are Metrics For?

The loop: Software process → Control measurements → feeds Predictor measurements and Management decisions, which feed back into the process. The process also produces the Software product directly.

Analysis: it is not always obvious what data means — analyzing collected data is genuinely difficult. Professional statisticians should be consulted if available, and data analysis must take local circumstances into account.

Metric Types, with Examples

four families you'll recognize by name

📦

Size-oriented

Defects, human effort, Lines of Code (LOC).

⚙️

Function-oriented

Function points.

🌐

Web Engineering

Number of static web pages (Nsp), number of dynamic web pages (Ndp), and a Customization Index: C = Nsp / (Ndp + Nsp)

"Too much medicine can kill the patient." Process sits on a spectrum between Chaos and Bureaucracy, and balance is crucial. Over-process causes Analysis Paralysis (nothing gets finished — 65% of professionals have experienced it); fear of that can overcorrect into Paralysis Paranoia (avoiding process altogether). Neither extreme is the goal.

§ 08 — CHEAT SHEET

The whole chapter, one table

Concept	What it means	Key detail to remember
Quality (definition)	Conformance to explicit requirements, specified standards, and implicit expectations	Missing any one of the three makes quality "suspect"
Software testing	Full-life-cycle assessment of quality at each development step	NOT for burning leftover budget/schedule
Quality Assurance (QA)	Process improvement + fault-tolerant design + all V&V (incl. testing)	Provides the framework for QC; goal = few defects, minimal disruption
Quality Control (QC)	Monitors specific results against quality standards, throughout the project	Includes taking control actions to correct problems
SQA Group	Prepares SQA plan; reviews, audits, records noncompliance	Noncompliance gets reported to senior management
Success metrics	Schedule, budget, requirements, quality	Only real measure = customer satisfaction
Why projects succeed	Executive support, user involvement, experienced PM, clear objectives…	Executive support is listed first / bolded
Process Spectrum	Balance between Chaos and Bureaucracy	"Too much medicine can kill the patient"
Measure / Measurement / Metric	Quantitative value / the act of obtaining it / a measure of a specific attribute	Metric = measure applied to system/component/process attribute
Static vs. Dynamic metrics	System representations vs. program in execution	Static → complexity/understandability/maintainability; Dynamic → efficiency/reliability
Private vs. Public process metrics	Individual/team-only vs. organization-wide	Never use metrics to evaluate individuals
Metric types	Size-oriented, function-oriented, web engineering, product metrics	C = Nsp / (Ndp + Nsp); cyclomatic complexity; fan-in/fan-out

§ 09 — POP QUIZ

Click a question to reveal the answer

Quick self-check before you move to the Practice Lab.

Answer: (1) Software requirements are the foundation quality is measured from — lack of conformance is lack of quality. (2) Specified standards define the development criteria; not following them almost surely produces a lack of quality. (3) Implicit requirements often go unmentioned (e.g. maintainability) — meeting explicit requirements but missing these still makes quality suspect.

Answer: False. "Quality cannot be tested into a product." Testing is a measure of quality, not a deliverer of it — quality comes from good process, tools, methods, and people.

Answer: (D). SQA includes process improvement (prevent defects), fault-tolerant design (tolerate defects), and all V&V including testing. "Using up remaining budget/schedule" is explicitly named as NOT the purpose of testing.

Answer: Perform quality control monitors specific project results for compliance with quality standards, throughout the project, and may include taking control actions to fix quality problems. Perform quality assurance is the process that provides the framework of activities and standards used to perform quality control.

Answer: Is the customer happy? Customer satisfaction is the only real measure — schedule, budget, requirements, and quality are all commonly quoted but secondary to this.

Answer: Private. (Public process metrics, by contrast, enable organizations to make strategic changes.)

Answer: False. That's what dynamic metrics assess (they're collected from a program in execution). Static metrics — collected from system representations — help assess complexity, understandability, and maintainability.

Answer: 65%.

§ 10 — PRACTICE LAB

Apply the definitions to scenarios

Same shape of question you'll see on an exam — worth drilling until the reasoning is automatic.

1. A team tracks each developer's individual defect rate, and only that developer and their lead ever see the number. Is this a private or public process metric — and is it being used appropriately?

→ Private process metric

Why: it's known only to the individual/team concerned — that's the definition of private. It's used appropriately as long as it isn't repurposed to evaluate that developer's performance, which the slides explicitly forbid.

2. Before running any code, an engineer reviews the design documents to estimate how complex and maintainable the resulting system will be. Should they use static or dynamic metrics?

→ Static metrics

Why: static metrics come from system representations (design docs, source text) rather than execution, and are the ones that assess complexity, understandability, and maintainability.

3. A website has 20 static pages (Nsp) and 80 dynamic pages (Ndp). Calculate the Customization Index.

→ C = Nsp / (Ndp + Nsp) = 20 / (80 + 20) = 0.2

Why: direct substitution into the formula from the "Metric Types with Examples" slide. A higher proportion of dynamic pages relative to the total lowers C.

4. A project ships exactly on schedule, exactly on budget, and every written requirement is met — but the customer is unhappy with the result. By the slides' definition, was this project a success?

→ No

Why: schedule, budget, and requirements are commonly quoted success metrics, but the slides state the only real measure is customer satisfaction — "is the customer happy?" An unhappy customer means the project did not truly succeed, regardless of the other three boxes being checked.

5. During an audit, the SQA group discovers that a team skipped a required code inspection. What is the SQA group supposed to do about it, per their defined responsibilities?

→ Record the noncompliance and report it to senior management

Why: one of the SQA group's explicit responsibilities is "records any noncompliance and reports to senior management" — it isn't quietly resolved at the team level.

Quality isn't tested in.It's built in, then verified.