SE401 · Annotated Slides · Black Box Testing Techniques

Exam weight Low

Slide 2: • The test development process • Categories of test design techniques

Exam weight Low

Slide 3: • Before we start testing, we need to know

Exam weight Low

Slide 4: • The test design process can be done in different ways, from very informal (little or no documentation), to very formal.

Exam weight Low

Slide 5: 1. Test analysis 2. Test design

Exam weight Low

Test development: test analysis (what to test: test conditions) → test design (how: test cases) → test implementation (procedures, scripts, schedule).

Slide 6: • The test basis documentation is analyzed in order to determine what to test, i.e. to identify the test conditions .

Exam weight Low

Slide 7: • “Throw a wide net!” • First; identify as many test conditions as possible

Exam weight Low

Exam weight Low

Slide 9: • In order to know what the system should do, we need to have a source of information about the correct behavior of the system an oracle

Exam weight Low

Slide 10: • During test implementation the test cases are organized in the test procedures

Exam weight Low

Slide 11: • The test execution schedule defines:

Exam weight Low

Slide 12: • Writing the test procedure is another opportunity to prioritize the tests, to ensure that the best testing is done in the time available.

Exam weight Low

Case Study – Knight Capital

High Frequency Trading (HFT)

Exam weight Low

Knight Capital (2012): dead "zombie" code reactivated during a deployment lost $440M in 45 minutes: missing regression and deployment testing.

Exam weight Low

Exam weight Low

Slide 16: "Zombie Software" Blamed for Knight Capital Trading Snafu • A new algorithmic trading program had just been installed, and began

Exam weight Low

Slide 17: • SEC launched an investigation in Nov 2012. Findings:

Exam weight Low

Exam weight Low

Slide 19: Recall the difference between static and dynamic test techniques:

Exam weight Low

Black-box (specification-based) vs. white-box (structure-based) vs. experience-based techniques.

Exam weight Low

Exam weight Low

Slide 22: • Derive test cases from the functional specifications

Exam weight Low

Slide 23: • Derive systematically test cases from specification • Test cases are derived from how the software is constructed for

Exam weight Low

Slide 24: • Testing software against a specification of its external behavior without knowledge of internal implementation details

Exam weight Low

Slide 25: • Incorrect or missing functions • Interface errors

Exam weight Low

Slide 26: • How is functional validity tested? • How are system behavior and performance tested?

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Slide 30: • Program code is not necessary

Exam weight Low

Slide 31: • Different testing strategies (functional, structural, fault-based, model- based) are most effective for different classes of faults

Exam weight Low

Slide 32: • Functional test is applicable in testing at all granularity levels:

Exam weight Low

Slide 33: 1. Decompose the specification

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Slide 38: Single Defect Assumption : Failures are rarely the result of the

Exam weight Low

Slide 39: • The basic idea is to divide a set of test conditions into sub groups or sub sets (partitions) that can be considered the same .

Exam weight High

Equivalence partitioning: divide inputs into classes expected to behave the same; one test per class (valid and invalid).

🎯 How this can come up · 1 question

MCQ

A field accepts integers 1–100. The equivalence classes are:

Slide 40: • A black-box testing method that divides the input domain of a program into classes of data from which test cases are derived

Exam weight Low

Slide 41: • Typically the universe of all possible test cases is so large that you cannot try them all

Exam weight Low

Slide 42: • Partition the test cases into "equivalence classes” • Each equivalence class contains a set of "equivalent" test cases

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Slide 46: • When designing test cases, you may use different definitions of “equivalence”, each of which will partition the test case space

Exam weight Low

Slide 47: • When designing test cases, you may use different definitions of “equivalence”, each of which will partition the test case space

Exam weight Low

Slide 48: • Test multiple values in each equivalence class. • Often you’re not sure if you have defined the equivalence classes

Exam weight Low

Exam weight Low

Slide 50: • Look for ranges of numbers or values • Look for memberships in groups

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight High

Boundary value analysis: errors cluster at edges: test min, max and just outside/inside them.

🎯 How this can come up · 1 question

MCQ

For a valid range 18–65, two-value BVA tests:

Exam weight Low

Slide 58: • Test values, sizes, or quantities near the design limits

Exam weight Low

Slide 59: • A greater number of errors occur at the boundaries of the input domain rather than in the "center"

Exam weight Low

Slide 60: 1. If an input condition specifies a range bounded by values a b and , test cases should be designed with values a b and as well as values just above and just below a b and

Exam weight Low

Slide 61: • When choosing values from an equivalence class to test, use the values that are most likely to cause the program to fail

Exam weight Low

Slide 62: • Create test cases to test boundaries of equivalence classes

Exam weight Low

Exam weight Low

Exam weight Low

Exam weight Low

Slide 66: • Numeric values are often entered as strings which are then converted to numbers internally [int x = atoi(str);]

Exam weight Low

Slide 67: • Don't get so wrapped up in testing boundary cases that you neglect to test "normal" input values

Exam weight Low

Slide 68: • Doesn’t require much thought • May miss internal boundaries

Exam weight Low

Exam weight Low

Slide 70: • Why do both equivalence partitioning and boundary value analysis? • Boundary values are usually extreme values

Exam weight Low

Slide 71: • Equivalence Class Testing is appropriate when input data is defined in terms of intervals and sets of discrete values.

Exam weight Low

Slide 72: • Decision tables are a good way

Exam weight High

Decision tables: combinations of conditions → actions; each rule (column) is a test case; collapse rules whose outcome doesn’t depend on a condition.

🎯 How this can come up · 1 question

MCQ

Four independent binary conditions give at most how many rules?

Slide 73: • When creating decision tables, the specification is analyzed, and actions of the system are identified.

Exam weight Low

Exam weight Low

Slide 75: Rule 1 Rule 2 Rule 3 Rule 4 Rule Rule Rule 7 Rule 8 5 6

Exam weight Low

Slide 76: Rule 1 Rule 2 Rule 3 Rule 4 Rule Rule Rule 7 Rule 5 6 8

Exam weight Low

Slide 77: Rule 1 Rule 2 & 6 Rule 3 Rule 4 Rule Rule & 8 5 7

Exam weight Low

Slide 78: • Example- Decision table for credit-card

Exam weight Low

Exam weight Low

Exam weight Low

Slide 81: • Decision tables are a good way to:

Exam weight Low

Slide 82: • A system can be in a finite number of different states. This aspects of the system can be described as a ‘finite state machine’ ; a state

Exam weight High

State transition testing: states, events, transitions and actions; cover every valid transition and try invalid (null) ones.

🎯 How this can come up · 1 question

MCQ

Testing an ATM that locks the card after 3 wrong PINs is best done with:

Exam weight Low

Slide 84: • A ‘finite state machine’ is often shown as a state diagram • ATM PIN example.

Exam weight Low

Slide 85: • How many tests do we need to exercise every state?

Exam weight Low

Exam weight Low

Slide 87: • Why state transition testing?

Exam weight Low

Exam weight Low

Slide 89: • Use case describes interactions between actors (users and the system), which produce a result of value to a system user

Exam weight Low

Use-case testing: one test per main and alternative flow; good for acceptance and system tests.

Slide 90: • Identify test cases that exercise the whole system on a transaction by transaction basis

Exam weight Low

Slide 91: • Each use case has pre-conditions, which need to be met for a use case to work successfully.

Exam weight Low

Exam weight Low

Slide 93: • Very useful for designing acceptance tests with customer/user participation. • Describe the ‘process flows’ through a system base on its actual likely use.

Exam weight Low

Exam weight Low

General Testing

Exam weight Low

Slide 96: • Many systems perform multiple concurrent activities

Exam weight Low

Slide 97: • Measure the system's performance

Exam weight High

Non-functional: performance (response time), load (expected volume), stress (beyond the limits), security, usability, recovery, compatibility.

🎯 How this can come up · 1 question

MCQ

Pushing a system beyond its normal capacity to see how it fails is:

Exam weight Low

Slide 99: • Test the system at the limits of normal use • Test every limit on the program's behavior defined in the requirements

Exam weight Low

Slide 100: • Test the system under extreme conditions (i.e., beyond the limits of normal use)

Exam weight Low

Slide 101: • Any system that manages sensitive information or performs sensitive functions may become a target for intrusion (i.e., hackers)

Exam weight Low

Exam weight Low

Slide 103: • Is the user interface intuitive, easy to use, organized, logical? • Does it frustrate users?

Exam weight Low

Exam weight Low

Slide 105: • Try turning the power off or otherwise crashing the program at arbitrary points during its execution

Exam weight Low

Slide 106: • Test on all required hardware configurations

Exam weight Low

Slide 107: • Test to make sure the program is compatible with other programs it is supposed to work with

Exam weight Low

Slide 108: • Test all instructions given in the documentation to ensure their completeness and accuracy

Exam weight Low

Exam weight Low

Slide 110: • Tests are derived from the tester’s skill and intuition and their experience with similar applications and technologies.

Exam weight Low

Slide 111: • Error guessing = a commonly used experienced-based technique. • Generally testers anticipate defects based on experience.

Exam weight Low

Error guessing: use experience to anticipate likely mistakes (empty inputs, zero, nulls).

Slide 112: • Exploratory testing = concurrent test design, test execution, test logging and learning,

Exam weight Low

Exploratory testing: simultaneous learning, test design and execution, often time-boxed.

Slide 113: • The most widely practiced form of functional testing • The tester uses his or her domain knowledge, experience, or intuition

Exam weight Low

Slide 114: • Complex mathematical (or algorithmic) calculations • Worst case situations (similar to robustness)

Exam weight Low

Slide 115: • Experience really helps • Frequently done by the customer or user

Exam weight Low

Slide 116: • The choice of which test techniques to use depends on a number of factors, including:

Exam weight Low

Slide 117: • Case Study – Knight Capital • Testing techniques

Exam weight Low

Black-Box Testing Techniques