SE401 · Annotated SlidesIntroduction — Introduction to Software Testing
Introduction · Annotated slides

Introduction to Software Testing

Why testing is necessary (famous failures, the cost of poor quality), errors vs. defects vs. failures, what testing is and its objectives, test concepts (test case, oracle, scaffolding, fixture, suite, adequacy), the seven testing principles, the fundamental test process, the psychology of testing and independence, and the limits of testing.

Slide text Study notes Highlight: key words Tips, tricks & traps Seen again elsewhere How it can come up
Slide 1: Midterm
Exam weight Low
    Slide 2: Slide 2
    Exam weight Low
      Slide 3: • Why is testing necessary? • What is testing?
      Exam weight Low

        Why is testing necessary?

        4
        Exam weight Low
        1. Testing reduces the risk of failures in operation (Ariane 5, Therac-25, failed satellites, bank errors): poor quality is very expensive.
        Slide 5: Slide 5
        Exam weight Low
          Slide 6: Slide 6
          Exam weight Low
            Slide 7: Slide 7
            Exam weight Low
              Slide 8: Slide 8
              Exam weight Low
                Slide 9: • The Cost of Poor Software Quality in the US 2020 Report by CISQ Consortium for Information & Software Quality:
                Exam weight Low
                  Slide 10: Slide 10
                  Exam weight Low
                    Slide 11: • T0 - T0 + 36s : normal • Within SRI 2:
                    Exam weight Low
                    1. Ariane 5 (1996): reused Ariane 4 code overflowed converting a 64-bit float to a 16-bit integer: not a coding slip, an untested assumption.
                    Slide 12: • Not a programming error • unprotected conversion = design decision (~1980)
                    Exam weight Low
                      Slide 13: • Test! Test! Test! • Test! Even when the code is reused.
                      Exam weight Low
                        Slide 14: Slide 14
                        Exam weight Low
                          Slide 15: Congratulations!
                          Exam weight Low
                            Slide 16: • Testing is expensive.
                            Exam weight Low
                              Slide 17: testing fixing
                              Exam weight Low
                                Slide 18: • Testing adds to Cost of Conformance
                                Exam weight Low
                                  Slide 19: Slide 19
                                  Exam weight Low
                                    Slide 20: • Human Error
                                    Exam weight Low
                                      Slide 21: • Both causes of errors produce defects( = faults, bugs) in the code. • Defects, if executed, may result in failures of the SW system (the
                                      Exam weight Low
                                        Slide 22: Slide 22
                                        Exam weight Low
                                          Slide 23: Slide 23
                                          Exam weight Low
                                            Slide 24: • Testing has an important role in all stages of a SW product’s life cycle:
                                            Exam weight Low
                                              Slide 25: • To reduce the risk of problems occurring during operation • To check if the SW system meets:
                                              Exam weight Low
                                                Slide 26: Slide 26
                                                Exam weight Low
                                                  Slide 27: Slide 27
                                                  Exam weight Low
                                                    Slide 28: • It is a systematic process used to identify the correctness , completeness , and quality of developed software. It includes a set of
                                                    Exam weight Low
                                                      Slide 29: • The process of testing all SW life-cycle activities:
                                                      Exam weight Low
                                                        Slide 30: • Depending on the objectives of the test process, testing can be focused on
                                                        Exam weight Low
                                                          Slide 31: • Finding defects
                                                          Exam weight Low
                                                            Slide 32: Slide 32
                                                            Exam weight Low
                                                              Slide 33: Slide 33
                                                              Exam weight Low
                                                                Slide 34: Slide 34
                                                                Exam weight Low
                                                                  Slide 35: Q: How challenging is software testing? A: Testing is easier than design and
                                                                  Exam weight Low
                                                                    Slide 36: Q: How challenging is software testing?
                                                                    Exam weight Low

                                                                      Assessment

                                                                      A Self

                                                                      Test

                                                                      37
                                                                      Exam weight Low
                                                                        Slide 38: • The program reads in 3 integer values that represent the lengths of the sides of a triangle.
                                                                        Exam weight Low

                                                                          39
                                                                          Exam weight Low
                                                                            Slide 40: Write a set of test cases that you feel would adequately test this program.
                                                                            Exam weight Low
                                                                              Slide 41: Test
                                                                              Exam weight Low

                                                                                Software Testing

                                                                                Terminologies

                                                                                42
                                                                                Exam weight Low
                                                                                  Slide 43: Slide 43
                                                                                  Exam weight Low
                                                                                    Slide 44: • Failures are
                                                                                    Exam weight High
                                                                                    1. A human makes an ERROR (mistake) → it leaves a DEFECT (fault, bug) in the code → running it may cause a FAILURE (wrong observable behaviour).
                                                                                    TipA human makes an ERROR (mistake) → it leaves a DEFECT (fault, bug) in the code → running it may cause a FAILURE (wrong observable behaviour).
                                                                                    🎯 How this can come up · 2 questions
                                                                                    MCQ

                                                                                    A programmer types < instead of <=. The wrong operator in the code is a:

                                                                                    True / False

                                                                                    Every defect causes a failure.

                                                                                    Slide 45: • Defects are
                                                                                    Exam weight Low
                                                                                      Slide 46: • Errors are
                                                                                      Exam weight Low
                                                                                        Slide 47: • A human being makes an error ( mistake )
                                                                                        Exam weight Low
                                                                                          Slide 48: • An example : For any integer n, square (n) = n*n.
                                                                                          Exam weight Low
                                                                                            Slide 49: • An example : For any integer n, square (n) = n*n.
                                                                                            Exam weight Low
                                                                                              Slide 50: • Test Case Verdicts
                                                                                              Exam weight Low
                                                                                              1. Verdicts: pass, fail, inconclusive.
                                                                                              Slide 51: Testing: Concepts
                                                                                              Exam weight Low
                                                                                                Slide 52: • Test
                                                                                                Exam weight Low
                                                                                                  Slide 53: oracle
                                                                                                  Exam weight High
                                                                                                  1. Test oracle: the mechanism that decides whether an output is correct (expected results, a reference implementation, assertions).
                                                                                                  🎯 How this can come up · 1 question
                                                                                                  MCQ

                                                                                                  The source that tells you the expected result of a test is the:

                                                                                                  Slide 54: • Test Scaffolding
                                                                                                  Exam weight Low
                                                                                                    Slide 55: fixture: a fixed state of the software under test used as a baseline for
                                                                                                    Exam weight Low
                                                                                                    1. Fixture: the fixed state the tests run against. Suite: a collection of test cases. Scaffolding: drivers, stubs, harnesses.
                                                                                                    Slide 56: suite
                                                                                                    Exam weight Low
                                                                                                      Slide 57: • Test
                                                                                                      Exam weight Low
                                                                                                        Slide 58: • Test adequacy
                                                                                                        Exam weight Low
                                                                                                          Slide 59: • Test
                                                                                                          Exam weight Low
                                                                                                            Slide 60: Slide 60
                                                                                                            Exam weight Low
                                                                                                              Slide 61: • The purpose of testing
                                                                                                              Exam weight Low
                                                                                                                Slide 62: • Software testing is a planned process that is used to identify the correctness, completeness, security and quality of
                                                                                                                Exam weight Low
                                                                                                                  Slide 63: • Testing is necessary, but not sufficient for quality assurance
                                                                                                                  Exam weight Low
                                                                                                                    Slide 64: Slide 64
                                                                                                                    Exam weight Low

                                                                                                                      The seven test principles

                                                                                                                      65
                                                                                                                      Exam weight High
                                                                                                                      1. Seven principles: testing shows presence of defects (not absence) · exhaustive testing is impossible · early testing · defect clustering · pesticide paradox · testing is context dependent · absence-of-errors fallacy.
                                                                                                                      TipSeven principles: testing shows presence of defects (not absence) · exhaustive testing is impossible · early testing · defect clustering · pesticide paradox · testing is context dependent · absence-of-errors fallacy.
                                                                                                                      Slide 66: Slide 66
                                                                                                                      Exam weight Low
                                                                                                                        Slide 67: • Testing can show that defects are present, but cannot prove that there are no defects.
                                                                                                                        Exam weight Low
                                                                                                                        1. Testing can show defects are present, never that they are absent.
                                                                                                                        Slide 68: • Testing everything(all combinations of input and preconditions) is not feasible except for specific cases.
                                                                                                                        Exam weight Low
                                                                                                                        1. Exhaustive testing is impossible except in trivial cases: use risk and priorities.
                                                                                                                        Slide 69: • Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined
                                                                                                                        Exam weight Low
                                                                                                                          Slide 70: • A small number of modules contains most of the defects discovered during pre release testing.
                                                                                                                          Exam weight Low
                                                                                                                          1. Defect clustering: a small number of modules contain most of the defects (Pareto).
                                                                                                                          Slide 71: • If the same tests are repeated over and over again, the same set of test cases will no longer find any new bugs.
                                                                                                                          Exam weight High
                                                                                                                          1. Pesticide paradox: repeating the same tests finds no new bugs: review and add tests regularly.
                                                                                                                          🎯 How this can come up · 1 question
                                                                                                                          MCQ

                                                                                                                          Running the same tests over and over stops finding new defects. This is the:

                                                                                                                          Slide 72: • Testing is done differently in different contexts.
                                                                                                                          Exam weight Low
                                                                                                                            Slide 73: • Finding and fixing defects does not help if the software system does not fulfill users' needs and expectations .
                                                                                                                            Exam weight Low
                                                                                                                            1. Absence-of-errors fallacy: a bug-free system that doesn’t meet users’ needs is still a failure.

                                                                                                                            Fundamental Test Processes

                                                                                                                            74
                                                                                                                            Exam weight Low
                                                                                                                            1. Fundamental test process: planning and control → analysis and design → implementation and execution → evaluating exit criteria and reporting → test closure.
                                                                                                                            Slide 75: • Test case
                                                                                                                            Exam weight Low
                                                                                                                              Slide 76: • Test plan
                                                                                                                              Exam weight Low
                                                                                                                                Slide 77: • Test planning • Test monitoring and control
                                                                                                                                Exam weight Low
                                                                                                                                  Slide 78: • Who, what , why, when and where
                                                                                                                                  Exam weight Low
                                                                                                                                    Slide 79: • Review test basis:
                                                                                                                                    Exam weight Low
                                                                                                                                      Slide 80: • Implement :
                                                                                                                                      Exam weight Low
                                                                                                                                        Slide 81: • Evaluate :
                                                                                                                                        Exam weight Low

                                                                                                                                          The psychology of testing

                                                                                                                                          82
                                                                                                                                          Exam weight Low
                                                                                                                                          1. Psychology: testers need curiosity, professional pessimism, attention to detail, good communication; report defects objectively.
                                                                                                                                          Slide 83: • Curiosity • Professional pessimism
                                                                                                                                          Exam weight Low
                                                                                                                                            Slide 84: • A certain degree of independence is often more effective at finding defects and failures.
                                                                                                                                            Exam weight Low
                                                                                                                                            1. Independence levels: the author tests their own code → another developer → an independent test team → outside organization. More independence finds different defects.
                                                                                                                                            Slide 85: • Independence levels:
                                                                                                                                            Exam weight Low
                                                                                                                                              Slide 86: • Be clear and objective
                                                                                                                                              Exam weight Low

                                                                                                                                                Limitations of

                                                                                                                                                Software Testing

                                                                                                                                                87
                                                                                                                                                Exam weight Low
                                                                                                                                                  Slide 88: • Both, testing and verification attempts to exhibit new failures • Debugging is a systematic process that finds and eliminates the defect
                                                                                                                                                  Exam weight Low
                                                                                                                                                    Slide 89: • Myth: “Principles are just for reference. I will not use them in practice .”
                                                                                                                                                    Exam weight Low
                                                                                                                                                      Slide 90: • Can we exhaustively test a
                                                                                                                                                      Exam weight Low
                                                                                                                                                        Slide 91: “ Program testing can be used to show the presence of bugs, but never their absence. ”
                                                                                                                                                        Exam weight Low
                                                                                                                                                        1. "Program testing can be used to show the presence of bugs, but never to show their absence." (Dijkstra)
                                                                                                                                                        Slide 92: Slide 92
                                                                                                                                                        Exam weight Low
                                                                                                                                                          Slide 93: • One of the most difficult problems in testing is not knowing when to stop.
                                                                                                                                                          Exam weight Low
                                                                                                                                                            Slide 94: • Take into account
                                                                                                                                                            Exam weight Low