SE401 · Annotated SlidesTesting Metrics and Tools — Testing Metrics and Tools
Testing Metrics and Tools · Annotated slides

Testing Metrics and Tools

Why measure testing, base vs. calculated metrics (test-case productivity, execution and pass rates, defect density, defect leakage, defect removal efficiency, coverage), test tool classes (management, static, design, execution, performance, monitoring), benefits and risks of tools, and interview questions on QA vs. QC and verification vs. validation.

Slide text Study notes Highlight: key words Tips, tricks & traps Seen again elsewhere How it can come up
Slide 1: Slide 1
Exam weight Low
    Slide 2: • Testing Metrics • Testing Tools
    Exam weight Low
      Slide 3: • Metrics can be defined as “STANDARDS OF MEASUREMENT” • Metric is a unit used for describing or measuring an attribute
      Exam weight Low
      1. "You cannot improve what you cannot measure": metrics show progress, quality and where to act.
      Slide 4: • "We cannot improve what we cannot measure" and Test Metrics helps us to do exactly the same.
      Exam weight Low
        Slide 5: • Software testing metrics or software test measurement is the quantitative indication of extent, capacity, dimension, amount or size of
        Exam weight Low
          Slide 6: • You cannot improve what you cannot measure
          Exam weight Low
            Slide 7: • Take decision for next phase of activities • Evidence of the claim or prediction
            Exam weight Low
              Slide 8: test metrics is to use the data for
              Exam weight Low
                Slide 9: Slide 9
                Exam weight Low
                  Slide 10: • Identification of correct testing metrics is very important. Few things need to be considered before identifying the test metrics
                  Exam weight Low
                    Slide 11: • Manual test metrics are classified into two classes
                    Exam weight Low
                      Slide 12: • Base metrics is the raw data collected by Test Analyst during the test case development and execution (# of test cases executed, # of test
                      Exam weight Low
                      1. Base metrics: raw counts (test cases written, executed, passed, failed, blocked; defects found). Calculated metrics are derived from them.
                      Slide 13: • Major Base Metrics:
                      Exam weight Low
                        Slide 14: • Major Calculated Metrics:
                        Exam weight High
                        1. Examples: % test cases executed = executed / total × 100; pass rate = passed / executed × 100; defect density = defects / size (KLOC).
                        🎯 How this can come up · 1 question
                        MCQ

                        200 test cases, 160 executed, 120 passed. The pass rate of executed tests is:

                        Slide 15: • Major Calculated Metrics:
                        Exam weight Low
                          Slide 16: • Major Calculated Metrics:
                          Exam weight Low
                            Slide 17: • Major Calculated Metrics:
                            Exam weight Low
                              Slide 18: • Major Calculated Metrics:
                              Exam weight Low
                                Slide 19: • Major Calculated Metrics:
                                Exam weight Low
                                  Slide 20: • Major Calculated Metrics:
                                  Exam weight Low
                                    Slide 21: • Major Calculated Metrics:
                                    Exam weight Low
                                      Slide 22: • Major Calculated Metrics:
                                      Exam weight Low
                                        Slide 23: • Major Calculated Metrics:
                                        Exam weight Low
                                          Slide 24: • Major Calculated Metrics:
                                          Exam weight Low
                                            Slide 25: • Major Calculated Metrics:
                                            Exam weight Low
                                              Slide 26: • Major Calculated Metrics:
                                              Exam weight Low
                                                Slide 27: • Major Calculated Metrics:
                                                Exam weight Low
                                                  Slide 28: • Major Calculated Metrics:
                                                  Exam weight Low
                                                    Slide 29: • The total amount of time spent actually testing the product measured in test hours
                                                    Exam weight Low
                                                      Slide 30: • Code Coverage (How much of the code is being exercised?)
                                                      Exam weight Low
                                                        Slide 31: • Defect removal percentage
                                                        Exam weight High
                                                        1. Defect removal efficiency (DRE) = defects found before release / (before + after release) × 100.
                                                        🎯 How this can come up · 1 question
                                                        MCQ

                                                        90 defects found in testing, 10 found by customers after release. DRE =

                                                        Slide 32: • Fundamental test metrics are a combination of Base Metrics that can then be used to produce Calculated Metrics .
                                                        Exam weight Low
                                                          Slide 33: • Major Calculated Software Test
                                                          Exam weight Low

                                                            Testing Tools

                                                            34
                                                            Exam weight Low
                                                              Slide 35: Slide 35
                                                              Exam weight Low
                                                                Slide 36: • Test tools can be used for one or more activities that support testing:
                                                                Exam weight Low
                                                                1. Tools support test management, static testing, test design, execution and logging, performance and monitoring.
                                                                Slide 37: • Tools can have one or more purposes, depending on the context:
                                                                Exam weight Low
                                                                  Slide 38: • Tools are classified according to the testing activities that they support.
                                                                  Exam weight Low
                                                                    Slide 39: • Characteristics
                                                                    Exam weight Low
                                                                      Slide 40: • Test management tools and application lifecycle management tool (ALM)
                                                                      Exam weight Low
                                                                        Slide 41: • Tools for static testing
                                                                        Exam weight Low
                                                                          Slide 42: • Review process tools
                                                                          Exam weight Low
                                                                            Slide 43: • Static analysis tools (D)
                                                                            Exam weight Low
                                                                            1. Static analysis tools (SpotBugs, checkstyle, JDepend) find defects in code without running it.
                                                                            Slide 44: • Static analysis tools (D)
                                                                            Exam weight Low
                                                                              Slide 45: Slide 45
                                                                              Exam weight Low
                                                                                Slide 46: • Static analysis tool example: Source Monitor
                                                                                Exam weight Low
                                                                                  Slide 47: • Static analysis tool example: Source Monitor
                                                                                  Exam weight Low
                                                                                    Slide 48: • JDepend
                                                                                    Exam weight Low
                                                                                      Slide 49: • Modeling tools (D)
                                                                                      Exam weight Low
                                                                                        Slide 50: • Modeling tool example: StarUML
                                                                                        Exam weight Low
                                                                                          Slide 51: • The major benefit of static testing tools and modeling tools is the cost effectiveness of finding more defects at an earlier time in the
                                                                                          Exam weight Low
                                                                                            Slide 52: • Test design tools • Model-based testing tools
                                                                                            Exam weight Low
                                                                                              Slide 53: • Test design tools
                                                                                              Exam weight Low
                                                                                                Slide 54: • Model-based testing
                                                                                                Exam weight Low
                                                                                                  Slide 55: • TDD -Test driven development tool (D)
                                                                                                  Exam weight Low
                                                                                                    Slide 56: • Test execution tools • Coverage tools
                                                                                                    Exam weight Low
                                                                                                      Slide 57: • Test execution tools
                                                                                                      Exam weight Low
                                                                                                      1. Test execution tools run scripted tests; harnesses and unit frameworks provide drivers and stubs.
                                                                                                      Slide 58: • Test harnesses (D) and Unit test framework tools (D)
                                                                                                      Exam weight Low
                                                                                                        Slide 59: • Drivers
                                                                                                        Exam weight Low
                                                                                                          Slide 60: • Test harnesses (D) and Unit test framework tools (D)
                                                                                                          Exam weight Low
                                                                                                            Slide 61: • Characteristics of test harness and unit test framework tools
                                                                                                            Exam weight Low
                                                                                                              Slide 62: • Test comparators
                                                                                                              Exam weight Low
                                                                                                                Slide 63: • Test data for performance testing
                                                                                                                Exam weight Low
                                                                                                                  Slide 64: • Tests should reflect realistic (correct) scenarios
                                                                                                                  Exam weight Low
                                                                                                                    Slide 65: • Common features of test data preparation tools
                                                                                                                    Exam weight Low
                                                                                                                      Slide 66: • Monitoring tools identifying problems and sending an alert message to the administrator (e.g. network administrator)
                                                                                                                      Exam weight Low
                                                                                                                        Slide 67: Slide 67
                                                                                                                        Exam weight Low
                                                                                                                          Slide 68: • Features or characteristics of dynamic analysis tools include support for
                                                                                                                          Exam weight Low
                                                                                                                            Slide 69: Slide 69
                                                                                                                            Exam weight Low
                                                                                                                              Slide 70: • There are tools specialized for use in a particular type of application
                                                                                                                              Exam weight Low
                                                                                                                                Slide 71: • There are tools specialized for use in a particular type of application. • Example:
                                                                                                                                Exam weight Low
                                                                                                                                  Slide 72: • Testers may use:
                                                                                                                                  Exam weight Low
                                                                                                                                    Slide 73: • Simply purchasing or leasing a tool does not guarantee success with that tool!
                                                                                                                                    Exam weight Low
                                                                                                                                      Slide 74: • Reduce repetitive work (running regression tests, re-entring the same test data, etc.)
                                                                                                                                      Exam weight Low
                                                                                                                                      1. Benefits: less repetitive work, consistency, objective assessment. Risks: unrealistic expectations, underestimated effort, over-reliance on the tool.
                                                                                                                                      Slide 75: • Greater consistency and repeatability
                                                                                                                                      Exam weight Low
                                                                                                                                        Slide 76: • Objective assessment
                                                                                                                                        Exam weight Low
                                                                                                                                          Slide 77: • Ease of access to information about the tests or test effort
                                                                                                                                          Exam weight Low
                                                                                                                                            Slide 78: • Unrealistic expectations for the tool (functionality and ease of use) • Underestimating time, cost, effort for the introduction of a tool (training,
                                                                                                                                            Exam weight Low
                                                                                                                                              Slide 79: • Tools are not magic! • They can do very well what they have been designed to do, but they
                                                                                                                                              Exam weight Low
                                                                                                                                                Slide 80: • This type of tool often requires significant effort in order to achieve significant benefits.
                                                                                                                                                Exam weight Low
                                                                                                                                                  Slide 81: • The design of the load to be generated by the tool • Timing aspects probe effect
                                                                                                                                                  Exam weight Low
                                                                                                                                                    Slide 82: • There is a risk that the changes to make old code to conform to new standard will introduce an unexpected side effect
                                                                                                                                                    Exam weight Low
                                                                                                                                                      Slide 83: • They need to interface with other tools or spreadsheets in order to produce information in the best format for the current needs of the
                                                                                                                                                      Exam weight Low

                                                                                                                                                        Interview Questions and Answers

                                                                                                                                                        84
                                                                                                                                                        Exam weight Low
                                                                                                                                                          Slide 85: • When can we stop testing?
                                                                                                                                                          Exam weight Low
                                                                                                                                                          1. Stop testing when exit criteria are met: coverage, defect rates, deadlines, budget, acceptable residual risk.
                                                                                                                                                          Slide 86: • What is difference between QA, QC and Software Testing?
                                                                                                                                                          Exam weight Low
                                                                                                                                                            Slide 87: • Quality Assurance (QA) : QA refers to the planned and systematic way of monitoring the quality of process which is followed to produce a quality product. QA
                                                                                                                                                            Exam weight Low
                                                                                                                                                              Slide 88: What is verification and validation? • Verification: process of evaluating work-products of a development
                                                                                                                                                              Exam weight Low
                                                                                                                                                                Slide 89: Explain Branch Coverage and Decision Coverage. • Branch Coverage is testing performed in order to ensure that every
                                                                                                                                                                Exam weight Low
                                                                                                                                                                  Slide 90: • The term "static analysis" is conflated, but here we use it to mean a collection of algorithms and techniques used to analyze source code in
                                                                                                                                                                  Exam weight Low
                                                                                                                                                                    Slide 91: Questions:
                                                                                                                                                                    Exam weight Low
                                                                                                                                                                      Slide 92: • Testing “Phases” • Static vs. Dynamic Testing
                                                                                                                                                                      Exam weight Low
                                                                                                                                                                        Slide 93: • Definition - What does Quality Assurance (QA)
                                                                                                                                                                        Exam weight Low
                                                                                                                                                                          Slide 94: • Software quality assurance (SQA) is a process that ensures that developed
                                                                                                                                                                          Exam weight Low
                                                                                                                                                                            Slide 95: • The area of Software Quality Assurance can be broken down into a number of smaller areas such as
                                                                                                                                                                            Exam weight Low
                                                                                                                                                                              Slide 96: don’t build it
                                                                                                                                                                              Exam weight Low
                                                                                                                                                                                Slide 97: • Recognizing the ACM and IEEE code of ethics for engineers: • PUBLIC - Software testers shall act consistently with the public interest. • CLIENT AND EMPLOYER - Software testers shall act in a manner that is in the best interests of their client and
                                                                                                                                                                                Exam weight Low