软件测试相关度量.ppt

1,Software Metrics,Lecture 10Software Test Metrics,清 华 大 学 软 件 学 院2004年5月11日,2,COCOMO,where a,b,c and d depend on the development mode.,Organic modeSemi-detached modeEmbedded mode,Basic model Intermediate modelAdvanced model,3,COCOMO:Basic,The Basic COCOMO,development time cannot be predicted at this stage.,4,COCOMO:Intermediate,The Intermediate COCOMO:,5,COCOMO II,Application Composition E=OP/PROD Early Design model E=2.45 KLOC EAFPost-Architecture model,6,Putnams SLIM,7,Test ConceptsEstimating Number of Test Case&Allocating Test Times Test Coverage MetricsRemaining Defects Metrics,Contents,8,Section 1Test Concepts,9,防不胜防的软件错误 例1：1963年,美国,飞往火星的火箭爆炸,损失$10 million.原因:FORTRAN循环 DO 5 I=1,3 误写为 DO 5 I=1.3,软件测试是保证软件质量的关键步骤，是对软件规格说明、设计和编码的最后复审，其工件量约占总工作量40%以上（对于人命关天的情况，测试相当于其它部分总成本的3 5倍）,Roles of Software Testing,10,What is software testing,“Testing is the process ofexecuting a program with theintent of finding errors.”Glen Myers,11,Software Testing Objectives,Needing minimum effort and timeShowing that software functions work according to specification,It cannot show absence of errors or defects.It can only show that errors are present,Uncovering different cases of errors,12,1:5-10 Mainframes i.e.,1 tester for every 5 to 10 developers 2:3 Microsoft,19922:1 Lotus(for 1-2-3 Windows)1:2 Average of 4 major companies,1992Microsoft,Borland,WordPerfect,Novell,Tester-to-developer ratios,13,What is a test case?,A set of test inputs,execution conditions,and expected results developed for a particular objective,14,A test is made up of(many)test casesA test procedure is the detailed instructions for setting up,starting,monitoring and restarting a given test case.a.k.a test plana test case may be used in more than one test procedurea test case may include many subtests,What is a test?,15,Test,test case,test case,test case,subtest,subtest,subtest,subtest,What is a test?(cont.),16,Testing Methodology,White-box,Black-box,17,Objective:Cover the software structure,Structural(White-Box)Test Criteria,Criteria based oncontrol flowdata flowMetric:percentage of coverage achievedAdequacy based on metric requirements for criteria,18,Graph representation of control flow and data flow relationships,Flow Graphs,Control FlowThe partial order of statement execution,as defined by the semantics of the languageData FlowThe flow of values from definitions of a variable to its uses,19,Statement Coverage Branch Coverage Condition Coverage Basis Path Coverage All-Edges Coverage All-Paths Coverage,White-Box Test,20,123456789101112131415,function P return INTEGER isbeginX,Y:INTEGER;READ(X);READ(Y);while(X 10)loopX:=X 10;exit when X=10;end loop;if(Y 20 and then X mod 2=0)thenY:=Y+20;elseY:=Y 20;end if;return 2*X+Y;end P;,A Sample Program to Test,21,2,3,4,5,6,9,10,12,14,T,T,F,F,9,T,F,7,T,F,9a,9b,Ps Control Flow Graph(CFG),22,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X=20,Y=10)(X=20,Y=30),All-Statements Coverage of P,23,minimum level coverage using structural testingmay be impossible to achieve 100%statement coveragecode only executed in exceptional or dangerous circumstancescode that is unreachable,Statement Testing:weaknesses,24,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,At least 2 test cases needed,(X=20,Y=10)(X=15,Y=30),All-Branches Coverage of P,25,Branch Testing：weaknesses,Branch testing is one level up from statement testing in degree of coveragesame problem in achieving 100%coverage,26,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,At least 3 test cases needed,(X=20,Y=20)5T,7T,9aF,9bT(X=15,Y=10)5T,7F,9aT,9bF(X=9,Y=10)5F,7F,9aT,9bF,Condition Coverage of P,X 10X=10Y 20X mod 2=0,27,Condition Coverage,Generate test data such that all conditions in a decision take on both outcomes(if possible)at least onceMay not achieve branch coverage:consider while(x 20)x=0 causes first true,second falsex=34 causes first false,second true,28,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,Basic-paths Coverage of P,cyclomatic complexity:V(G)=e-n+2,29,2,3,4,5,6,10,12,14,T,F,9,T,F,7,T,F,4 test cases needed,(X=10,Y=10)(X=10,Y=20)(X=20,Y=10)(X=15,Y=10),Basic-paths Coverage of P,30,Basis path testing：Strengths/weaknesses,Basis set is easy to computethe technique is readily applicable to both structured and unstructured programsDrawback:the basis set is not unique;there will be obvious paths omittedDrawback:the number of paths in the basis set can greatly exceed the number necessary to achieve branch coverage,31,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,At least 3 test cases needed,(X=20,Y=10)(X=5,Y=30)(X=21,Y=10),All-Edges Coverage of P,32,2,3,4,5,6,9b,10,12,14,T,T,F,F,9a,T,F,7,T,F,Infinitely many test cases needed,(X=5,Y=10)(X=15,Y=10)(X=25,Y=10),All-Paths Coverage of P,33,All-Paths testing,Generate test data to exercise all distinct paths in a program!a path that makes i iterations through a loop is distinct from a path that makes i+1 iterations through the loop,even if the same nodes are visited in both iterations.Thus,there are an infinite number of paths in the string processing program!,34,All-Paths testing(cont.),包含的不同执行路径数达520条，对每一条路径进行测试需要1毫秒，假定一年工作365 24小时，要想把所有路径测试完，需3170年,35,2,3,4,5,6,9b,10,12,14,T,F,9a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,Ps Control and Data Flow Graph,36,How many test cases are needed?,2,3,4,5,6,9b,10,12,14,T,F,9a,T,F,Y,X,X,Y,Y,X,X,Y,Y,X,X,X,T,F,7,T,F,X,X,All-Uses Coverage of P,37,all-statements,all-edges,boundary-interiorloop testing,min-maxloop testing,all-paths,all-defs,all-uses,all-DU-paths,all-p-uses,all-c-uses,C1,C2,subsumes,Structural Subsumption Hierarchy,all-branches,38,Black Box Test,Equivalence Class Boundary Analysis,39,Testing Levels,Testing occurs throughout lifecycleUnitIntegrationSystemAcceptanceRegression,40,Overall Approach,Unit testing(module)Integration testingSystem testingAcceptance testing,White/Black boxBlack boxBlack box-requirementsBlack box-performance,41,Section 2Estimating Number of Test Case&Allocating Test Times,42,Estimate No.of Test Cases,Time(available time available staff)/(average time to prepare a test case),How many test cases do we need?,Cost(available budget)/(average preparation cost per test case),Select the minimum number of the two,43,Example,Development budget:$4 million,%10 of which spent on preparing test casesEach test case costs$250 or 4 hoursThe duration of project is set to be 25 weeks(of 40 hours each)and 5 staffs for test cases How many test cases should be prepared?,From cost N1=(4,000,000 0.1)/250=1,600 From time N2=(25 40 5)/4=1,250 N=min(N1,N2),therefore,N=1,250,44,Allocating Test Time,How to allocate test time among system components(e.g.,acquired components,developed components)test type(e.g.,unit test,integrated test,system test)and,45,Test Time:Systems/1,Allocate less than 10%of the remaining time to certification test of acquired components Allocate time to the developed components based on their significance,OS,System software,Developed components,Acquired components,Interface to other systems,46,Test Time:Systems/2,Example:Total test time:340 hTest case proportion:Associated system components:64.7%Operating system component:23.5%,Interface to other systems(40 h),Acquired components(20 h),Developed components(200 h),OS,System software(80 h),Associated System Components 220 h,47,Test Case Invocation,In what order the system should be tested?Recommended sequence of system test:Acquired components(certification test only)Developed product Other systems,OS,etc.(load test only),48,Section 3Test Coverage Metrics,49,Test Coverage Metrics/1,Coverage of what?Statement coverage Branch coverage Component/Module coverage Specification coverage GUI coverage,50,Test Coverage/1,Statement coverage(CVs)Portion of the statements tested by at least one test case.,51,Test Coverage/2,Branch coverage(CVb)Portion of the branches in the program tested by at least one test case.,52,Test Coverage/3,Component coverage(CVcm)Portion of the components in the software covered and tested by at least one test case,53,Test Coverage/4,GUI coverage(CVGUI)Portion of the GUI elements(e.g.,menus,buttons,multiple selections,text fields,etc.)in the software covered and tested by at least one test case.,54,Test Pass,Failure&Pending Rate/1,Test Pass Rate(Rtp)Portion of the test cases that were executed successfully(i.e.,produced expected output).,55,Test Pass,Failure&Pending Rate/2,Test Failure Rate(Rtf)Portion of the test cases that were not executed successfully(i.e.,produced different output than expected).,56,Test Pass,Failure&Pending Rate/3,Test Pending Rate(Rtpend)Portion of the test cases that were pending(i.e.,couldnt be executed or correctness of the output couldnt be verified).,57,Section 4Remaining Defects Metrics,58,Remaining Defects Metrics/1,The idea is to inject(seed)some faults in the program and calculate the remaining bugs based on detecting the seeded faults Mills 1972,Assuming that the probability of detecting the seeded and non-seeded faults are the same,59,Remaining Defects Metrics/2,60,Remaining Defects Metrics/3,Ns is already known;nd and ns are measured for a certain period of time,61,Example,Assume that,62,Comparative Remaining Defects/1,Two testing teams will be assigned to test the same product,63,Example,64,Summary,Estimating Number of Test Case?,Test Coverage MetricsRemaining Defects Metrics,65,Further reading 1.Software testing,Homework,66,Thanks for your time and attention!,