Validity and Validation Methods.ppt

Validityand Validation Methods,Workshop Flow,The construct of MKTGain familiarity with the construct of MKTExamine available MKT instruments in the fieldAssessment DesignGain familiarity with the Evidence-Centered Design approachBegin to design a framework for your own assessment Assessment DevelopmentBegin to create your own assessment items in line with your frameworkAssessment ValidationLearn basic tools for how to refine and validate an assessment Plan next steps for using assessments,Domain Modeling(Design Pattern),(Define Test Specs),Domain Analysis,Define item Template,Define item Specs,Assemble Test,Document TechnicalInfo,Assessment Development Process,Validity:The Cardinal Virtue of Assessment,The degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.-Mislevy,Steinberg,and Almond,2003Validation is a process of accumulating evidence to provide a scientifically sound validity argument to support the intended interpretation of test scores-Standards for Educational and Psychological Testing(AERA/APA/NCME,1999),Jargon Note:Two kinds of“evidence”,Assessment Reliability,The extent to which an instrument yields consistent,stable,and uniform results over repeated administrations under the same conditions each time,Figure obtained from the website:http:/,Steps of item Validation,Iterative Refinement,Expert Panel Review(Formative),Are the items aligned withThe test specifications?Content covered in the curriculum?State or national standards?Is the complexity level aligned with intended use(e.g.,target population,grade-level)?Are the items prompts and rubrics aligned?,2.Feasibility of Items(Think-Alouds),Does the item make sense to the teacher?Does the item elicit the cognitive processes intended?Can the item be completed in the available time?Can respondents use the diagrams,charts,tables as intended?Is the language clear?Are there differences in approaches by experts and novices(or teachers exposed or not to the relevant instruction)?,SimCalc Example:Think-Alouds,SimCalc,Expected proportional reasoning:3.5 white x white-=-3 dark 5 dark,Found:Just draw the bars!,Proportional Reasoning Problem#3,Conducting Think-Alouds,SampleN:You learn the most in the first 3-6WhoExperts and NovicesLow,Medium,and High AchieversVarying in proficiency in EnglishData capture and analysisData can be extremely rich analyzed with varying levels of detailOften sufficient to do real-time note-takingVideotaping can be helpfulDocumentProblems with item clarity(language,graphics)Response processes What strategies are they using?,3.Field Testing,Item-level concernsAre there ceiling or floor effects?What is the range of responses we can expect from a variety of teachers?Is the amount of variation in responses sufficient to support statistical analysis?What is the distribution of responses across distracters?Do the items discriminate among teachers performing at different levels?,Assessment-level concernsAre there biases among subgroups?Does the assessment have high internal reliability?What is the factor structure of the test?,Key Item Statistic:Percent Correct,What percent of people get it correct?Gives us a sense of:The item difficultyThe range of responsesAlerts you to potential problems:Floor=roughly 0-10%Ceiling=roughly 85-100%,SimCalc Example:Exploratory Results for item#20,SimCalc,Quartiles of total test score,SimCalc Example:Exploratory Results for item#43,Skip,SimCalc,SimCalc Example:Exploratory Results for item#6,SimCalc,Conducting a Field Test,Test under conditions as close to“real”as possibleAnalogous population of teachersAdministration conditionsFormattingScoringGather and use demographic dataDetermine sample size based onThe number of teachers you can getThe kinds of statistical analyses you decide to conducte.g.,5-10 respondents per item for fancy statisticsCan use simple and fancy statistics,Field Testing with Teachers by Mail,Purchasing national mailing lists of teachershttp:/practices mailing sequence(Cook et al.,2000)An introductory postcard announcing that a survey will be sentAbout a week later,a packet containing the surveyAbout two weeks later,a reminder postcardAbout two weeks later,a second packet containing the survey and a reminder letter About three weeks later,a third appeal postcard,Steps of item Validation,Iterative Refinement,4.Expert Panel Review(Summative),Similar questions as in Step 1(Formative review)Same or different panel of expertsRatings and alignment collected after items are fully refinedResults of summative expert panel review provide evidence of alignment of items with standards/curriculum,content validity,and grade-level appropriatenessThis could be reported in technical documentation,Steps of item Validation,Iterative Refinement,Creating a Validity Argument,Integrates all evidence into a coherent account of the degree to which existing evidence and theory support the intended interpretation of test scores,For a Sound Validity Argument,at Minimum,Pay Attention to,Activity#5Conduct Think-Aloud,Break into groups of 3 and select roles1 interviewer1 interviewee1 observer to complete observation recording sheetSelect set of 2 itemsConduct think-alouds.Interviewer and observers take notes on the form in the protocol.Repeat two more times,switching roles,with new items.Revise your own items.Following,we will have a discussion aboutInsights about development of assessment itemsQuestions and challenges,Be the observer for your own items!,Activity#5Think-Aloud Pointers,Find out how long problems take to doUncover issues of item clarity and level of difficultyDerive a model of the knowledge and thinking that the students engage when solving each problem.In observation notes,describe:How problems are solved,focusing on the underlying knowledge,skills,and structures of item performanceActions,thought processes,and strategies,Activity#5Think-Aloud Pointers,Interviewers SHOULDPrompt the teacher to keep talkingAsk clarifying questions about what teachers are saying(but not as scaffolding)Interviewers SHOULD NOTHelp teachers in any way during the interview(e.g.,no hints,tips,or scaffolding).Be sure to avoid unintentional hints by being more encouraging when answers are correct.,Steps of item Validation,Iterative Refinement,Some Useful References,ValidationAERA,APA,&NCME(1999).Standards for educational and psychological testing.Washington,DC:AERA.Baxter,G.P.,Shavelson,R.J.,Herman,S.J.,Brown,K.A.,&Valadez,J.R.(1993).Mathematics performance assessment:Technical quality and diverse student impact.Journal for Research in Mathematics Education,24(3),190-216.Cronbach,L.J.(1971).Test validation.In R.L.Thorndike(Ed.),Educational measurement(2nd ed.,pp.443-507).Washington,DC:American Council on Education.Hoag,R.D.,Meginbir,L.,Khan,Y.,&Weatherall,D.(1985).A multitrait-multimethod analysis of the Preschool Behavior Questionnaire.Journal of Abnormal Child Psychology,13,119-127.Mehta,P.D.,Foorman,B.R.,Branum-Martin,L.,&Taylor,W.P.(2005).Literacy and a unidimensional multilevel construct:Validation,sources of influence,and implications in a longitudinal study in Grades 1 to 4.Scientific Studies of Reading,9,85-116.,Some Useful References,Validation contdMessick,S.(1989).Validity.(In R.L.Linn(Ed.),Educational measurement(3rd ed.,pp.13103).Messick,S.(1994).The interplay of evidence and consequences in the validation of performance assessments.Educational Researcher,23(2),13-23.Pellegrino,J.,Chudowsky,N.,Glaser,R.(Eds.).(2001).Knowing what students know:The science and design of educational assessment.Washington,DC:National Academy Press.Tremblay,R.E.,Vitaro,F.,Gagnon,C.,Piche,C.&Royer,N.(1992).A prosocial scale for the Preschool Behavior Questionnaire:Concurrent and predictive correlates.International Journal of Behavioral Development,15,227-245.Weir,K.,&Duveen,G.(1981).Further development and validation of the Prosocial Behavior Questionnaire for use by teachers.Journal of Child Psychology and Psychiatry,22,357-374.,Some Useful References,Expert Panel ReviewWebb,N.L.(2002).Alignment:Powerful tool for focusing instruction,curricula,and assessment.Presentation at the CCSSO State Collaborative on Assessment and Students Standards,San Diego,CA.Webb,N.L.(2005).Alignment,depth of knowledge,and change.Paper presented at the 50th Annual Meeting of the Florida Educational Research Association,Miami,FL.Think-AloudsEricsson,K.A.,&Simon,H.A.(1993).Protocol Analysis:Verbal reports on data.Cambridge,MA:MIT Press.Flaherty,E.G.(1974).The thinking aloud technique and problem-solving ability.Journal of Educational Research,68,223-225.PsychometricsCrocker,L.,&Algina,J.(1986).Introduction to classical&modern test theory.Orlando,FL:Harcourt Brace Jovanovich,Inc.,