经济资本模型验证方法.ppt
Validation of Economic Capital Models:State of the Practice,Supervisory Expectations and Results from a Bank Study,Michael Jacobs,Ph.D.,CFASenior Financial Economist/Credit Risk Analysis DivisionU.S.Office of the Comptroller of the CurrencyRisk Conference on Economic Capital,NYC,February 2010The views expressed herein are those of the author and do not necessarily represent the views of the Office of the Comptroller of the Currency or the Department of the Treasury.,Outline,Introduction,Background and MotivationFitness for Use of Economic Capital(EC)ModelsProviding Confidence Regarding EC Model AssumptionsAssessing the Value of Validation MethodologiesQualitative ApproachesQuantitative ApproachesSupervisory Concerns and ExpectationsLimitations of ValidationEffective Reporting of Model Outputs Technical Challenges in Testing the Accuracy of EC ModelsAssessing Accuracy in the Tails of the Loss DistributionExample:Alternative Models for Risk Aggregation,Introduction,Background and Motivation,The validation of EC models is at a very preliminary stageInternal capital adequacy assessment process(ICAAP):not a model for EC,but an overall framework for assessing if EC is appropriateEC may be a quantitative component of ICAAP,but it is not required of all banks by supervisors(only the largest)Validation of the quantitative EC component of ICAAP,if there is one,is a component of ICAAP EC models can be complex,having many components,and it may not be immediately obvious that a such model works satisfactorilyModels may embody assumptions about relationships amongst or behavior of variables that may not always hold(e.g.,stress)Validation can provide a degree of confidence that assumptions are appropriate,increasing the confidence of users in the model outputsAdditionally,validation can be also useful in identifying the limitations of EC models(i.e.,where embedded assumptions do not fit reality),Introduction,Background and Motivation(continued),There exists a wide range of validation techniques,each providing evidence regarding only some of the desirable properties of a model Such techniques are powerful in some areas(risk sensitivity)but not in others(accuracy of absolute EC or quantile estimator)A range of validation techniques can provide more substantial evidence for or against the performance of the modelParticularly in an environment of good controls and governanceThere appears to be scope for the industry to improve the validation practices that shed light on the overall calibration of modelsParticularly in cases where assessment of overall capital is an important application of the model,Fitness for Purpose of Economic Capital Models,In some cases the term validation is used exclusively to refer to statistical ex post validation(e.g.,backtesting of a VaR)In other cases it is seen as a broader but still quantitative process that also incorporates evidence from the model development stage Herein,“validation”is meant broadly,meaning all the processes that provide evidence-based assessment of a models fitness for purpose This assessment might extend to the management and systems environment within which the model is operated It is advisable that validation processes are designed alongside development of the models,rather than chronologicallyThis interpretation of validation is consistent with the Basel Committee(2005)in relation to the Basel II FrameworkHowever,that was phrased in terms of the IRB parameters&developed in the context of assessment of risk estimates for use in minimum capital requirementsValidation of EC differs to an IRB model as the output is a distribution rather than a single predicted forecast against which actual outcomes may be compared,Fitness for Purpose of EC Models(continued),EC are conceptually similar to VaR models,but several differences force validation methods to differ in practice from those used in VaRLong time horizon,high confidence levels,and the scarcity of dataFull internal EC models are not used for Pillar 1 minimum capital requirements,so fitness for purpose needs to cover a range of usesMost of which and perhaps all these uses are internal to the firm in questionNote that EC models and regulatory capital serve different objectives&may reasonably differ in some details of implementationBCBSs Validation Principle 1 refers to predictive ability of credit rating systems,an emphasis on performance of model forecasts The natural evolution of this principle for EC is that validation is concerned with the predictive properties of those modelsI.e.,embody forward-looking estimates of risk&their validation involves assessing those estimates,so this related principle remains appropriate Broadly interpreted validation processes set out herein in different ways all provide insight into the predictive ability of EC model,Providing Confidence Regarding EC Model Assumptions,Properties of an EC model that can be assessed using powerful tools,and hence that are capable of robust assessment,include:Integrity of model implementation Degree to which grounded in historical experienceSensitivity to risk&to the external environment Good marginal propertiesRank ordering&relative quantification Properties of an EC model for which only weaker validation processes are available include:Conceptual soundness&validity of assumptionsDegree to which model is forward-lookingAbsolute risk quantification&predictive accuracy of risk estimateIt is important to stress the power of individual tests&acknowledge that views as to strength and weakness are likely to differ,Providing Confidence Regarding EC Model Assumptions(cont.),There is great difficulty in validating conceptual soundness of an EC model due to many untestable or hard-to-test assumptions made:Family of statistical distributions for risk factorsEconomic processes driving default or loss(e.g.,observable vs.latent)Dependency structure among risks or losses(e.g.,copulae)Behavior of management or economic agents&how these vary over time Some EC models are of risk aggregation models where estimates for individual categories are combined to generate a single risk figureThere may be no best or unique way to do this aggregationSince many of these assumptions may be untestable,it may be impossible to be certain that a model is conceptually sound While the conceptual underpinnings may appear coherent and plausible,they may in practice be no more than untested hypothesesOpinions may reasonably differ about the strength or weakness of any particular process in respect of any given property,Validation of EC Models:Introduction to Range of Practice,While we will describe the types of validation processes that are in use or could be used,note that the list is not comprehensiveWe do not suggest that all techniques should or could be used by banks We wish to demonstrate that there is a wide range of techniques potentially covered by our broad definition of validationThis is creating a layered approach,the more(fewer)of which that can be provided,the more(less)comfort that validation is able to provide evidence for or against the performance of the model Each validation process provides evidence for(or against)only some of the desirable properties of a model The list presented below moves from the more qualitative to the more quantitative validation processes,and the extent of use is briefly discussed,Validation of EC Models:Range of Practice in Qualitative Approaches,The philosophy of the use test as incorporated into the Basel II framework:if a bank is actually using its risk measurement systems for internal purposes,then we can place more reliance on itApplying the use test successfully will entail gaining a careful understanding of which model properties are being used and which are notBanks tend to subject their models to some form of qualitative review process,which could entail:Review of documentation or development workDialogue with model developers or model managersReview and derivation of any formulae or algorithmsComparison to other firms or with publicly available information Qualitative review is best able to answer questions such as:Does the model work in theory?Does the model incorporate the right risk drivers?Is any theory underpinning it conceptually well-founded?Is the mathematics of the model right?,Range of Practice in Qualitative Approaches to Validation(continued),Extensive systems implementation testing is standard for production-level risk measurement systems prior to implementationE.g.,user acceptance testing,checking of model code,etc.These processes could be viewed as part of the overall validation effort,since they would assist in evaluating whether the model is implemented with integrityManagement oversight is the involvement of senior management in the validation processE.g.,reviewing output from the model&using the results in business decisions Senior management knowing how the model is used&outputs are interpreted This should take account of the specific implementation framework adopted and the assumptions underlying the model and its parameterizationData quality checks refer to the processes designed to provide assurance of the completeness,accuracy and appropriateness of data used to develop,validate and operate the model E.g.,Review of:data collection and storage,data cleaning of errors,extent of proxy data,processes that need to be followed to convert raw data into suitable model inputs,and verification of transaction data such as exposure levels While not traditionally viewed by the industry as a form of validation,increasingly forming a major part of supervisory thinking,Range of Practice in Qualitative Approaches to Validation(concluded),As all models rest on premises of various kinds,varying in the degree to which obvious,we have examination of assumptionsCertain aspects of an EC model are built-in and cannot be altered without fundamentally changing the model To illustrate,these assumptions could be about:Fixed model parameters(PDs,correlations or recovery rates)Distributional assumptions(margins,copulae&shape of tail distributions)Behavior of senior management or of customers Some banks go through a deliberate process of detailing the assumptions underpinning their models,including examination of:Impact on model outputsLimitations that the assumptions place on model usage and applicability,Range of Practice in Quantitative Approaches to Validation:Inputs,A complete validation of an EC model would involve the inputs and parameters,both statistically estimated and notExamples of estimated(assumed)parameters are the main IRB parameters(PD or LGD)(correlations,PD in a low default portfolio)Techniques could include assessing parameters against:Historical data through replication of estimatorsOutcomes over time through backtestingMarket-implied parameters(e.g.,implied vol or correlation,CDS spreads for PD)Materiality through sensitivity testingThis testing of input parameters could complement examination of assumptions previously&sensitivity testing to described laterHowever,that checking of model inputs is unlikely to be fully satisfactory since,every model is based on underlying assumptionsThe more sophisticated the model,the more susceptible to model error,so checking input parameters will not help here However,model accuracy and appropriateness can be assessed,at least to some degree,using the processes described in this section,Range of Practice in Quantitative Validation:Model Replication,Model replication is useful technique that attempts to replicate EC model results obtained by the bank This could use independently developed algorithms or data sources,but in practice replication might leverage a banks existing processes E.g.,run a model of the same type or class on a the banks data-set However,but once the either the original or test model has been validatedThis technique and the questions that often arise in implementing replication can help identify if:Definitions&algorithms the bank claims to use correctly are understood by staff who develop,maintain,operate and validate the model The bank is using in practice the modeling framework that it purports to Computer code is correct,efficient and well-documented Data claimed to be used by the bank to obtain its results is in fact being usedHowever,this technique is rarely sufficient to validate models,and in practice there is little evidence of it being used by banks for either validation or to explore the degree of accuracy of their models Note that replication simply by re-running a set of algorithms to produce an identical set of results would not be sufficient model validation due diligence,Range of Practice in Quantitative Validation:Benchmarking,Benchmarking the comparison of a banks EC model to alternative models on the banks portfolio E.g.,to a vendor model after standardization of parametersAmong the most commonly used forms of quantitative validation used internallyA limitation of benchmarking is it only provides relative assessments and provides little assurance that any model accurately reflects reality or about the absolute levels of capital Therefore,as a validation technique,benchmarking is limited to providing comparison of one model against another or one calibration to others,but not testing against realityIt is therefore difficult to assess the degree of comfort provided by such benchmarking methods,as they may only be capable of providing broad comparisons confirming that input parameters or model outputs are broadly comparable,Range of Practice in Quantitative Validation:Benchmarking(continued),There may be good reasons why models produce outliers in benchmarking,all of which complicate interpretation of the results:May be designed to perform well under differing circumstancesMay be more or less conservatively parameterizedMay differ in their economic foundations Comparisons of internal EC are made with varied alternatives:Industry survey resultsRating agency or industry-wide modelsConsultancy marketed models Academic papersRegulatory capital models,Range of Practice in Quantitative Validation:Hypothetical Portfolios,Hypothetical portfolio testing is an examination of either different banks EC models on a reference portfolio,or different banks EC output from a given reference modelThis is typically a either a reference model or portfolio external to any one bankFrom a supervisory perspective:permits identification of models that produce outliers amongst a set of banks A“model risk management”toolAlternatively,this helps supervisors identify banks that are outliers in risk with respect to a reference model A“bank portfolio risk management”toolIn either case this means comparison across banks models against the same reference portfolio(external to the bank)or of banks themselves(their EC for a given reference model)Capable of addressing similar questions to benchmarking,but by different means The technique is a powerful one and can be adapted to analyze many of the p
