经济资本模型验证方法.ppt

《经济资本模型验证方法.ppt》由会员分享，可在线阅读，更多相关《经济资本模型验证方法.ppt（37页珍藏版）》请在三一办公上搜索。

1、Validation of Economic Capital Models:State of the Practice,Supervisory Expectations and Results from a Bank Study,Michael Jacobs,Ph.D.,CFASenior Financial Economist/Credit Risk Analysis DivisionU.S.Office of the Comptroller of the CurrencyRisk Conference on Economic Capital,NYC,February 2010The vie

2、ws expressed herein are those of the author and do not necessarily represent the views of the Office of the Comptroller of the Currency or the Department of the Treasury.,Outline,Introduction,Background and MotivationFitness for Use of Economic Capital(EC)ModelsProviding Confidence Regarding EC Mode

3、l AssumptionsAssessing the Value of Validation MethodologiesQualitative ApproachesQuantitative ApproachesSupervisory Concerns and ExpectationsLimitations of ValidationEffective Reporting of Model Outputs Technical Challenges in Testing the Accuracy of EC ModelsAssessing Accuracy in the Tails of the

4、Loss DistributionExample:Alternative Models for Risk Aggregation,Introduction,Background and Motivation,The validation of EC models is at a very preliminary stageInternal capital adequacy assessment process(ICAAP):not a model for EC,but an overall framework for assessing if EC is appropriateEC may b

5、e a quantitative component of ICAAP,but it is not required of all banks by supervisors(only the largest)Validation of the quantitative EC component of ICAAP,if there is one,is a component of ICAAP EC models can be complex,having many components,and it may not be immediately obvious that a such model

6、 works satisfactorilyModels may embody assumptions about relationships amongst or behavior of variables that may not always hold(e.g.,stress)Validation can provide a degree of confidence that assumptions are appropriate,increasing the confidence of users in the model outputsAdditionally,validation c

7、an be also useful in identifying the limitations of EC models(i.e.,where embedded assumptions do not fit reality),Introduction,Background and Motivation(continued),There exists a wide range of validation techniques,each providing evidence regarding only some of the desirable properties of a model Su

8、ch techniques are powerful in some areas(risk sensitivity)but not in others(accuracy of absolute EC or quantile estimator)A range of validation techniques can provide more substantial evidence for or against the performance of the modelParticularly in an environment of good controls and governanceTh

9、ere appears to be scope for the industry to improve the validation practices that shed light on the overall calibration of modelsParticularly in cases where assessment of overall capital is an important application of the model,Fitness for Purpose of Economic Capital Models,In some cases the term va

10、lidation is used exclusively to refer to statistical ex post validation(e.g.,backtesting of a VaR)In other cases it is seen as a broader but still quantitative process that also incorporates evidence from the model development stage Herein,“validation”is meant broadly,meaning all the processes that

11、provide evidence-based assessment of a models fitness for purpose This assessment might extend to the management and systems environment within which the model is operated It is advisable that validation processes are designed alongside development of the models,rather than chronologicallyThis inter

12、pretation of validation is consistent with the Basel Committee(2005)in relation to the Basel II FrameworkHowever,that was phrased in terms of the IRB parameters&developed in the context of assessment of risk estimates for use in minimum capital requirementsValidation of EC differs to an IRB model as

13、 the output is a distribution rather than a single predicted forecast against which actual outcomes may be compared,Fitness for Purpose of EC Models(continued),EC are conceptually similar to VaR models,but several differences force validation methods to differ in practice from those used in VaRLong

14、time horizon,high confidence levels,and the scarcity of dataFull internal EC models are not used for Pillar 1 minimum capital requirements,so fitness for purpose needs to cover a range of usesMost of which and perhaps all these uses are internal to the firm in questionNote that EC models and regulat

15、ory capital serve different objectives&may reasonably differ in some details of implementationBCBSs Validation Principle 1 refers to predictive ability of credit rating systems,an emphasis on performance of model forecasts The natural evolution of this principle for EC is that validation is concerne

16、d with the predictive properties of those modelsI.e.,embody forward-looking estimates of risk&their validation involves assessing those estimates,so this related principle remains appropriate Broadly interpreted validation processes set out herein in different ways all provide insight into the predi

17、ctive ability of EC model,Providing Confidence Regarding EC Model Assumptions,Properties of an EC model that can be assessed using powerful tools,and hence that are capable of robust assessment,include:Integrity of model implementation Degree to which grounded in historical experienceSensitivity to

18、risk&to the external environment Good marginal propertiesRank ordering&relative quantification Properties of an EC model for which only weaker validation processes are available include:Conceptual soundness&validity of assumptionsDegree to which model is forward-lookingAbsolute risk quantification&p

19、redictive accuracy of risk estimateIt is important to stress the power of individual tests&acknowledge that views as to strength and weakness are likely to differ,Providing Confidence Regarding EC Model Assumptions(cont.),There is great difficulty in validating conceptual soundness of an EC model du

20、e to many untestable or hard-to-test assumptions made:Family of statistical distributions for risk factorsEconomic processes driving default or loss(e.g.,observable vs.latent)Dependency structure among risks or losses(e.g.,copulae)Behavior of management or economic agents&how these vary over time So

21、me EC models are of risk aggregation models where estimates for individual categories are combined to generate a single risk figureThere may be no best or unique way to do this aggregationSince many of these assumptions may be untestable,it may be impossible to be certain that a model is conceptuall

22、y sound While the conceptual underpinnings may appear coherent and plausible,they may in practice be no more than untested hypothesesOpinions may reasonably differ about the strength or weakness of any particular process in respect of any given property,Validation of EC Models:Introduction to Range

23、of Practice,While we will describe the types of validation processes that are in use or could be used,note that the list is not comprehensiveWe do not suggest that all techniques should or could be used by banks We wish to demonstrate that there is a wide range of techniques potentially covered by o

24、ur broad definition of validationThis is creating a layered approach,the more(fewer)of which that can be provided,the more(less)comfort that validation is able to provide evidence for or against the performance of the model Each validation process provides evidence for(or against)only some of the de

25、sirable properties of a model The list presented below moves from the more qualitative to the more quantitative validation processes,and the extent of use is briefly discussed,Validation of EC Models:Range of Practice in Qualitative Approaches,The philosophy of the use test as incorporated into the

26、Basel II framework:if a bank is actually using its risk measurement systems for internal purposes,then we can place more reliance on itApplying the use test successfully will entail gaining a careful understanding of which model properties are being used and which are notBanks tend to subject their

27、models to some form of qualitative review process,which could entail:Review of documentation or development workDialogue with model developers or model managersReview and derivation of any formulae or algorithmsComparison to other firms or with publicly available information Qualitative review is be

28、st able to answer questions such as:Does the model work in theory?Does the model incorporate the right risk drivers?Is any theory underpinning it conceptually well-founded?Is the mathematics of the model right?,Range of Practice in Qualitative Approaches to Validation(continued),Extensive systems im

29、plementation testing is standard for production-level risk measurement systems prior to implementationE.g.,user acceptance testing,checking of model code,etc.These processes could be viewed as part of the overall validation effort,since they would assist in evaluating whether the model is implemente

30、d with integrityManagement oversight is the involvement of senior management in the validation processE.g.,reviewing output from the model&using the results in business decisions Senior management knowing how the model is used&outputs are interpreted This should take account of the specific implemen

31、tation framework adopted and the assumptions underlying the model and its parameterizationData quality checks refer to the processes designed to provide assurance of the completeness,accuracy and appropriateness of data used to develop,validate and operate the model E.g.,Review of:data collection an

32、d storage,data cleaning of errors,extent of proxy data,processes that need to be followed to convert raw data into suitable model inputs,and verification of transaction data such as exposure levels While not traditionally viewed by the industry as a form of validation,increasingly forming a major pa

33、rt of supervisory thinking,Range of Practice in Qualitative Approaches to Validation(concluded),As all models rest on premises of various kinds,varying in the degree to which obvious,we have examination of assumptionsCertain aspects of an EC model are built-in and cannot be altered without fundament

34、ally changing the model To illustrate,these assumptions could be about:Fixed model parameters(PDs,correlations or recovery rates)Distributional assumptions(margins,copulae&shape of tail distributions)Behavior of senior management or of customers Some banks go through a deliberate process of detailin

35、g the assumptions underpinning their models,including examination of:Impact on model outputsLimitations that the assumptions place on model usage and applicability,Range of Practice in Quantitative Approaches to Validation:Inputs,A complete validation of an EC model would involve the inputs and para

36、meters,both statistically estimated and notExamples of estimated(assumed)parameters are the main IRB parameters(PD or LGD)(correlations,PD in a low default portfolio)Techniques could include assessing parameters against:Historical data through replication of estimatorsOutcomes over time through back

37、testingMarket-implied parameters(e.g.,implied vol or correlation,CDS spreads for PD)Materiality through sensitivity testingThis testing of input parameters could complement examination of assumptions previously&sensitivity testing to described laterHowever,that checking of model inputs is unlikely t

38、o be fully satisfactory since,every model is based on underlying assumptionsThe more sophisticated the model,the more susceptible to model error,so checking input parameters will not help here However,model accuracy and appropriateness can be assessed,at least to some degree,using the processes desc

39、ribed in this section,Range of Practice in Quantitative Validation:Model Replication,Model replication is useful technique that attempts to replicate EC model results obtained by the bank This could use independently developed algorithms or data sources,but in practice replication might leverage a b

40、anks existing processes E.g.,run a model of the same type or class on a the banks data-set However,but once the either the original or test model has been validatedThis technique and the questions that often arise in implementing replication can help identify if:Definitions&algorithms the bank claim

41、s to use correctly are understood by staff who develop,maintain,operate and validate the model The bank is using in practice the modeling framework that it purports to Computer code is correct,efficient and well-documented Data claimed to be used by the bank to obtain its results is in fact being us

42、edHowever,this technique is rarely sufficient to validate models,and in practice there is little evidence of it being used by banks for either validation or to explore the degree of accuracy of their models Note that replication simply by re-running a set of algorithms to produce an identical set of

43、 results would not be sufficient model validation due diligence,Range of Practice in Quantitative Validation:Benchmarking,Benchmarking the comparison of a banks EC model to alternative models on the banks portfolio E.g.,to a vendor model after standardization of parametersAmong the most commonly use

44、d forms of quantitative validation used internallyA limitation of benchmarking is it only provides relative assessments and provides little assurance that any model accurately reflects reality or about the absolute levels of capital Therefore,as a validation technique,benchmarking is limited to prov

45、iding comparison of one model against another or one calibration to others,but not testing against realityIt is therefore difficult to assess the degree of comfort provided by such benchmarking methods,as they may only be capable of providing broad comparisons confirming that input parameters or mod

46、el outputs are broadly comparable,Range of Practice in Quantitative Validation:Benchmarking(continued),There may be good reasons why models produce outliers in benchmarking,all of which complicate interpretation of the results:May be designed to perform well under differing circumstancesMay be more

47、or less conservatively parameterizedMay differ in their economic foundations Comparisons of internal EC are made with varied alternatives:Industry survey resultsRating agency or industry-wide modelsConsultancy marketed models Academic papersRegulatory capital models,Range of Practice in Quantitative

48、 Validation:Hypothetical Portfolios,Hypothetical portfolio testing is an examination of either different banks EC models on a reference portfolio,or different banks EC output from a given reference modelThis is typically a either a reference model or portfolio external to any one bankFrom a supervis

49、ory perspective:permits identification of models that produce outliers amongst a set of banks A“model risk management”toolAlternatively,this helps supervisors identify banks that are outliers in risk with respect to a reference model A“bank portfolio risk management”toolIn either case this means com

50、parison across banks models against the same reference portfolio(external to the bank)or of banks themselves(their EC for a given reference model)Capable of addressing similar questions to benchmarking,but by different means The technique is a powerful one and can be adapted to analyze many of the p