Tag: Testing

How AVMetrics Tests AVMs Using our New Testing Methodology

Testing an AVM’s accuracy can actually be quite tricky. You might think that you simply compare an AVM valuation to a corresponding actual sales price – technically a fair sale on the open market – but that’s just the beginning. Here’s why it’s hard:

  • You need to get those matching values and benchmark sales in large quantities – like hundreds of thousands – if you want to cover the whole nation and be able to test different price ranges and property types (AVMetrics compiled close to 4 million valid benchmarks in 2021).
  • You need to scrub out foreclosure sales and other bad benchmarks.
  • And perhaps most difficult, you need to test the AVMs’ valuations BEFORE the corresponding benchmark sale is made public. If you don’t, then the AVM builders, whose business is up-to-date data, will incorporate that price information into their models and essentially invalidate the test. (You can’t really have a test where the subject knows the answer ahead of time.)

Here’s a secret about that third part: some of the AVM builders are also the same companies that are the premier providers of real estate data, including MLS data. What if the models are using MLS data listing price feeds to “anchor” their models based on the listing price of a home? If they are the source of the data, how can you test them before they get the data? We now know how.

We have spent years developing and implementing a solution because we wanted to level the playing field for every AVM builder and model. We ask each AVM to value every home in America each month. They each provide +/-110 million AVM valuations each month. There are over 25 different commercially available AVMs that we test regularly. That adds up to a lot of data.

A few years ago, it wouldn’t have been feasible to accumulate data at that scale. But now that computing and storage costs make it feasible, the AVM builders themselves are enthusiastic about it. They like the idea of a fair and square competition. We now have valuations for every property BEFORE it’s sold, and in fact, before it’s listed.

As we have for well over a decade now, we gather actual sales to use as the benchmarks against which to measure the accuracy of the AVMs.  We scrub these actual sales prices to ensure that they are for arm’s-length transactions between willing buyers and sellers — the best and most reliable indicator of market value. Then we use proprietary algorithms to match benchmark values to the most recent usable AVM estimated value. Using our massive database, we ensure that each model has the same opportunity to predict the sales price of each benchmark.

AVMetrics next performs a variety of statistical analyses on the results, breaking down each individual market, each price range, and each property type, and develops results which characterize each model’s success in terms of precision, usability, error and accuracy.  AVMetrics analyzes trends at the global, market and individual model levels. We also identify where there are strengths and weaknesses and where performance improved or declined.

In the spirit of continuous improvement, AVMetrics provides each model builder an anonymized comprehensive comparative analysis showing where their models stack up against all of the models in the test; this invaluable information facilitates their ongoing efforts to improve their models.

Finally, in addition to quantitative testing, AVMetrics circulates a comprehensive vendor questionnaire semi-annually.  Vendors that wish to participate in the testing process answer roughly 100 parameter, data, methodology, staffing and internal testing questions for each model being tested.  These enable AVMetrics and our clients to understand model differences within both testing and production contexts. The questionnaire also enables us and our clients to satisfy certain regulatory requirements describing the evaluation and selection of models (see OCC 2010-42 and 2011-12).

 

 

 

Four Points to Consider Before Outsourcing AVM Validation

AVMs are not only fairly accurate, they are also affordable and easy to use.  Unfortunately, using them in a “compliant” fashion is not as easy.  Regulatory Bulletins OCC 2010-42 and OCC 2011-12 describe a lot of requirements that can be challenging for a regional or community institution:

  1. ongoing independent testing and validation and documentation of testing;
  2. understanding each AVM model’s conceptual and methodological soundness;
  3. documenting policies and procedures that define how to use AVMs and when not to use AVMs;
  4. establishing targets for accuracy and tolerances for acceptable discrepancies. 

The extent to which these requirements are applied by your regulator is most likely proportional to the extent to which AVMs are used within your organization; if AVMs are used extensively, regulatory oversight will likely demand much tighter adherence to the requirements as well as much more comprehensive policies and procedures.

Although compliance itself is not a function that can be outsourced (it is the sole responsibility of the institution), elements of the regulatory requirements can be effectively handled outside the organization through outsourcing.  As an example, the first bullet point, “ongoing independent testing and validation and documentation of testing,” requires resources with the competencies and influences to effectively challenge AVM models. In addition, the “independent” aspect is challenging to accomplish unless a separate department within the institution is established that does not report up through the product and/or procurement verticals (e.g. similar to Audit, or Model Risk Management, etc.). Whether your institution is a heavy AVM user or not, the good news is that finding the right third-party to outsource to will facilitate all of the bullet points above:

  1. documentation is included as part of an independent testing and validation process and it can be incorporated into your policies and procedures;
  2. the results of the testing will help you shape your understanding of where and when AVMs can and cannot be used;
  3. the results of the testing will inform your decisions regarding the accuracy and performance thresholds that fit within your institution’s risk appetite. In addition,
  4. an outsourced specialist may also be able to provide various levels of consultation assistance in areas where you may not have the internal expertise.

Before deciding whether outsourcing makes sense for you, here are some potential considerations. If you can answer “no” to all of these questions, then outsourcing might be a good option, especially if you don’t have an independent Analytics unit in-house that has the resource bandwidth to accommodate the AVM testing and validation processes:

  1. Is this process strategically critical?  I.e., does your validation of AVMs benefit you competitively in a tangible way?
  2. If your validation of AVMs is inadequate, can this substantially affect your reputation or your position within the marketplace?
  3. Is outsourcing impractical for any reason?  I.e., are there other business functions that preclude separating the validation process?  
  4. Does your institution have the same data availability and economies of scale as a specialist?

The Way Forward

Here are some suggestions on how to go about preparing yourself for selecting your outsource partner:

  1. Specify what you need outsourced.  If you already have Policies and Procedures documented and processes in place, there may be no need to look for that capability, but there will necessarily still be the need to incorporate any testing and validation results into your existing policies and procedures.  If you have previously done extensive evaluations of the AVMs that you use, in terms of their models’ conceptual soundness and outcomes analysis, there’s no need to contract for that, either.  See our article on Regulatory Oversight to get some ideas about those requirements.
  2. Identify possible partners, such as AVMetrics, and evaluate their fit.  Here’s what to look for:
    • Expertise.  It’s a technical job, requiring a fair amount of analysis and a tremendous amount of knowledge about regulatory requirements in general, and specifically knowledge relative to AVMs; check the résumés of the experts with whom you plan to partner.
    • Independence.  A vendor who also sells, builds, resells, uses or advocates for certain AVMs may be biased (or may appear to be biased) in auditing them; validation must be able to “effectively challenge” the models being tested.
    • Track record.  Stable partners are better, and a long term relationship lowers the cost of outsourcing; so look for a partner with a successful track record in performing AVM validations.
  3. Open up conversations with potential partners early because the process can take months, particularly if policies and procedures need to be developed; although validations can be successfully completed in a matter of days, that is not the norm.
  4. Make sure your staff has enough familiarity with the regulatory requirements so as to be able to oversee the vendor’s work; remember that the responsibility for compliance is ultimately on you. Make sure the vendor’s process and results are clearly and comprehensively documented and then ensure that Internal Audit and Compliance are part of that oversight.  “Outsource” doesn’t mean “forget about it;” thorough and complete understanding and documentation is part of the requirements.
  5. Have a plan for ongoing compliance, whether it is to transition to internal resources or to retain vendors indefinitely.  Set expectations for the frequency of the validation process, which regulations require to be at least annually or more often, commensurate with the extent of your AVM usage.

In Conclusion

AVM testing and validation is only one component in your overall Valuation and evaluation program. Unlike Appraisals and some other forms of collateral valuation, AVMs, by their nature as a quantitative predictive model, lend themselves to just the type of statistically-based outcomes analysis the regulators set forth. Recognizing this, elements of the requirements can be an outsourced process, but it must be a compliment to enterprise-wide policies and practices around the permissible, safe and prudent use of valuation tools and technologies.

The process of validating and documenting AVMs may seem daunting at first, but for the past 10 years AVMetrics has been providing ease-of-mind for our customers, whether as the sole source of an outsourced testing and validation process (that tests every commercial AVM four times a year), or as a partner in transitioning the process in-house.  Our experience, professional resources and depth of data have enabled us to standardize much of the processing while still providing the customization every institution needs.  And probably one of the most critical boxes you can check off when outsourcing with AVMetrics is the very large one that requires independence. It also bears mentioning that having been around as long as we have, our customers have generally all been through at least one round of regulatory scrutiny, and the AVMetrics process has always passed regulatory muster.  Regulatory reviews already present enough of a challenge, so having a partner with established credentials is critical for a smooth process.

Coronavirus Response – Safety and Service

AVMetrics is committed to providing our service consistently while protecting the health and safety of our customers, employees, partners and vendors.

We are closely monitoring the coronavirus pandemic. We have already taken several steps to prepare for and limit potential impacts to our customers and employees. We have activated our Disaster Recovery Plan and will be cooperating with guidance from the Centers for Disease Control (CDC) and the World Health Organization (WHO).

Regarding the support of our clients, we are committed to continuing to provide our services without interruption. The effects of the coronavirus on market movements is very uncertain at this time. Given the integrated nature of the flow of data from government entities such as County recorders offices to data aggregators and to AVM vendors, we expect that the flow of data will be little changed. However, given the emphasis on social distancing and quarantine for workers, public and private, this too is an unknown.  We will remain in close communication with our clients and partners who provide AVMs so we can communicate any developments in AVM testing participation and results.

Our DR Plan includes measures to: 1) prevent transmission of the highly infectious coronavirus which poses a significant risk to the world’s population; 2) minimize all illness among employees, customers, contractors, and visitors; 3) maintain our business functions and testing processes; and 4) minimize any financial impact on ourselves and our customers, vendors and partners.

AVMetrics has implemented our employee work-from-home plans, flexible work schedules and other adaptive procedures to support social distancing recommendations.

AVMetrics’ planning includes multiple scenarios, including significant absenteeism, failure of critical information technology systems or infrastructure components, loss of facility (workspace) and loss of vendor/partners for an extended period of time.

In response, AVMetrics has taken these and other measures:

    • Work-from-home protocols;
    • Restriction of non-essential employee travel;
    • Use of virtual meetings where feasible (all in-person meetings are being kept short in duration and social distancing is encouraged in the meeting);
    • Avoidance of large gatherings, industry events, etc.;
    • Instructions to employees to stay home if they are sick;
    • Visitation restrictions and protocols for all vendors and visitors into company workspaces; and
    • Posting of office flyers about social distancing and how employees can protect themselves (e.g., frequently wash hands with soap and water or utilize hand sanitizer; often wipe highly touched surfaces, such as desks and doorknobs; and avoid touching your eyes, nose, and mouth).

AVMetrics is committed to supporting the health and safety of our employees, customers, partners and the communities in which we live and work. We will continue to monitor the situation, modify our protocols as needed based on evolving recommendations of the CDC and WHO, as well as other industry experts, and communicate via email, on our blog and on our LinkedIn page.