AVMetics’ ongoing independent AVM testing continues in 2023 according to the schedule in the link below (updated as of January 3, 2023). Please download the pdf with the revised 2023 testing schedule.
How AVMetrics Tests AVMs Using our New Testing Methodology
Testing an AVM’s accuracy can actually be quite tricky. You might think that you simply compare an AVM valuation to a corresponding actual sales price – technically a fair sale on the open market – but that’s just the beginning. Here’s why it’s hard:
- You need to get those matching values and benchmark sales in large quantities – like hundreds of thousands – if you want to cover the whole nation and be able to test different price ranges and property types (AVMetrics compiled close to 4 million valid benchmarks in 2021).
- You need to scrub out foreclosure sales and other bad benchmarks.
- And perhaps most difficult, you need to test the AVMs’ valuations BEFORE the corresponding benchmark sale is made public. If you don’t, then the AVM builders, whose business is up-to-date data, will incorporate that price information into their models and essentially invalidate the test. (You can’t really have a test where the subject knows the answer ahead of time.)
Here’s a secret about that third part: some of the AVM builders are also the same companies that are the premier providers of real estate data, including MLS data. What if the models are using MLS data listing price feeds to “anchor” their models based on the listing price of a home? If they are the source of the data, how can you test them before they get the data? We now know how.
We have spent years developing and implementing a solution because we wanted to level the playing field for every AVM builder and model. We ask each AVM to value every home in America each month. They each provide +/-110 million AVM valuations each month. There are over 25 different commercially available AVMs that we test regularly. That adds up to a lot of data.
A few years ago, it wouldn’t have been feasible to accumulate data at that scale. But now that computing and storage costs make it feasible, the AVM builders themselves are enthusiastic about it. They like the idea of a fair and square competition. We now have valuations for every property BEFORE it’s sold, and in fact, before it’s listed.
As we have for well over a decade now, we gather actual sales to use as the benchmarks against which to measure the accuracy of the AVMs. We scrub these actual sales prices to ensure that they are for arm’s-length transactions between willing buyers and sellers — the best and most reliable indicator of market value. Then we use proprietary algorithms to match benchmark values to the most recent usable AVM estimated value. Using our massive database, we ensure that each model has the same opportunity to predict the sales price of each benchmark.
AVMetrics next performs a variety of statistical analyses on the results, breaking down each individual market, each price range, and each property type, and develops results which characterize each model’s success in terms of precision, usability, error and accuracy. AVMetrics analyzes trends at the global, market and individual model levels. We also identify where there are strengths and weaknesses and where performance improved or declined.
In the spirit of continuous improvement, AVMetrics provides each model builder an anonymized comprehensive comparative analysis showing where their models stack up against all of the models in the test; this invaluable information facilitates their ongoing efforts to improve their models.
Finally, in addition to quantitative testing, AVMetrics circulates a comprehensive vendor questionnaire semi-annually. Vendors that wish to participate in the testing process answer roughly 100 parameter, data, methodology, staffing and internal testing questions for each model being tested. These enable AVMetrics and our clients to understand model differences within both testing and production contexts. The questionnaire also enables us and our clients to satisfy certain regulatory requirements describing the evaluation and selection of models (see OCC 2010-42 and 2011-12).
Property Inspection Waivers Took Off After the Pandemic Set In
Appraisals are the gold standard when it comes to valuing residential real estate, but they aren’t always necessary. They’re expensive and time-consuming, and in the era of COVID-19, they’re inconvenient. What’s the alternative?
Well, Fannie and Freddie implemented a “Property Inspection Waiver” (PIW) alternative more than a decade ago. However, it’s been slow to catch on.
But now, maybe the tipping point has arrived during the pandemic. Recently published data by Fannie and Freddie show approximately 33% of properties were valued without a traditional appraisal! (Most, if not all, would have used an AVM as part of the appraisal waiver process.) Ed Pinto at AEI’s Housing Center calls it a hockey stick.
So, what changed? Here are some thoughts and hypotheses:
- Guidelines changed a little. We can see in the data that Freddie did almost zero PIWs on cash out loans, but in May that changed, and at lease for LTVS below 70%, they did almost 15,000 cash out loans with no appraisal.
- AVMs changed. Back when PIWs were introduced, AVMs operated in a +/- 10% paradigm. They were more concerned with hit rates than anything else, and they worked best on track homes. But, today they are operating in a +/- 4% world, hit rates are great, and cascades allow lenders to pick the AVM that’s most accurate for the application.
- Borrowers changed. These days, borrowers have grown up with online tools that give them answers. They are more likely to read about their symptoms on WebMD before going to the doctor, and they are more likely to look their home up on Zillow before calling their realtor. In the past, if home was purchased with a low LTV, who was it that required an appraisal? Typically, it was borrowers that wanted the appraisal – more as a safety blanket than anything else. They wanted reassurance that they were not getting ripped off. Today, for some people, Zillow can provide that reassurance without the $500 expense.
- Lenders changed. You would think that they are nimble and adaptable to new opportunities. But where the rubber meets the road, it’s still people talking to customers, and underwriters signing off on loans. If loan officers aren’t aware of the guidelines, they’ll just order an appraisal. Often ordering an appraisal, because it can take so long, is just about one of the first things done in the process, regardless of whether it’s necessary. After all, it’s usually necessary, and it takes SO long (relatively speaking, of course). I have known lenders who required their loan officers to collect money for an appraisal to demonstrate customer commitment. But, lenders are starting to incorporate PIWs into their processes and take advantage of those opportunities to present a loan option with $500 less in costs.
Accurate AVMs are a necessary but not sufficient criteria for PIWs, and now that AVMs are much more accurate, PIWs are much more practical, and we’re seeing much higher adoption.
So now what should we expect going forward? The trend will likely continue. There’s a lot of room left in some of those categories for PIWs to grab a larger share.
If agencies are doing it, everyone else will. If there are lenders not using PIWs to the extent possible, they are going to be at a disadvantage.
2020 Testing Schedule Announced
AVMetics’ ongoing independent AVM testing continues in 2020 according to the schedule in the link below. Please download the pdf with the entire 2020 testing schedule.
AVM Regulatory Outlook
As always, changes are coming to the valuation industry. These changes have been germinating in government and industry for a long time, but they’ve made progress in the last year, and I believe that they’re likely to emerge sometime this year. I expect that we may see more regulatory changes liberalizing the use of AVMs soon.
I think that you’ll come to that same conclusion, too, if I share a couple milestones that I’ve observed and put them together with some insights I’ve gathered from talking to industry leaders.
The first milestone I will highlight was the July 2018 Financial System report by Secretary Mnuchin, which is consistent with the administration’s new attitude towards regulation. The report is far-reaching, and it includes thoughtful commentary about the uses of AVMs (see, for example, page 103-106). It recommends updating FIRREA appraisal requirements to accommodate increased usage of AVMs and hybrids. It also advocates for increased monitoring of AVMs and the application of rigorous market standards. And, it recommends focusing the use of AVMs and hybrids on loan programs with other mitigating risk factors.
The next milestone I will highlight was the proposed change in the de minimis threshold that was put out for comment in November of last year. The change would raise the threshold below which a residential mortgage could be originated with an evaluation, utilizing an AVM in lieu of a traditional appraisal. It would be raised from $250,000 to $400,000.
To those milestones I would add a third data point. Last November I attended the Appraisal Subcommittee roundtable entitled: “The Evolving Real Estate Valuation Landscape.” As part of the of the Federal Financial Institutions Examination Council, the roundtable brought together industry representatives and government officials (see the table below) to discuss real estate valuation.
The day was split into two sessions; the morning and afternoon sessions each began with a panel of industry experts who addressed a series of prepared questions. In addition, there was a roundtable discussion focused on quotes from the July 2018 Financial System report referenced above.
The topic for the morning discussion was “Harmonizing Real Estate Valuation Requirements Across the Federal Government.” This session focused on identifying various federal appraisal statutory and regulatory requirements and exploring opportunities to harmonize those requirements, e.g., VA, FHA, and FHFA all having differing valuation requirements and standards.
The afternoon panel discussion topic was; “The Evolution of Real Estate Valuation” which focused on evolving valuation needs in commercial and mortgage lending. A key area of this session was focused on Alternative Valuation Products inclusive of AVM’s and their increasing used by lenders and the secondary market.
The roundtable discussion started with quotes about AVMs and hybrid valuation products and focused on standards. The group also contemplated how alternative valuation techniques can impact quality and mitigate risk. Finally, one quote that focused on speeding the adoption of technology was discussed.
As I write this six months later, I see the pieces of the puzzle coming together. Obviously, there is momentum behind the increased usage of AVMs, for their independence, increasing accuracy, speed and efficiency. But there is also an implicit concern to avoid opening the door to more risk. I see this being expressed by talk about “standards,” alternative products, such as “hybrids” and increased monitoring.
As I have written elsewhere, I welcome changes that make better use of our valuable and limited resources, namely the appraisers themselves. As AVM quality improves and the number of appraisers shrinks, we should encourage appraisers to be focused on their highest and best use. Their expertise should be focused on the complex, qualitative aspects of property valuation such as the property condition and market and locational influences. They should also be focused on performing complex valuation assignments in non-homogeneous markets. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser, and I expect that Treasury, the FDIC and legislators are moving in this same direction.
Lee Kennedy
| Participants in “The Evolving Real Estate Valuation Landscape” Appraisal Subcommittee, Federal Financial Institutions Examination Council, 2018 | ||
| Government | Trade Organizations | Industry Participants |
| The Appraisal Foundation (TAF) | American Bankers Association | AVMetrics, LLC |
| Association of Appraiser Regulatory Officials (AARO) | American Society of Appraisers | Bank of America |
| Consumer Financial Protection Bureau (4) | American Society of Farm Managers and Rural Appraisers | Clarocity Valuation Services |
| Federal Deposit Insurance Corporation(3) | Appraisal Institute | ClearBox |
| Federal Housing Finance Agency(4) | Homeownership Preservation Foundation | CoreLogic |
| Federal Reserve Board(5) | Independent Community Bankers of America | Cushman & Wakefield Global Services, Inc. |
| Freddie Mac | Mortgage Bankers Association | Farm Credit Mid-America |
| Internal Revenue Service | National Association of Home Builders | First American Mortgage Solutions |
| National Credit Union Administration | National Association of Realtors | Genworth Financial |
| Office of the Comptroller of the Currency (4) | Real Estate Valuation Advocacy Association (REVAA) | JPMorgan Chase & Company |
| Tennessee Real Estate Appraisers Commission | State appraiser coalitions representative | Old Line Bank |
| Texas Appraiser Licensing and Certification Board | Quicken Loans | |
| U.S. Department of Agriculture | ServiceLink | |
| U.S. Department of Justice (2) | ||
| U.S. Department of the Interior (2) | ||
| U.S. Department of Veterans Affairs | ||
| US. Department of Housing and Urban Development |