Automated Valuation Metrics
Headline for Lynn Kennedy’s Market Summary
Lynn Kennedy, a contributing consultant at AV Metrics, provides us the Market Summary for Los Angeles County featured in Mobility Magazine, July 2019.
Learn more about Mobility Magazine, the magazine of the Worldwide ERC, dedicated to professionals involved in talent mobility. www.worldwideerc.org
As always, changes are coming to the valuation industry. These changes have been germinating in government and industry for a long time, but they’ve made progress in the last year, and I believe that they’re likely to emerge sometime this year. I expect that we may see more regulatory changes liberalizing the use of AVMs soon.
I think that you’ll come to that same conclusion, too, if I share a couple milestones that I’ve observed and put them together with some insights I’ve gathered from talking to industry leaders.
The first milestone I will highlight was the July 2018 Financial System report by Secretary Mnuchin, which is consistent with the administration’s new attitude towards regulation. The report is far-reaching, and it includes thoughtful commentary about the uses of AVMs (see, for example, page 103-106). It recommends updating FIRREA appraisal requirements to accommodate increased usage of AVMs and hybrids. It also advocates for increased monitoring of AVMs and the application of rigorous market standards. And, it recommends focusing the use of AVMs and hybrids on loan programs with other mitigating risk factors.
The next milestone I will highlight was the proposed change in the de minimis threshold that was put out for comment in November of last year. The change would raise the threshold below which a residential mortgage could be originated with an evaluation, utilizing an AVM in lieu of a traditional appraisal. It would be raised from $250,000 to $400,000.
To those milestones I would add a third data point. Last November I attended the Appraisal Subcommittee roundtable entitled: “The Evolving Real Estate Valuation Landscape.” As part of the of the Federal Financial Institutions Examination Council, the roundtable brought together industry representatives and government officials (see the table below) to discuss real estate valuation.
The day was split into two sessions; the morning and afternoon sessions each began with a panel of industry experts who addressed a series of prepared questions. In addition, there was a roundtable discussion focused on quotes from the July 2018 Financial System report referenced above.
The topic for the morning discussion was “Harmonizing Real Estate Valuation Requirements Across the Federal Government.” This session focused on identifying various federal appraisal statutory and regulatory requirements and exploring opportunities to harmonize those requirements, e.g., VA, FHA, and FHFA all having differing valuation requirements and standards.
The afternoon panel discussion topic was; “The Evolution of Real Estate Valuation” which focused on evolving valuation needs in commercial and mortgage lending. A key area of this session was focused on Alternative Valuation Products inclusive of AVM’s and their increasing used by lenders and the secondary market.
The roundtable discussion started with quotes about AVMs and hybrid valuation products and focused on standards. The group also contemplated how alternative valuation techniques can impact quality and mitigate risk. Finally, one quote that focused on speeding the adoption of technology was discussed.
As I write this six months later, I see the pieces of the puzzle coming together. Obviously, there is momentum behind the increased usage of AVMs, for their independence, increasing accuracy, speed and efficiency. But there is also an implicit concern to avoid opening the door to more risk. I see this being expressed by talk about “standards,” alternative products, such as “hybrids” and increased monitoring.
As I have written elsewhere, I welcome changes that make better use of our valuable and limited resources, namely the appraisers themselves. As AVM quality improves and the number of appraisers shrinks, we should encourage appraisers to be focused on their highest and best use. Their expertise should be focused on the complex, qualitative aspects of property valuation such as the property condition and market and locational influences. They should also be focused on performing complex valuation assignments in non-homogeneous markets. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser, and I expect that Treasury, the FDIC and legislators are moving in this same direction.
Lee Kennedy
| Participants in “The Evolving Real Estate Valuation Landscape” Appraisal Subcommittee, Federal Financial Institutions Examination Council, 2018 | ||
| Government | Trade Organizations | Industry Participants |
| The Appraisal Foundation (TAF) | American Bankers Association | AVMetrics, LLC |
| Association of Appraiser Regulatory Officials (AARO) | American Society of Appraisers | Bank of America |
| Consumer Financial Protection Bureau (4) | American Society of Farm Managers and Rural Appraisers | Clarocity Valuation Services |
| Federal Deposit Insurance Corporation(3) | Appraisal Institute | ClearBox |
| Federal Housing Finance Agency(4) | Homeownership Preservation Foundation | CoreLogic |
| Federal Reserve Board(5) | Independent Community Bankers of America | Cushman & Wakefield Global Services, Inc. |
| Freddie Mac | Mortgage Bankers Association | Farm Credit Mid-America |
| Internal Revenue Service | National Association of Home Builders | First American Mortgage Solutions |
| National Credit Union Administration | National Association of Realtors | Genworth Financial |
| Office of the Comptroller of the Currency (4) | Real Estate Valuation Advocacy Association (REVAA) | JPMorgan Chase & Company |
| Tennessee Real Estate Appraisers Commission | State appraiser coalitions representative | Old Line Bank |
| Texas Appraiser Licensing and Certification Board | Quicken Loans | |
| U.S. Department of Agriculture | ServiceLink | |
| U.S. Department of Justice (2) | ||
| U.S. Department of the Interior (2) | ||
| U.S. Department of Veterans Affairs | ||
| US. Department of Housing and Urban Development |
After determining that a transaction or property is suitable for valuation by an Automated Valuation Model (AVM), the first decision one must make is “Which AVM to use?” There are many options – over 20 commercially available AVMs – significantly more than just a few years ago. While cost and hit rate may be considerations, model accuracy is the ultimate goal. A few additional estimates that are off by more than 20 percent can seriously increase costs. Inaccuracy can increase second-looks, cause loans not to close at all or even stimulate defaults down the road.
Which is the best AVM?
We test the majority of residential models currently available, and in the nationwide test in Figure #1 below, Model AM-39 (not its real name) was the top of the heap. It has the lowest average (absolute) error (MAE) by .1 over the 2nd place model. Model AM-39 is a full percentage point better than the 5th ranked model, which is good, but that’s not everything. Model AM-39 has the highest percentage of estimates within +/- 10% (PPE10%). Model AM-39 has the 2nd lowest percentage of extreme overvaluations (>=20%, or RT20 Rate), an especially bad type of error indicating a significant overvaluation or Right Tailed error.
If you were shopping for an AVM, you might think that Model AM-39 is the obvious choice. This model performs at the top of the list in just about every measure, right? Well, not so fast. Consider that those measurements are based on testing AVM’s across the entire nation, and if you are only doing business in certain geographies, you might only care about which model or AVM is most accurate in those areas. Figure 2 shows a ranking of models in Nevada, and if your heart was set on Model AM-39, then you would be relieved to see that it is still in the top 5. And, in fact, it performs even better when limited to the State of Nevada. However, three models outperform Model AM-39, with Model X-24 leading the pack in accuracy (albeit with a lower Hit Rate).
So, now you might be sold on Model X-24, but you might still look a little deeper. If, for example, you were a credit union in Clark County, you might focus on performance there. While Clark County is pretty diverse, it’s quite different from most other counties in Nevada. In this case, Figure 3 shows that the best model is still, Model X-24, and it performs very well at avoiding extreme overvaluations.
However, if your Clark County Credit Union is focused on entry level home loans with properties values below $100K, you might want to check just that segment of the market. Figure 4 shows that Model X-24 continues to be the best performer in Clark County for this price tier. Note that the other top models, including Model AM-39, show significant weaknesses as their overvaluation tendency climbs into the teens. This is not a slight difference, and it could be important. Model AM-39 is seven times more likely than Model X-24 to overvalue a property by 20%, and those are high-risk errors.
Look carefully at the model results in Figure 4 and you’ll see that Model X-24, while being the most accurate and precise, has the lowest hit rate. That means that about 40% of the time, it does not return a value estimate. The implication is: you really want a second and a third AVM option.
Now let’s consider a different lending pattern for the Clark County credit union. Consider a high value property lending program and look at figure 5, which is an analysis of the over-$650K properties and how the models perform in that price tier. Figure 5 shows that Model X-24 is no longer in the top five models. The best performer in Clark County for this price tier is Model AM-39, with 92% within +/-10% and zero overvaluation error in excess of 20%. The other models in the top five also do a good job of valuing properties in this tier.
Figure 6 summarizes this exercise, which demonstrates the proper thinking when selecting models. First, focus on the market segment that you do business in – don’t use the model that performs best outside your service area. Second, rather than using a single model, you should use several models prioritized into what we call a “Model Preference Table®” in which models are ranked #1, #2, #3 for every segment of your market. Then, as you need to request an evaluation, the system should call the AVM in the #1 spot, and if it doesn’t get an answer, try the next model(s) if available.
In this way, you get the most competent model for the job. Even though one model will test better overall, it won’t be the best model everywhere and for every property type and price range. In our example, the #1 model in the nation was not the preferred model in every market segment we focused on. If we had focused on another geography or market segment, we almost certainly would have seen a reordering of the rankings and possibly even different models showing up in the top 5. The next quarter’s results might be different as well, because all the models’ developers are constantly recalibrating their algorithms; inputs and conditions are changing, and no one can afford to stand still.
For more than 12 years we’ve been testing AVMs and watching them improve over time. More model builders have developed better techniques, and with the falling cost of processing and storage, and with the improving availability of data, AVMs just continue to get better and better.
We aren’t the only ones noticing. We recently read with pleasure Craig Gilbert’s observations of the same phenomenon (Craig is an expert appraiser and co-founder of RAC – Relocation Appraisers and Consultants).
Since co-developing the AVM for Veros in 1999+, I’ve been predicting that AVMs would eventually morph over from Mortgage Origination & Portfolio Valuations, the primary intended uses, into Relocation buyouts. The question has been “when”, not “if”. Relocation represents a microcosm sub-market of the overall residential appraisal business – maybe 5% of the total?
Back in the early days, AVMs were not as accurate as they are today. This has changed. I was thinking about this very thing this morning before opening the current issue of Mobility Magazine, and there it was. The time has arrived.
Read Mobility Magazine December 2018 article “TECHNOLOGY TODAY – What’s Hot for Mobility” written by Steven M. John and Mary-Grace Ellington of HomeServices Relocation.
Here are a few excerpts from the article:
– Recent experiments to test reliability of AVMs show the results to be comparable to formal, in-person appraisals.”
– These valuation tools can save significant time and money while offering convenience.”
– A typical FAVM can be obtained for a fraction of the cost of a traditional appraisal.” [“F” = Forecasting]
– Target values are not fed into the models, and they are not subject to obvious human bias, so theirs perceived impartiality”
– Fidelity Residential Solutions has been at the forefront of testing these new tools.”
Other Resources
Some of you may know Lee Kennedy, an Independent AVM Expert, of AVMetrics, started by Lee in 2005. Lee is a really great guy, has been an appraiser since the mid-80’s, has testified as an expert witness on cases involving use of AVMs and the Financial Crisis and has spoken at recent A.I. Symposium. He’s like the AVM gate-keeper. In his blog titled “The Wild, Wild West of Automated Valuations”, there is a graph showing that the mean absolute error of tested AVMs decreased from 14.7% in 2009 to 5.8% in 2017 and 2018. This is for all AVMs in entire U.S.. Some of course are more accurate than a +-5.7% error rate, when drilling down to specific neighborhoods and AVMs, on a case-by-case basis.
AVMetics’ ongoing independent AVM testing continues in 2019 according to the attached schedule. Please use the link below to download a pdf with the entire 2019 testing schedule.
Recently the OCC, FDIC and the Federal Reserve proposed raising the de minimis threshold for residential properties below which appraisals are not required to complete a home loan. Currently, most homes transacting at $250K and above require an appraisal, but Federal regulators propose to raise that level to $400K. A November 30th Wall Street Journal article raises some interesting issues about the topic. They reported that the number of appraisers is down 21% since the housing crisis, but more homes require an appraiser, since more and more homes exceed the threshold each year. The article also states that these factors open the door for cheaper, faster and “largely untested” property valuations based on computer algorithms, also known as Automated Valuation Models (AVMS).
At AVMetrics, we have been continuously testing AVMs for over 15 years, so we’ve seen how they’ve performed over time. As an example, the accompanying chart shows model performance accuracy as measured by mean absolute error, a statistical metric of valuation error. We utilize many statistical measures of evaluating model accuracy and precision, and they all show significant improvement in AVMs over time. And, as these automated tools get better and the workforce of appraisers continues to shrink, the FFIEC members’ proposed change seems warranted, but that doesn’t mean they don’t have their critics.
Ratish Bansal of Appraisal Inc was quoted in The Journal describing the state of AVMs as “a wild, wild West,” inviting, “abuse of all kind.” Furthermore, he contrasts that with the voluminous regulatory standards covering the use of appraisals.
We note much of those voluminous standards represent nearly the same quality control that was in place before the Credit Crisis. In other words, appraisals are not a guarantee against collateral risk. They are simply one tool in the toolbox – an effective, but comparatively time consuming and expensive tool. Also of note, far from being the “wild, wild west,” AVMs are also governed by regulators, most notably, Appendix B of the Appraisal and Evaluation Guidelines (OOC 2010-42) and Model Risk Management guidance (OCC 2011-12). These regulatory guidelines require that AVM developers be qualified, users of AVMs use robust controls, incentives be appropriate, and models be tested regularly and thoroughly with out-of-sample benchmarks. They require documentation of risk assessments and stipulate that a Board of Directors must oversee the use of all models. In other words, if AVMs were the “the wild, wild west” they would be rooted in a town with oversight of the legendary Wyatt Earp.
My strong feeling is that appraisals should not be a sole and exclusive tool when evaluations can be effectively employed in appropriate, lower-risk scenarios. Appraisers are a valuable and limited resource, and they should be employed at (to use appraisal terminology) their highest and best use. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser. Their expertise should be focused on the qualitative aspects of property valuation such as the property condition and market and locational influences. They should also be focused on performing complex valuation assignments in non-homogeneous markets. AVMs do not capture and analyze the qualitative aspects of a property very well, and they still stumble in markets with highly diverse house stock or houses with less quantifiable attributes such as view properties.
However, several companies are developing ways of merging the robust data processing capabilities of an AVM with the qualitative assessment skills of appraisers. Today, these products typically use an AVM at their core and then satisfy additionally required evaluation criteria (physical property condition, market and location influences) with an additional service. For example, a lender can wrap a Property Condition Report (PCR) around the AVM and reconcile that data in support of a lending decision. This type of “Hybrid valuation” is on the track we’re headed down. Many companies have already created these types of products for commercial and proprietary use.
We at AVMetrics believe in using the right tool for the job, and we believe there is a place for automated valuations in prudent lending practices. We think the smarter approach would be to marginally raise the de minimis threshold, but simultaneously to provide additional guidance for considering other aspects of a lending decision, specifically, collateral considerations and eligibility criteria for appraisal exemptions such neighborhood homogeneity, property conformity, market conditions and more.
AVMetrics hand-picks the best articles related to the development and use of AVMs, as well as articles about the companies and people involved in the field of AVMs around the world.
Use this link to subscribe to the AVMNews for free.
AVMetrics took over the publishing of the AVMNews in January of 2017. Newsletters are published Quarterly. You can reach the AVMNews at AVMNews@AVMetrics.net.
AVMetics’ ongoing independent AVM testing continues in 2018 according to the attached schedule. Please use the link below to download a pdf with the entire 2018 testing schedule.
IAC Opinion Letter to Congress with IAC Listing
On September 13, 2017, AVMetrics participated with its many partners in The Appraisal Foundation in sending an open letter to the House Committee on Banking, Housing and Urban Affairs urging the committee to refrain from weakening the oversight of the Appraisal Subcommittee (ASC). The letter – linked to above – emphasizes the importance of the ASC’s National Registry and the benefits of consistency brought about by ASC’s oversight.
The letter proposes several enhancements to the ASC system, including improvements in background checks for appraisers and a standard definition of a “Federally Related Transaction.” Other proposed enhancements detailed in the letter are designed to improve consistency and transparency.
On Monday, June 26, 2017 the Appraisal Institute’s Northern California Chapter hosted an educational seminar in Oakland, CA, with Lee Kennedy as an invited expert. The panel was moderated by Paul E. Chandler, MAI, and Lee’s co-panelists were Michael Simmons of AXIS Appraisal Management and Todd Krell of CrossCheck Compliance. The topic of this educational session was Third Party Vendors, Tools and Compliance: The Role of AMCs and AVMs in the Appraisal Process. The panel was part of the Commercial and Residential Symposium entitled The Role of Valuation Experts in the Current Regulatory Environment and provided both state (7hrs) and Appraisal Institute Continuing Education.
Synopsis:
Appraiser and Appraisal Management is at the heart of quality loan production. How does a lender know it is using competent appraisers providing quality reports? Policies for the empanelment of appraisers, procurement of vendors and review and quality control of assignments must be documented, managed and audited. An expanding selection of data and technology options are available to lenders to manage all aspects of collateral due diligence. Tools include fully integrated loan origination systems, appraisal management platforms and robust data and review tools. This session will review how to find, screen and manage different third party providers, including AMCS, AVM sellers and QA compliance firms.
