Tag: AVM

Our Perspective on Brookings’ AVM Whitepaper

As the publisher of the AVMNews, we felt compelled to respond to the Brookings’ very thorough whitepaper on AVMs (Automated Valuation Models) published on October 12, 2023, and share our thoughts on the recommendations and insights presented therein.

First and foremost, I would like to acknowledge the thoroughness and dedication with which Brookings conducted their research. Their whitepaper contains valuable observations, clear explanations and wise recommendations that unsurprisingly align with our own perspective on AVMs.

Here’s our stance on key points from Brookings’ whitepaper:

  1. Expanding Public Transparency: We wholeheartedly support increased transparency in the AVM industry. In fact, Lee’s recent service on the TAF IAC AVM Task Force led to a report recommending greater transparency measures. Transparency not only fosters trust but also enhances the overall reliability of AVMs.
  2. Disclosing More Information to Affected Individuals: We are strong advocates for disclosing AVM accuracy and precision measures to the public. Lee’s second Task Force report also recommended the implementation of a universal AVM confidence score. This kind of information empowers individuals with a clearer understanding of AVM results.
  3. Guaranteeing Evaluations Are Independent: Ensuring the independence of evaluations is paramount. Compliance with this existing requirement should be non-negotiable, and we fully support this recommendation.
  4. Encouraging the Search for Less Discriminatory AVMs: Promoting the development and use of less discriminatory AVMs aligns with our goals. We view this as a straightforward step toward fairer AVM practices.

Regarding Brookings’ additional points 5, 6, and 7, we find them to be aspirational but not necessarily practical in the current landscape. In the case of #6, regulating Zillow, it appears that existing and proposed regulations adequately cover entities like Zillow, provided they use AVMs in lending.

While we appreciate the depth of Brookings’ research, we would like to address a few misconceptions within their paper:

  1. Lender Grade vs. Platform AVMs: We firmly believe that there is a distinction between lender-grade and platform AVMs, as evidenced by our testing and assessments. Variations exist not only between AVM providers but also within the different levels of AVMs offered by a single provider.
  2. “AVM Evaluators… Are Not Demonstrably Informing the Public:” We take exception to this statement. We actively contribute to public knowledge through articles, analyses, newsletters (AVMNews and our State of AVMs), quarterly GIF, a comprehensive Glossary, and participation in industry groups, task forces. We also serve the public by making AVM education available, and we would have been more than willing to collaborate or consult with Brookings during their research.

But, we’re obligated not to just give away our analysis or publish it. Our partners in the industry provide us their value estimates and we provide our analysis back to them. It’s a major way in which they improve, because they’re able to see 1) an independent test of accuracy, and 2) a comparison to other AVMs. They can see where they’re being beaten, which means opportunity for improvement. But, in order to participate, they require some confidentiality to protect their IP and reputation.

We should comment on the concept of independence that Brookings emphasized. Independent evaluation is exceedingly important in our opinion, as the only independent AVM evaluator. Brookings mentioned in passing that Mercury is not independent, but they also mentioned Fitch as an independent evaluator. We agree with Brookings that a vendor who also sells, builds, resells, uses or advocates for certain AVMs may be biased (or may appear to be biased) in auditing them; validation must be able to “effectively challenge” the models being tested.

We do not believe Fitch satisfies ongoing independent testing, validation and documentation of testing which requires resources with the competencies and influences to effectively challenge AVM models. Current guidelines require validation to be performed in real-world conditions, to be ongoing, and to be reported on at least annually.  When there are changes to the models, the business environment or the marketplace, the models need to be re-validated.

Fitch’s assessment of AVM providers is focused on each vendor’s model testing results, review of management and staff experience, data sourcing, technology effectiveness and quality control procedures. Fitch’s methodology of relying on analyses obtained from the AVM providers’ model testing results would not categorize them as an “independent AVM evaluator,” as reliance on testing done by the AVM providers themselves does not meet any definition of “independent” per existing regulatory guidance. AVMetrics is in no way beholden to the AVM developers or the resellers in any way; we draw no income from selling, developing, or using AVM products.

For almost two decades, we have continued to test AVMs against hundreds of thousands (sometimes millions) of transactions per quarter and use a variety of techniques to level the playing field between AVMs. We provide detailed and transparent statistical summaries and insights to our newsletter readers, and we publish charts that give insights into the depth and thoroughness of our analysis, whereas we have not observed this from other testing entities. Our research spanning eighteen years shows that even overall good-preforming models are less reliable in certain circumstances, so one of the less obvious risks that we would highlight is reliance on a “good” model that is poor in a specific geography, price level or property type. Models should be tested in each one of these subcategories in order to assess their reliability and risk profile. Identifying “reliable models” isn’t straightforward. Performance varies over time as market conditions change and models are tweaked. Performance also varies between locations, so a model that is extremely reliable overall may not be effective in a specific region. Furthermore, models that are effective overall may not be effective at all price levels, for example: low-priced entry-level homes or high-priced homes. Finally, very effective models will also produce estimates that they admit have lower confidence scores (and higher FSDs), and which should in all prudence be avoided, but without adequate testing and understanding may be inadvertently relied upon. Proper testing and controls can mitigate these problems.

Regarding cascades, the Brookings’ paper leans on cascades as an important part of the solution for less discriminatory AVMs. We agree with Brookings: a cascade is the most sophisticated way to use AVMs.  It maximizes accuracy and minimizes forecast error and risk. By subscribing to multiple AVMs, you can rank-order them to choose the highest performing AVM for each situation, which we call using a Model Preference Table™. The best possible AVM selection approach is a cascade, which combines that MPT™ with business logic to define when an AVM’s response is acceptable and when it should be set aside for the next AVM or another form of valuation.  The business logic can incorporate the Forecast Standard Deviation provided by the model and the institution’s own risk-tolerance to determine when a value estimate is acceptable.

Mark Sennott (industry insider) recently published a whitepaper describing current issues with cascades, namely that some AVM resellers will give favorable positions to AVMs based on favors, pricing or other factors that do NOT include performance as evaluated by independent firms like AVMetrics. This goes to the additional transparency for which Brookings’ advocates. We’re all in favor.

We actually see a strong parallel between Mark Sennott’s whitepaper and the Brookings’ paper. Brookings makes the case to regulators, whereas Sennott was speaking to the AVM industry, but both of them argue for more transparency and responsible leadership by the industry. Sennott appears to be very prescient, in retrospect.

In order to ensure that adequate testing is done regularly we recommend that a control be implemented to create transparency around how the GSE’s or other originators are performing their testing. This could be done in a variety of ways. One method might require the GSE or lending institution to indicate their last AVM testing date on each appraisal waiver. Regardless of how it’s done, the goal would be to create a mechanism that would increase commitment to appropriate testing. The GSE’s could provide a leadership role by demonstrating how they would like lending institutions to demonstrate their independent AVM testing as required by OCC 2010-42 and 2011-12.

In conclusion, we appreciate Brookings’ dedication to asking questions and providing perspective on the AVM industry. We share their goals for transparency, fairness, and accuracy. We believe that open dialogue and collaboration by all the valuation industry participants are the keys to advancing the responsible use of AVMs.

We look forward to continuing our contributions to the AVM community and working toward a brighter future for this essential technology.

How AVMetrics Tests AVMs Using our New Testing Methodology

Testing an AVM’s accuracy can actually be quite tricky. You might think that you simply compare an AVM valuation to a corresponding actual sales price – technically a fair sale on the open market – but that’s just the beginning. Here’s why it’s hard:

  • You need to get those matching values and benchmark sales in large quantities – like hundreds of thousands – if you want to cover the whole nation and be able to test different price ranges and property types (AVMetrics compiled close to 4 million valid benchmarks in 2021).
  • You need to scrub out foreclosure sales and other bad benchmarks.
  • And perhaps most difficult, you need to test the AVMs’ valuations BEFORE the corresponding benchmark sale is made public. If you don’t, then the AVM builders, whose business is up-to-date data, will incorporate that price information into their models and essentially invalidate the test. (You can’t really have a test where the subject knows the answer ahead of time.)

Here’s a secret about that third part: some of the AVM builders are also the same companies that are the premier providers of real estate data, including MLS data. What if the models are using MLS data listing price feeds to “anchor” their models based on the listing price of a home? If they are the source of the data, how can you test them before they get the data? We now know how.

We have spent years developing and implementing a solution because we wanted to level the playing field for every AVM builder and model. We ask each AVM to value every home in America each month. They each provide +/-110 million AVM valuations each month. There are over 25 different commercially available AVMs that we test regularly. That adds up to a lot of data.

A few years ago, it wouldn’t have been feasible to accumulate data at that scale. But now that computing and storage costs make it feasible, the AVM builders themselves are enthusiastic about it. They like the idea of a fair and square competition. We now have valuations for every property BEFORE it’s sold, and in fact, before it’s listed.

As we have for well over a decade now, we gather actual sales to use as the benchmarks against which to measure the accuracy of the AVMs.  We scrub these actual sales prices to ensure that they are for arm’s-length transactions between willing buyers and sellers — the best and most reliable indicator of market value. Then we use proprietary algorithms to match benchmark values to the most recent usable AVM estimated value. Using our massive database, we ensure that each model has the same opportunity to predict the sales price of each benchmark.

AVMetrics next performs a variety of statistical analyses on the results, breaking down each individual market, each price range, and each property type, and develops results which characterize each model’s success in terms of precision, usability, error and accuracy.  AVMetrics analyzes trends at the global, market and individual model levels. We also identify where there are strengths and weaknesses and where performance improved or declined.

In the spirit of continuous improvement, AVMetrics provides each model builder an anonymized comprehensive comparative analysis showing where their models stack up against all of the models in the test; this invaluable information facilitates their ongoing efforts to improve their models.

Finally, in addition to quantitative testing, AVMetrics circulates a comprehensive vendor questionnaire semi-annually.  Vendors that wish to participate in the testing process answer roughly 100 parameter, data, methodology, staffing and internal testing questions for each model being tested.  These enable AVMetrics and our clients to understand model differences within both testing and production contexts. The questionnaire also enables us and our clients to satisfy certain regulatory requirements describing the evaluation and selection of models (see OCC 2010-42 and 2011-12).

 

 

 

Property Inspection Waivers Took Off After the Pandemic Set In

Appraisals are the gold standard when it comes to valuing residential real estate, but they aren’t always necessary. They’re expensive and time-consuming, and in the era of COVID-19, they’re inconvenient. What’s the alternative?

Well, Fannie and Freddie implemented a “Property Inspection Waiver” (PIW) alternative more than a decade ago. However, it’s been slow to catch on.

But now, maybe the tipping point has arrived during the pandemic. Recently published data by Fannie and Freddie show approximately 33% of properties were valued without a traditional appraisal! (Most, if not all, would have used an AVM as part of the appraisal waiver process.) Ed Pinto at AEI’s Housing Center calls it a hockey stick.

https://www.aei.org/research-products/report/prevalence-of-appraisal-waivers-at-the-gses-including-cltv-statistics/

So, what changed? Here are some thoughts and hypotheses:

  1. Guidelines changed a little. We can see in the data that Freddie did almost zero PIWs on cash out loans, but in May that changed, and at lease for LTVS below 70%, they did almost 15,000 cash out loans with no appraisal.
  2. AVMs changed. Back when PIWs were introduced, AVMs operated in a +/- 10% paradigm. They were more concerned with hit rates than anything else, and they worked best on track homes. But, today they are operating in a +/- 4% world, hit rates are great, and cascades allow lenders to pick the AVM that’s most accurate for the application.
  3. Borrowers changed. These days, borrowers have grown up with online tools that give them answers. They are more likely to read about their symptoms on WebMD before going to the doctor, and they are more likely to look their home up on Zillow before calling their realtor. In the past, if home was purchased with a low LTV, who was it that required an appraisal? Typically, it was borrowers that wanted the appraisal – more as a safety blanket than anything else. They wanted reassurance that they were not getting ripped off. Today, for some people, Zillow can provide that reassurance without the $500 expense.
  4. Lenders changed. You would think that they are nimble and adaptable to new opportunities. But where the rubber meets the road, it’s still people talking to customers, and underwriters signing off on loans. If loan officers aren’t aware of the guidelines, they’ll just order an appraisal. Often ordering an appraisal, because it can take so long, is just about one of the first things done in the process, regardless of whether it’s necessary. After all, it’s usually necessary, and it takes SO long (relatively speaking, of course). I have known lenders who required their loan officers to collect money for an appraisal to demonstrate customer commitment. But, lenders are starting to incorporate PIWs into their processes and take advantage of those opportunities to present a loan option with $500 less in costs.

Accurate AVMs are a necessary but not sufficient criteria for PIWs, and now that AVMs are much more accurate, PIWs are much more practical, and we’re seeing much higher adoption.

So now what should we expect going forward? The trend will likely continue. There’s a lot of room left in some of those categories for PIWs to grab a larger share.

If agencies are doing it, everyone else will. If there are lenders not using PIWs to the extent possible, they are going to be at a disadvantage.

AVM Regulatory Outlook

As always, changes are coming to the valuation industry. These changes have been germinating in government and industry for a long time, but they’ve made progress in the last year, and I believe that they’re likely to emerge sometime this year.  I expect that we may see more regulatory changes liberalizing the use of AVMs soon.

I think that you’ll come to that same conclusion, too, if I share a couple milestones that I’ve observed and put them together with some insights I’ve gathered from talking to industry leaders.

The first milestone I will highlight was the July 2018 Financial System report by Secretary Mnuchin, which is consistent with the administration’s new attitude towards regulation. The report is far-reaching, and it includes thoughtful commentary about the uses of AVMs (see, for example, page 103-106). It recommends updating FIRREA appraisal requirements to accommodate increased usage of AVMs and hybrids. It also advocates for increased monitoring of AVMs and the application of rigorous market standards. And, it recommends focusing the use of AVMs and hybrids on loan programs with other mitigating risk factors.

The next milestone I will highlight was the proposed change in the de minimis threshold that was put out for comment in November of last year. The change would raise the threshold below which a residential mortgage could be originated with an evaluation, utilizing an AVM in lieu of a traditional appraisal. It would be raised from $250,000 to $400,000.

To those milestones I would add a third data point.  Last November I attended the Appraisal Subcommittee roundtable entitled: “The Evolving Real Estate Valuation Landscape.” As part of the of the Federal Financial Institutions Examination Council, the roundtable brought together industry representatives and government officials (see the table below) to discuss real estate valuation.

The day was split into two sessions; the morning and afternoon sessions each began with a panel of industry experts who addressed a series of prepared questions. In addition, there was a roundtable discussion focused on quotes from the July 2018 Financial System report referenced above.

The topic for the morning discussion was “Harmonizing Real Estate Valuation Requirements Across the Federal Government.” This session focused on identifying various federal appraisal statutory and regulatory requirements and exploring opportunities to harmonize those requirements, e.g., VA, FHA, and FHFA all having differing valuation requirements and standards.

The afternoon panel discussion topic was; “The Evolution of Real Estate Valuation” which focused on evolving valuation needs in commercial and mortgage lending. A key area of this session was focused on Alternative Valuation Products inclusive of AVM’s and their increasing used by lenders and the secondary market.

The roundtable discussion started with quotes about AVMs and hybrid valuation products and focused on standards. The group also contemplated how alternative valuation techniques can impact quality and mitigate risk. Finally, one quote that focused on speeding the adoption of technology was discussed.

As I write this six months later, I see the pieces of the puzzle coming together. Obviously, there is momentum behind the increased usage of AVMs, for their independence, increasing accuracy, speed and efficiency. But there is also an implicit concern to avoid opening the door to more risk. I see this being expressed by talk about “standards,” alternative products, such as “hybrids” and increased monitoring.

As I have written elsewhere, I welcome changes that make better use of our valuable and limited resources, namely the appraisers themselves. As AVM quality improves and the number of appraisers shrinks, we should encourage appraisers to be focused on their highest and best use. Their expertise should be focused on the complex, qualitative aspects of property valuation such as the property condition and market and locational influences.  They should also be focused on performing complex valuation assignments in non-homogeneous markets. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser, and I expect that Treasury, the FDIC and legislators are moving in this same direction.

Lee Kennedy

Participants in “The Evolving Real Estate Valuation Landscape” Appraisal Subcommittee, Federal Financial Institutions Examination Council, 2018
GovernmentTrade OrganizationsIndustry Participants
The Appraisal Foundation (TAF)American Bankers AssociationAVMetrics, LLC
Association of Appraiser Regulatory Officials (AARO)American Society of AppraisersBank of America
Consumer Financial Protection Bureau (4)American Society of Farm Managers and Rural AppraisersClarocity Valuation Services
Federal Deposit Insurance Corporation(3)Appraisal InstituteClearBox
Federal Housing Finance Agency(4)Homeownership Preservation FoundationCoreLogic
Federal Reserve Board(5)Independent Community Bankers of AmericaCushman & Wakefield Global Services, Inc.
Freddie MacMortgage Bankers AssociationFarm Credit Mid-America
Internal Revenue ServiceNational Association of Home BuildersFirst American Mortgage Solutions
National Credit Union AdministrationNational Association of RealtorsGenworth Financial
Office of the Comptroller of the Currency (4)Real Estate Valuation Advocacy Association (REVAA)JPMorgan Chase & Company
Tennessee Real Estate Appraisers CommissionState appraiser coalitions representativeOld Line Bank
Texas Appraiser Licensing and Certification Board Quicken Loans
U.S. Department of Agriculture ServiceLink
U.S. Department of Justice (2)  
U.S. Department of the Interior (2)  
U.S. Department of Veterans Affairs  
US. Department of Housing and Urban Development