Author: Lee Kennedy

Right now, what Volkswagen needs most is probably not what you think.  Yes, they need lawyers and public relations help, and probably more lawyers.  And, they need some internal investigations and a new mission statement.  But, what they need most is that magic thing that money simply cannot buy: Volkswagen needs to regain the consumer’s (and the world’s) trust, since the damage to their reputation will take years to repair; this was not simply a faulty product to recall and repair, as staggering as those reparations can also be, this was a direct assault on our belief systems, and that gets personal.

So what shocked you about this deception?  Was it the tons of emissions pumped into the air by drivers who thought that they were driving “clean diesel?” Was it the brazen advertising of “clean diesel” by VW that (someone at) VW knew was not clean?  Was it the deliberate creation of code loaded into every single TDI 2.0 liter engine that was designed to foil emissions tests?  Not for me.  The most shocking detail was that the car companies self-certify their emissions and mileage stats.

The reality is that we should never expect that people (or industries or corporations) will always be ethical.  In almost all arenas of society we put systems of oversight in place to detect malicious acts in order to deter them.  The police are trusted to enforce the laws and use lethal force if necessary; yes, they have an Internal Affairs division, but their oversight is ultimately civilian.  The military has a civilian Commander in Chief.  The federal government’s three branches are designed to provide mutual oversight.

Federal regulations in banking have long since recognized that banks and credit unions cannot self-regulate either. And as technology and social changes advance far more quickly than the regulators can keep up, they put controls in place to allow carefully vetted entities to serve as their proxy for oversight, most commonly in the form of independent divisions within an organization or third party service providers.

As a case in point, the financial industry depends heavily on sophisticated models to facilitate their business decisions. Specifically within the property valuation segment of the industry, Automated Valuation Models (AVMs) are the most common models used.

Banks use AVMs to estimate property values, quickly, cheaply and accurately.  AVMs aren’t as accurate as an appraisal, but they cost a few dollars instead of a few hundred dollars.  Banks use them for all sorts of things: portfolio valuation, sales, marketing, servicing, appraisal quality control, even equity lending decisions.  And if lenders base their decisions on systematically erroneous data, or improperly built models, they can run into big problems not only to their bottom line, but also to their reputation. Understanding the risk that this creates, regulators have required all such models to be independently validated.  There are two critical aspects to this precaution.

First, validation must be independent.  The validation must be done by a person or team that is separate from the developers, users and buyers (most AVMs are built by independent companies that specialize in their development).  Very specifically, in the regulatory guidance, the model’s builders cannot be relied upon as an objective source of validation.

Second, the validation must be conducted by staff qualified in modeling or analytics, with the adequate authority to blow the whistle if they find issues.  The validation must be performed in real-world conditions, it should be ongoing, and it should be reported on at least annually.  When there are changes to the models, the business environment or the marketplace, the models need to be re-validated.

Banks and credit unions can do their own AVM validation, but most find it generally difficult to meet all of the requirements, including the authority to blow the whistle if they find issues. These validation caveats are the best justifications for looking to a company like AVMetrics to provide this service for you.

AVMetrics is in no way beholden to banks and credit unions, the AVM developers, or the resellers in any way; we draw no income from selling, developing or using AVM products.  AVMetrics conducts extensive ongoing AVM validation and has done so for over a decade.  Independence is essential, because self-certification can be an invitation to abuse, and as Volkswagen just found out, it can be devastating.  Our industry implemented independent testing years ago.  It is rigorous, exhaustive and frequent, but that is appropriate because there is a lot at stake.

If Volkswagen ever hopes to get the public’s trust back, they need to open the curtains and put on all the lights. In addition, every result and claim will need to be independently validated, since no one is likely to believe them otherwise.

These are common-sense controls, and while the application to the auto industry is not perfectly analogous to the testing of AVMs, the basic lesson is clear: we should not trust companies to test and certify themselves.

Lee Kennedy, principal and founder of AVMetrics in 2005, has specialized in collateral valuation, AVM testing and related regulation for over three decades.  Over the years, AVMetrics has guided companies through regulatory challenges, helped them meet their AVM validation requirements, and commented on pending regulations to help bring clarity and sanity to the situation.  Lee is an author, speaker and expert witness on the testing and use of AVMs.  Lee’s conviction is that independent, rigorous validation is the healthiest way to ensure that models serve their business purposes.  Every commercially available AVM vendor trusts AVMetrics to provide feedback to them on their models, facilitating each model’s continuous improvement.