Reflections on the Day the Models Died

Response to 2018 Call for Essays on The Risks Posed by Predictive Models

Welcome to the not-too-distant future, where the societal risks posed by predictive models are all but nil. Many technologies have come, captured our fancy, and gone over the years. Some of them, like distributed ledgers, ultimately transformed how we consume financial products. Others, like flying cars, never really got off the ground. But the one development that almost all of us in insurance unfailingly believed in was the power of predictive models — at least until we witnessed the rise and fall of Model Insurance Company of America (“Mica”).

Most of us remember what happened to Mica, but few remember how it all began. So, it is fitting that we begin this retrospective at ground zero. Back in the early days of predictive modeling in insurance, when actuaries were still using simplistic approaches like generalized linear models, carriers that leveraged analytics were seeing much greater profitability and growth than their comparatively “naïve” competitors who suffered from adverse selection. However, despite strong evidence of models’ benefits, factors such as organizational risk aversion, regulatory impediments, and legacy systems kept these approaches from truly becoming mainstream. No entity was more instrumental in changing this game than Mica.

The Rise

Like her entrepreneurial heroes, Mica dreamed of changing the world. She despised how larger insurers were using models to gobble up market share, and how the industry as a result was spiraling towards monopolies not unlike those seen in the Gig Economy of the 21st century. She believed in variety and choice. She saw with her eyes that models ‘worked’ but also believed in good old fashioned underwriting. Mica yearned to reduce the frictions that prevented smaller insurers from experiencing the existential benefits of modeling (and save them from obliteration), so she developed a state-of-the-art policy to cover assorted model risks and, in turn, provide incentive and risk management tools to help carriers get in the game.

Mica’s products were called Type I and Type II, and they covered loss ratio deterioration resulting from model risk realization in pricing and reserving models, respectively. The prices were a fixed percentage of subject premium, and credits were available for lower model risk portfolios that exhibited factors such as strong data quality practices. Mica’s underwriters carefully evaluated each applicant’s team, infrastructure, and model implementation plan. Her actuaries personally validated subject models on holdout and all manner of k-fold. Her risk consultants ensured adherence to a rigorous battery of best practices. Her claims department cultivated objective metrics and data reporting guidelines to measure model misfire over time. When substantial misfire coincided with adverse results, her cover took effect.

Mica realized her dreams sooner than she ever expected. Hundreds of insurance companies were emboldened by the ability to manage the cost of their model risk, and they either improved their capabilities or developed them for the first time. Mica’s carefully curated portfolio and hands-on relationship with policyholders resulted in models that were beyond reproach to the naked eye. Material model misfire was relatively minimal, adverse results were hardly anywhere to be seen, and loss ratios on her product were phenomenal. More importantly to Mica, insurance markets stratified and consumers had more options than in recent memory.

The Inflection Point

Situations like Mica’s rarely exist undisturbed into perpetuity. Once modeling became commonplace, many carriers drifted towards greater model complexity and opacity to regain a competitive edge. These models placed them outside of Mica’s underwriting guidelines, and carriers who stayed with her tended to have models built in her image and likeness, which did not present true variety or choice in the marketplace, at least from a pricing perspective. Mica’s results began to deteriorate as policyholders faced adverse selection inflicted by her expatriates, and as models aged and became less relevant, misfire (and policy payouts) increased. New prediction errors were realized that were initially undetectable via mostly in-time validation. To add insult to injury, competitors emerged and used models to cherry pick Mica’s best accounts, when her success had always been so heavily reliant on the human touch.

Mica’s board decided it was time for a change in direction. She had become the intersection of the two types of insurers she sought to help – those without strong enough modeling bona fides, and those who suffered economic consequences of model risk. And so Mica embarked upon a high stakes gambit. She replaced half her underwriters with data scientists and charged them to develop the best algorithm possible to identify her lowest risk policyholders, and to consider the best practices which she once championed as optional. Three weeks later, they came back with a complex ensemble that passed nearly every statistical test with flying colors. Mica promptly non-renewed half her policyholders who, it appeared, were high risk, and aggressively discounted current and prospective risks who the ensembles indicated had a favorable profile.

The Fall

This marks the beginning of the end of our chronicle. Mica’s loss ratios improved dramatically and she was able to regain some of her share for a time. However, when industry observers witnessed a market leader operating so profitably (and ruthlessly), critics began to question the realism of model risk and demand for the product plummeted. Longtime customers missed the underwriters they had come to know over the years and moved on. Mica experimented with demand models to extract greater premiums from those who stayed, but this destroyed what was left of her standing in the regulatory community. She was a shell of her former self.

As it happened, the model risks Mica made us all aware of were real on so many fronts. It turned out her ensemble models had actually retained a cohort of customers who appeared to be lower risk than they actually were because of low quality data reporting, yet another dimension contributing to model risk. When these turkeys came home to roost Mica went insolvent. Moreover, many of her customers who had moved on to new modeling methods – that is, those who ceased to carry model risk cover, or who insured with providers imposing lower jurisprudence requirements – also went bust. The handful of insurers left that were using very simple models or no models, and who truly valued the human touch, grew massively.

The Impact

Mica leaves behind a mixed legacy. She was neither the first nor the last startup whose balloon inflated quickly then burst into flames. It is true she lost her soul eventually, but most of us have moments when we are not completely true to who we are. In the ashes of Mica’s crash landing, the values she stood for — variety, choice, sound model risk management, and the human touch — exist in greater quantities in insurance and other industries she served than they did before. Aggregate model risk has never been lower. It remains to be seen if predictive modeling will reach the heights it once did, and no one can predict with certainty whether and when that will happen. But at her core Mica was never so much about predicting the future through models, and more about creating a better one. The fact that we are still talking about her rise and fall so many years later shows the movement she started has us well on our way to that future.

Note: This essay is a purely fictional account occurring at an unspecified future point in time, designed to help us better understand the consequences of model risk.