Upstart lending model

Upstart Network is a US lending platform that works with banks. Its machine-learning model decides whether to approve a personal loan and sets its price. It uses education and other alternative data, meaning information beyond the usual credit score, and no person reviews each application.

The no-action letter and its results

In 2017 the Consumer Financial Protection Bureau (CFPB), the US federal consumer-finance regulator, issued its first no-action letter, to Upstart. Such a letter tells a company that the regulator's staff has no present intention to recommend enforcement or supervisory action on a specific matter. Upstart's letter required it to report to the CFPB, and it ran until 2022.

In August 2019 the CFPB published highlights of Upstart's own simulations and analyses under the letter. The CFPB noted it had not separately replicated them. They compared Upstart's model with a traditional lending model.

Upstart's model approved 27 percent more applicants, at 16 percent lower average APRs for approved loans. The APR, or annual percentage rate, is the yearly cost of borrowing. Near-prime applicants, with FICO credit scores of about 620 to 660, were approved about twice as often.

Across the race, ethnicity, and sex groups tested, approvals rose by 23 to 29 percent and average APRs fell by 15 to 17 percent. Among its lending cases, the case file counts these as the only results on how well a lending system serves people that a regulator published.

The monitorship

Under a separate private agreement, Upstart, the NAACP Legal Defense Fund, and the Student Borrower Protection Center set up a fair-lending monitorship. A monitorship is an independent outside review, here of the live model. The monitor was the law firm Relman Colfax. It engaged Sentrana, a firm in machine learning and artificial intelligence, as a consultant. It also engaged Dr. Bernard Siskin of BLDS, an expert on statistical analyses that measure discrimination in financial services.

It published four reports, in April 2021, November 2021, September 2022, and March 2024. Its statistical tests found no close proxies for protected traits among the model's inputs. A proxy is an input that stands in for race or another trait that fair-lending law protects.

It did identify approval disparities for Black applicants. Its September 2022 report identified what would likely have been a viable less-discriminatory alternative. That is a different model with smaller gaps between groups, and this one appeared to perform comparably. Upstart updated its model before that analysis was finished.

The final report, in March 2024, recorded an impasse. Upstart, the NAACP Legal Defense Fund, and the Student Borrower Protection Center disagreed over the right, legally required way to judge comparable performance.

What a finding about inputs does not settle

Finding no close proxies is a finding about the inputs. It does not clear the outcomes. A feature that looks neutral can still affect protected groups unequally, even when no single input is a close proxy.

One example is a school's cohort default rate, the share of its former students who default, used to price one person's loan. Disparate-impact testing exists to catch this. It compares outcomes, such as approval rates, across the groups fair-lending law protects. The sources here do not say whether Upstart's model used that feature.

Who decides

Underwriting, deciding whether to approve a loan and at what price, is fully automated. So no person reviews a single application. The compliance and model-risk team tests the model's decisions in aggregate, across many applications.

Saying the model decided does not end accountability. It moves all of it to choices made before any decision: which model, what testing, how hard to look for alternatives, and what to report. The question is whether those choices are paid for and staffed, or only declared.

The duty to explain a denial

When the model declines someone, Upstart must still give that applicant specific, accurate principal reasons, meaning the main reasons for the denial. The duty comes from the Equal Credit Opportunity Act and its rule, Regulation B.

A circular is a CFPB guidance document. In Circular 2022-03, in May 2022, the CFPB said a model's complexity is no excuse. If a system cannot produce accurate reasons, the lender may not use it. Circular 2023-03, in September 2023, said its sample denial forms, with reasons to tick, do not by themselves comply.

This duty is separate from fairness testing. A lender can pass its bias tests and still fail here, by being unable to tell a declined applicant why.

What this network is drawn from

This network follows the pattern in the case file. It is not a reconstruction of Upstart's actual model. It shows the model, the compliance and model-risk team, the decision record, the monitorship, and the CFPB's reporting under the letter.

The applicants are outside the network, and no credit outcome is computed on it. Repayment results exist only for funded loans, so a wrong decline leaves no trace in them.

What the available tools can and cannot address

A failure pathway is a link between two parts of the network where a mistake made by one part can be passed on to the other. A tool closes a pathway when mistakes stop passing along it. The work along it goes on.

This case has a budget of 9 units. Understand the system, a tool that lowers other tools' prices, is not offered here, so every price is full. Every tool costs the same at every target level.

The case starts with one pressure on, Monitoring goes stale. Both scheduled outside looks at this model have ended: the no-action letter in 2022, and the monitorship with its final report in March 2024.

The network starts at a tipping point, the Lab's term for a network where mistakes could either die out or build on each other. The aim is self-correcting: mistakes are corrected rather than building on each other.

Explore (No Targets) sets no targets. There, no single tool makes the network self-correcting. Pairs do, and the cheapest is Review on schedule with Assign a challenger, for 4 units.

Under Service Targets Only, the targets can be met. That level asks for the network to be self-correcting, and for the model to be helping the work enough. The model already helps enough at the start, so the mistakes are what must change.

The Lab starts with its Dynamics setting at Both, where side effects and lingering effects are both on. At that setting, 39 combinations of tools and settings meet these targets within the budget. They use 13 different sets of tools.

The cheapest is the same pair, for 4 units. Assign a challenger with Store less data, or with Upgrade model, costs 5 units. No single tool meets the targets alone.

With Dynamics at Off, 49 combinations meet them, using 20 sets of tools. There, Assign a challenger with Check with a second model also costs 5 units, and so does Review on schedule with Store less data.

Service and Safety Targets and All Governance Targets are not fully addressable with the available tools. Both levels ask you to close every failure pathway, among other targets. Seven are open at the start.

Four are "Decision results for testing", "Testing findings shape the model", "Test results filed", and "Repayment data retrains the model". The other three are "Decision record read for testing", "Monitorship reads testing evidence", and "Results reported to the CFPB".

Store less data closes "Test results filed". No tool offered here acts on the other six. Every combination was checked, within the budget and with the budget lifted, and none meets these two levels.

That is a finding about the deployment, not a gap in your approach. Those links are how the team, the record, the model, and the two outside reviewers do their work.

More is not better here. Using every tool at once, each at its stronger setting, costs 22 units, well over the budget of 9. It makes the network self-correcting. But the model's benefit to the work falls below what the targets ask, so it meets them at no level.