Return to Sender Who Pays to Correct an Automated Decision?

In the year 20001, the state of Arkansas sent citizen Gary Jones a certified letter telling him that his house would be sold unless he paid overdue property taxes he owed. Nobody signed for the letter and it was sent back marked “unclaimed.” The state sent another certified letter, which also came back. Having not heard back, the house was sold.2

Jones learned about the sale from his daughter, after the buyer had an eviction notice delivered to the property. This (slightly more direct) process had managed to reach someone, though arguably too late for the information to be useful.

In 2006, the Supreme Court held that the state should have taken additional reasonable steps to notify him, if practicable, once the letters came back unclaimed. While it didn’t require guaranteed delivery or an unlimited search3, there were other means available, including ordinary mail (which doesn’t require a signature from the recipient). Taken at face value, Jones v. Flowers might seem like a fairly ordinary administrative problem. The state had managed to collect evidence that its own process wasn’t working, and marched headlong toward the consequence regardless.

There was no AI involved, just a calcified workflow that failed to take into account the obvious failure mode. In a way, the envelope had supplied all the feedback anyone needed.

Now it has become trivial for organizations to make and communicate decisions at a speed turn-of-the-millennium Arkansas’s tax officials would have found extraordinary. The key question, however, is the same: how quickly can someone on the receiving end get a wrong one corrected, and who is financially on the hook for the intervening work? As you might expect, most automation business cases spend considerably more time on how quickly decisions go out than on either of those questions.

I.Insufficient Information

Australia’s Centrelink had been comparing welfare recipients’ reported earnings with tax data for years. When a discrepancy was identified, it prompted certain questions. If the recipient couldn’t (or wouldn’t) supply the information, the agency was empowered to approach employers or banks for the records needed to establish the individual in question’s earnings.

Benefit entitlement depends on income earned in individual fortnights4. The tax data used for comparison supplied annual earnings. Someone working full time for part of a year and studying for the remainder could have an accurate annual total that would not necessarily reconcile with the data gathered from a specific fortnight. Averaging the earnings across the period could turn money earned while working into income supposedly earned while also receiving benefits.

Enter Robodebt, the automated approach to the work Centrelink had been doing for years. With this new tool, that average could become the basis for a debt demand in situations where the person didn’t provide the requested information. The recipient then had to produce records to disprove it. The automation of the institution had moved the work of verification onto the people it was pursuing.5

In one highly publicized example, Deanna Amato was quite surprised when she found out that Centrelink had taken her entire tax refund. It had sent eight letters to an old address and added a 10 percent penalty when it received no response.6 After litigation commenced, Centrelink took back over the investigative work, contacted her former employer and bank to verify her earnings, and wound up recalculating the alleged overpayment as a little more than a dollar, which it graciously decided to waive. The agency’s own investigation had reduced the demand to an amount that wasn’t worth collecting.7

The sequence is key, because clearly the information-gathering powers were available before litigation commenced. They’d been doing it for years before introducing the automated system. In this case, the agency waited until after the suit was brought before bringing them to bear. Whatever efficiency had been achieved should probably be remeasured against that fairly substantial addition to the workflow.

On November 27, 2019, the Federal Court made consent orders8 in Amato v Commonwealth of Australia after the government’s concessions. It declared the debt demands invalid because the information before the decision-maker clearly couldn’t have established that the alleged amounts were owed, or even justify the penalty.9

This was an early case of administrative automation, well before the current generative AI boom. The governance problem that has multiplied today was already fully formed. Institutions could act on assumptions, and attempt to shift the burden of proof onto the person affected.

II.Postage Due

In Blowing Smoke or Smoking Gun?, we asked who was paying the verification tax behind the claims of AI productivity. That article was primarily concerned with costs migrating between departments. What’s new here is that work crosses the organizational boundary altogether, which makes the calculation more challenging, and potentially more significant.

Consider a hypothetical automated billing system that saves a company ten minutes per account. On the accounts it gets wrong, customers spend an hour locating statements, explaining the discrepancy, and submitting the same information through a second channel because the first one didn’t accept attachments. The company’s internal dashboard captures the ten minutes saved. It might even count the support tickets (though probably only the minutes an employee spends handling it). The customer’s hour is functionally invisible until they attrit.10

Despite not being measured, that hour is still real work, as is time spent waiting on hold (during working hours), or becoming a self-taught expert in a previously unfamiliar process to discover which decisions need appealing. If automation creates more of that work, the business may have reduced its own cost while increasing the total effort required to arrive at the correct answer.

And don’t take any of this as a call for a blanket prohibition on engaging the customer in the investigatory process. There are legitimate reasons to ask a customer for information. Sometimes they’re the only person who has it, and a well-designed self-service process really can make everyone’s life easier. The question is whether the request is necessary, proportionate, and directed to the person best placed to answer it. Asking someone to confirm a transaction is one thing. Requiring them to reconstruct years of records to rebut an inference you haven’t fully established is a different service model entirely.

There is also a selection problem in the numbers. An appeal rate tells you how many people appealed. It doesn’t tell you how many decisions were wrong. Some people never receive the notice (see Jones and Amato above), some can’t understand it, and some look at the time required and decide that paying is more cost-effective. A process that introduces sufficient friction to challenge can produce fewer complaints without producing better decisions.11

You can probably see the incentive. If the business counts a decision as complete when the notice goes out, and counts an unchallenged decision as correct, making decisions cheaply and making challenges difficult will improve both sides of the dashboard.

III.Account Settled

In the Michigan Supreme Court’s account of Bauserman v. Unemployment Insurance Agency, Grant Bauserman was told he owed nearly $20,000 following determinations that he had received benefits improperly and committed fraud. He managed to get the appeal in on time, but the agency nevertheless garnished his tax refund (though it later reversed its determinations, and returned the money). The litigation concerned alleged due-process violations associated with Michigan’s MiDAS automated system.12

In July 2022, the court held that the plaintiffs could pursue monetary damages for the alleged violations of Michigan’s Constitution. It was deciding whether the claim could proceed, rather than finally establishing every allegation. The critical distinction ended up being around simply getting your money back or obtaining a remedy for the constitutional injury. A $20 million class settlement received final approval in January 2024.13

While a reversal can correct the balance on an account, it doesn’t rewind the period during which the account holder was deprived of the money. For someone facing a similar error, that interval may involve borrowing, missed payments, or hours away from work. How those losses are compensated is a legal question that depends on the claim and the jurisdiction.

These three cases arose under different jurisdictions, with different laws governing public bodies. The enterprise lesson is an operational one, to follow decisions through to their consequences, including the work required to undo them. Your obligations will depend on the decision and the law that applies. Your efficiency claim should depend on whether you measured both sides of the process.

IV.The Cost of Return

Let’s look practically at what that kind of measurement might entail. Start with a unit of work (i.e., a claim processed, an application assessed, or an account reviewed). Follow it through notices, challenges, repeat contacts, and eventual correction. Compare that whole sequence before and after the introduction of automation, with comparable cases and sufficient time for the appeals to arrive. A deployment that looks exceptionally productive in its first month can still be hiding a correction queue that doesn’t become visible until the third.

Keep the institution’s time and the recipient’s time visible separately (combining them into a single average makes it too easy to lose the transfer again). Record what people are asked to provide, how often they have to provide it, how long resolution takes, and whether consequences continue while a dispute is pending. You won’t know every customer’s lost hour to the minute, but you can estimate through sampled case reviews and direct research. Show the uncertainty and use it to improve the estimate.

Look outside the appeal queue. Review a sample of unchallenged decisions, including cases where notices failed, people abandoned the process, or customers left. Otherwise you might end up measuring correctness only among the people who had the time, knowledge, and persistence to reach you.

Then test the route back: can the reviewer see the evidence and assumptions behind the original decision? Can they obtain missing records without sending the customer away to do the same work again? Do they have authority to change the result? Where the stakes justify it, can someone pause the consequence while the dispute is examined?14 An appeals function requires people with access, time, and authority, and those are a key part of the deployment budget.

Correction needs to traverse the same path that the error followed. If a decision changed an account status, triggered a collection instruction, or supplied information to another system, changing the originating record may leave the consequences orphaned (but still running) elsewhere. The person who proved the error shouldn’t have the onus to uncover each of those destinations and negotiate a separate correction. Whoever owns the original workflow should own the return journey, including checking that it finished.

When an appeal reveals a defective rule, look for the blast radius of everyone exposed to it. Requiring each affected person to make the same successful argument would turn a known system issue into a series of individual administrative chores.

This extends the argument in Room to Be Wrong. We discussed the conditions that let an institution examine its systems and correct what it finds. That room matters to the people impacted by its decisions as well. An organization’s ability to learn from mistakes has limited value if the cost of every lesson is charged to someone trying to get their account unlocked.

V.Return Address

The useful detail in Jones is that the feedback physically arrived back at the institution. The state had the returned envelope. Amato’s case exposed a different failure: the institution had the power to obtain evidence, but exercised it only after she went to court. In both stories, the next necessary step belonged to the organization.

At Privlex, this is where legal oversight and technical evaluation need to meet. The assessment has to reach the decision in production and follow what happens when it is challenged, with counsel directing the legal analysis and the people responsible for operations able to act on the findings. Privilege can protect qualifying legal advice;15 the practical value comes from fixing the process before another person has to fight their way through it.

Before calling an automated process efficient, count the work it creates for the people subject to its decisions. If the saving depends on leaving their time out, be clear about whether productivity has actually been improved.

Let us help you follow the decision all the way back.