Formal Control, Real Dependence: Human Oversight and Automation Bias

TLDR: Artificial Intelligence is moving from something we use to something we rely on. We often define human control as the ability to override a system, but there is another problem: a person can remain formally in control while becoming less able, willing, or confident to exercise independent judgment. If AI makes the decision, and I merely approve it, am I really in control?

Table of Contents

1. A case for grandma

2. How did you get in this mess?

3. Big word: Epistemic dependence

4. In that case, what should we govern?

Previously, I explained that consent goes beyond clicking ‘I agree’, and needs comprehension. Today, my argument is that a person can be formally in control without ability to actually exercise it. Let’s start off with a question:

A case for grandma

Since April 2012, Jane’s granny has been battling lung cancer. Her husband passed on 4 years ago. Now her children and grandkids are grown, with jobs, families and have their own life problems. Nobody has the time to talk to her, so she spends much of her day alone. Eventually, she starts talking to an AI companion, named Arivo ¹, and it becomes part of the routine her family no longer has the time to provide

Initially, it was harmless, because Arivo actually kept her engaged, asking about her day, and whether 2AM or 2PM, it is responding to her when she feels lonely. Not long after, grandma starts talking to this chatbot everyday, or several times a day, and shares very personal details, as well as disagreements with her children.

Now imagine she asks:

“Should I stop taking this medication?”

“Should I invest my savings in this business?”

“Should I move away from my family?”

“Should I stop talking to my daughter?”

Now, that’s a different territory. That is why her people are concerned, and claim the relationship is unhealthy. However, the complicated question hiding in plain sight is:

If Arivo keeps Grandma busy, when does good enough become too much?

The answer to that question is bigger than just frequency, and I hope to shed more light as we proceed in this publication.

Quick definition of terms

Tee hee. Welcome to class

• Epistemic dependence is relying on another person or system for knowledge, judgment or understanding. This can be rational, and dependence itself is not a failure of reasoning. For Gen AI, a 2025 survey associated higher confidence in agents and tools with less critical thinking. ²

• Emotional dependence is relying on someone or something for emotional regulation, comfort, validation or companionship. Read more

• Harmful dependence: a pattern of reliance that impairs a person’s autonomy, functioning, relationships, judgment, or well-being, or creates significant difficulty when they try to withdraw or the source of dependence is unavailable. Reliance on a calculator is total and harmless because nothing collapses without it.

• Safety risk is a condition, behavior, or interaction that increases the likelihood or severity of harm to a person, their property, or others.

• Emotional influence is the capacity to change, reinforce, or shape someone’s emotions, attitudes, decisions, or behavior through words, actions, or relationship with them.

How did you get in this mess?

Doing life alone is hard, and this has nothing to do with people who deliberately download AI companions. A 2026 study of over 7000 respondents across Germany, China, South Africa and the United States found that 33% reported attachment-related behaviors toward chatbots, with perceived emotional support(including reduced loneliness and freedom from judgment), among the strongest predictors.

Humans have always depended on tools and one another.

It is human nature to learn, partner, outsource and seek advice, and we have been doing it for the longest time, with doctors, lawyers, teachers, friends & family members. Plus, we already depend on technology like GPS to locate places, calculators to do math, and Google search without memorizing an entire encyclopedia.

The relationship with artificial intelligence however, is, very different. It exerts a lot of emotional influence, talks back, explains, and reassures, and can adapt its response to your emotional state. For some users, AI output can move from a logical starting point, to confirmation, and eventually to a substitute for independent reasoning.

I know you did not get here over night. It started with creating an initial draft with ChatGPT, and not long after you were letting it suggest, and decide. Nowadays, all you say is ‘I will just ask AI’. Let’s just say you may be in soup. Hot soup.

According to human factors literature, complacency rises with automation reliability. Years later, Microsoft Research (with Carnegie Mellon’s Hao-Ping Lee) conducted a survey of 319 knowledge workers, and found the same association with generative AI:

Higher confidence in a tool predicted less critical thinking applied to its output

Big word: Epistemic dependence

Epistemic dependence on AI is having it as my moral compass, and driver of my intuition. Sounds funny huh? For Jerome, who makes 120-150 decisions per day, on AI-generated code³, it is a lived experience. Initially, he would inspect carefully, cleaning up redundancies. Five months later, he learnt to trust the process, and now approves without confirming the recommendations. On paper, all is well, but in reality, he contributed nothing to the outcome, and relied wholly on the system.

Practical dependence is the opposite of that, using AI to save time, and focus efforts on the core human aspects of a trade, business, practice.

In personal relationships or other high stake areas(like legal decisions, healthcare, education, finance, public services), AI consumers may lower skepticism because interaction feels conversational, fluent, and steady. In those moments, these folks forget explaining like a human or sounding certain do not equate to being right.

In these settings, ‘ChatGPT said so' should never become an adequate substitute for logic, or even the simple yet complex hunches.

Beware of ‘addiction’

A person does not need to be hooked to a system for dependence to matter. I mean, today, many developers (or should I say vibe engineers) struggle to cope when Claude Code credits run out. Many students are unable to write essays without chatbots, and even managers are now too lazy to challenge AI slop.

On the more emotional side, a lonely person treats an AI companion as their primary source of emotional support, and patients are willing to open up to medical agents.

There is a gap (huge or small, you decide) between formal status and substantive reality, and as a civilization, we should wonder the extent to which artificial intelligence shapes a person’s human elements.

In that case, what should we govern?

Thanks to all the frontier work(AGI talks too), we know for sure that control is NOT a single measure, and to define it rightly, we have to look past conventional metrics, to consider stuff like:

Authority: human oversight, and the right not to be subject to solely automated decisions.

Capability: practical means to exercise authority when and where it still matters, with access to resources, tooling required.

Willingness: appetite to absorb the cost of [asking/acting for] accountability. Am I willing to expose the culprits?

Confidence: the self belief, and ability of a person to trust their own judgment against the system's output. This degrades as a function of accuracy. [How much] do I trust my judgement, intuition?

Once a system is right often enough, the amount of proof required before contradicting it rises, and it rises without anyone deciding it should.

Understanding: comprehension of a system, and associated risks, that is enough to identify a wrong answer, or distinguish a correct output from a plausible incorrect one.

Loong before humans are removed from the loop, AI will keep making decisions for us, but:

1. where does dependence become consequential enough to require safeguards?

2. are users becoming less capable of performing important tasks without AI?

3. can users recognize when the system is uncertain or when independent review is warranted?

4. what happens when the human remains formally responsible, but AI has become the de facto decision-maker?

This piece is part of my efforts to understand how policies shape technology, especially AI. Or how AI shapes policy? Well, both. Subscribe to get every new publication in your inbox.

¹ Arivo is non-existent and was used for sake of example.

² Hardwig, “Epistemic Dependence,” Journal of Philosophy 82(7) (1985), argued that trusting expert testimony is rational rather than a failure of reason.

³ My estimate, assuming a 5-day week, tool-call rates from typical agentic session behavior, The Register, Claude Code Credits. Anthropic Series G Announcement