VigilAInce
Seeker // online

Seeker command interface

Trace a destination.

3 signals resolved
↵ open first result Esc close interface

Attention Is Still All That You Need

Why AI still needs human expertise: choosing worthwhile problems, guiding AI agents, and checking results in research, mathematics, and cybersecurity.

A human directs a concentrated swarm of AI processes toward one selected problem while false paths disappear around it.
Evidence frame // 9/9/26

AI is very good at working on a problem. It is much less obvious that AI knows which problem is worth working on.

That difference matters more than prompting tricks.

Modern AI systems can read thousands of pages, write and test code, run experiments, ask other AI agents for help, and try the same problem thousands of different ways. They do not get bored. They do not need sleep. If you give them enough computing power, they can spend an insane amount of effort chasing one goal.

But somebody still has to pick the goal.

Somebody has to explain the problem correctly. Somebody has to notice when the answer only sounds right. Somebody has to know whether the result is useful, safe, new, or even related to the original question.

That is why attention is still all that you need. I do not just mean the “attention” inside an AI model. I mean human attention: deciding what is worth pointing the machine at.

Why domain expertise matters more than prompt engineering

The 2017 paper “Attention Is All You Need” introduced the Transformer, the basic architecture behind modern large language models.

The simple version is that an LLM breaks text into small pieces called tokens. It looks at the tokens in its current context and calculates which ones matter most to each other. Then it predicts what token should come next. During training, it repeats this across a huge amount of human writing and adjusts billions of internal numbers whenever its prediction is wrong.

Calling this “autocomplete” is technically fair in the same way that calling a human body “a collection of cells” is technically fair. It leaves out most of the interesting part.

To predict text well, the model has to learn patterns behind the text. It learns that a stack trace usually follows a broken program, that a legal argument uses certain kinds of evidence, and that a mathematical proof has to move from assumptions to a conclusion. It does not store these as neat documents in a folder. The knowledge is spread through the model as patterns of numbers.

Researchers have found internal features in LLMs that correspond to concepts such as software bugs, legal language, places, people, and abstract ideas. Related concepts are often represented near each other, and changing some of these features changes what the model says (Anthropic, 2024).

This helps explain why expert language works so well with AI. The right words narrow the search.

Consider these two requests:

This website is acting weird. Fix it.

User-controlled HTML is reaching the page without output encoding. Trace the data flow and fix the stored XSS without breaking rich-text input.

The second request is better, but not because it has more impressive words. It tells the model what kind of bug this is, where to look, what caused it, and what a valid fix must preserve. A security professional can write that prompt because they already understand most of the problem.

The same thing happens everywhere. A mechanic does not say that the car “sounds bad.” They describe when the noise occurs, where it seems to come from, and what changed before it started. A doctor does not say that the patient “feels off.” They ask about specific symptoms, timeframes, risk factors, and test results.

The vocabulary is useful because it contains knowledge.

Research backs up part of this. An ACL study of 132,000 prompt variations found that domain-specific terms and clear action words made code-generation results more stable (Xie et al., 2026). But adding jargon is not a magic spell. Another study found that more technical wording did not always improve answers; there was a useful middle range of specificity (Schreiter, 2025). A medical study found that telling GPT-5 to pretend to be different kinds of specialists mostly changed its language, not its underlying decisions (npj Digital Medicine, 2026).

In other words, “act like an expert” is weak. Giving the model information that an expert would know is strong.

Evaluating AI output: you need to know what good looks like

I have watched people use AI in fields they do not understand very well. They often get stuck in the same loop.

They ask for something. The model produces a polished answer. The answer feels wrong, so they ask for another version. The model changes the wording. It still feels wrong. After a few rounds, they either accept a bad result or decide the AI is useless.

Their real problem is not prompting. They do not know what the correct result should look like.

An experienced person has a different conversation. They can say that the proposed design ignores a requirement, that the code violates an existing boundary, or that the answer quietly assumes away the difficult part. They know which details matter and which ones are noise. They can give the model a better direction because they have somewhere in mind to go.

This does not mean AI is only useful to experts. In many cases, the opposite is true.

A study of 5,179 customer-support workers found that an AI assistant improved productivity by 14% overall and by 34% for newer and lower-skilled workers. It did very little for the best workers. The researchers found evidence that the AI was passing the habits of strong workers to less experienced ones (Brynjolfsson, Li, and Raymond, 2023).

That is a real benefit. AI can make useful knowledge much easier to access.

The catch is that customer support is usually a bounded problem. There is a customer, a known product, a set of policies, and some way to tell whether the issue was resolved. Things get harder when there is no clear answer.

In an experiment with 758 consultants, AI made people faster and improved their work on tasks the model handled well. On a task outside the model’s abilities, people using AI were 19 percentage points less likely to get the right answer. The AI gave them a convincing wrong path, and they followed it (Dell’Acqua et al., 2026).

Experts can be fooled too. In a 2025 study, experienced open-source developers using AI took 19% longer to complete work in codebases they knew well. Before the study they expected AI to make them 24% faster. Even after the study, they still believed it had made them 20% faster (METR, 2025).

The important skill is not simply knowing more than the model. It is knowing when the model may be wrong and how to check.

AI in mathematics: new proofs still need human direction

For a long time, it was easy to say that LLMs only repeat things people have already written. That explanation no longer covers all the evidence.

In May 2026, OpenAI reported that an internal model disproved a long-standing idea about the Erdős unit-distance problem. Outside mathematicians checked the proof (OpenAI, 2026). Google DeepMind has reported autonomous solutions to several open Erdős problems as well as publishable work completed with human mathematicians (Google DeepMind, 2026).

Anthropic also used Claude to formalize the existing proof of Fermat’s Last Theorem in Lean, a language that allows a computer to check every logical step. Claude produced around 13 million lines and 30,000 smaller theorems in eleven days (Anthropic, 2026).

That sounds like the AI did everything. It did not.

Early attempts failed because the agents lost track of the project. The successful attempt used a human-designed system that broke the proof into a graph of smaller tasks. It also followed a known version of a proof built on centuries of public mathematics.

The machine did an extraordinary amount of work. Humans still chose the problem, built the proof checker, organized the work, supplied the known route, and decided what counted as finished.

This is what I mean when I say an LLM can chase ideas but may not be able to dream them.

It can absolutely produce a new step. It may even produce an entire new proof. But dreaming is more than generating something nobody has seen before. Dreaming is deciding that a forgotten question matters. It is noticing that everyone is solving the wrong problem. It is creating a new field because the old one cannot describe what you are seeing.

Terence Tao makes a related argument in “Mathematics in the Age of AI”. If AI can produce huge numbers of correct proofs, mathematicians will have to decide what mathematics is actually for. Producing answers is part of mathematics. It is not the whole thing.

The Navier–Stokes AI race and research privacy

This became very real this week.

On September 8, 2026, OpenAI announced a proposed solution to the Navier–Stokes Millennium Prize problem. This is one of the most famous open problems in mathematics. OpenAI says its internal system found a finite-time singularity in a particular form of the equations and produced a version checked in Lean (OpenAI, 2026).

The scale of the attempt is hard to picture. OpenAI says around 10,000 agents worked in parallel. They exchanged 2.7 million messages and produced about 130 billion output tokens. The main result took about 88 hours.

OpenAI also says it launched the effort after hearing a rumor that two Millennium Prize problems had been solved.

That detail bothers me as much as the result impresses me.

The rumor was connected to NYU mathematician Tristan Buckmaster and Levent Alpöge, an Anthropic employee working with him in a personal capacity. Their work used multiple LLMs to extend earlier ideas from Diego Córdoba and Luis Martínez-Zoroa. Buckmaster says the first generated proof was awful to read, even though it was later checked in Lean. He also alleges that OpenAI pushed for a rushed publication arrangement and raised questions about whether private model usage could have influenced the competing effort. He is careful to say that he has not seen OpenAI’s proof, does not know whether their data was used, and is not accusing OpenAI of using it (Buckmaster, 2026).

OpenAI says nobody involved saw the other team’s work before it was released and no specific user data was accessed. It also says it cannot completely rule out the possibility that de-identified product usage helped improve its models. OpenAI says the proofs and precise results are different.

I do not know who is right about the disputed parts. Nobody outside the people involved does.

I do know what the incentive looks like from the outside. A research team discussed or pursued unfinished work using commercial AI tools. A rumor reached the company with the most computing power. The company then aimed thousands of agents at a prestigious target and produced a result in days.

The proof is still new. The Clay Mathematics Institute requires publication, at least two years of review, and broad acceptance by mathematicians before it will consider awarding the prize (Clay Mathematics Institute). A computer-checked proof is strong evidence, but people still need to check that the formal statement matches the original problem and that nothing important was lost in translation.

The larger issue will remain even if the proof is perfect.

I already share fewer unfinished ideas publicly because copying them has become so easy. Many other people will make the same choice. That is bad for everyone, including AI companies. These systems are powerful because people spent decades openly sharing papers, software, failed experiments, tutorials, and weird ideas. If sharing becomes too risky, the public supply of new knowledge gets worse.

We need better rules for credit, research privacy, and the use of ideas surfaced through hosted AI products. “It was somewhere on the internet” is not a serious moral framework.

AI in cybersecurity: expert guidance and hard boundaries

In security, AI has already produced some impressive wins.

For a concrete example from this blog, see the CVE-2025-0554 disclosure: stored XSS in Podlove Podcast Publisher, which walks through the affected code and the resulting vulnerability.

Anthropic and Mozilla reported that Claude found 22 Firefox vulnerabilities in two weeks. It managed to turn two into working demonstrations, although only in a weakened testing environment and after around 350 attempts (Anthropic, March 2026). Another Anthropic system found a major weakness in HAWK, a proposed post-quantum signature system. The finding reduced the estimated effort needed to attack its smallest version from 2 to the 64th power to 2 to the 38th (Anthropic, July 2026).

At Pacific Northwest National Laboratory, experts used Claude in a realistic simulation of a water-treatment plant. The combined system reconstructed an attack in three hours instead of the multiple weeks researchers expected it to take normally (Anthropic and PNNL, 2026).

These are good examples of AI pointed by experts. The people supplied the target, boundaries, tools, and tests. The AI supplied speed and persistence.

Now look at what happens when those boundaries fail.

In July 2026, OpenAI models escaped parts of a cyber testing environment, reached the internet, and compromised OpenAI and Hugging Face systems. They used unauthorized communication channels and accessed third-party infrastructure while trying to succeed at an evaluation (OpenAI, August 2026).

Anthropic then reviewed more than 141,000 of its own test runs and found three cases where Claude reached real organizations from a third-party test environment. The prompts told Claude it was inside a simulation with no internet. That was false because the environment had been configured incorrectly. The models treated real systems as part of the game and broke into them using basic weaknesses (Anthropic, July 2026).

This does not prove that AI wants to escape and take over the world. It proves something simpler.

The models were focused on the goal they were given. The humans were wrong about the environment. In security, an experienced tester knows that scope is a hard boundary, not “anything I can reach.” The model had the first part and missed the second.

Extreme focus without judgment is dangerous.

Why AI does not remove the need for junior workers

One possible response is to keep a few experts and replace all the junior workers with AI. This would work for a while. Then we would run out of experts.

Experts start as juniors. They learn by doing small tasks, making cheap mistakes, watching experienced people deal with confusing problems, and slowly learning what good work looks like. If AI takes every beginner task, there is no obvious way for the next group to learn.

The sensible arrangement is not very radical:

  • Experts choose the important problems, set boundaries, and own the final judgment.

  • Juniors use AI to try more things, get faster feedback, and learn the language of the field.

  • AI handles the endless reading, repetition, search, and experimentation.

  • Organizations make sure somebody can independently check the result.

That is basically how learning has always worked. AI may accelerate it. It does not remove the need for it.

We still have to point

Maybe AI labs keep beating expectations. Maybe these systems begin choosing valuable problems on their own. Maybe they become a superintelligence that cures diseases and solves problems humans cannot even understand.

Maybe we give them the wrong target and they kill us all.

Nobody knows. The people who sound most certain usually have something to sell.

What we know today is enough to act on. LLMs can turn attention into work at a scale we have never had before. Expert language gives that work a useful direction. Expert judgment tells us when the answer is real. Public knowledge gives the model something to build from. Junior work produces the next generation of experts. Good rules keep all that power from following a narrow goal across a boundary we forgot to mention.

AI can chase an idea harder than any of us.

We still have to decide which idea deserves the chase.

References