Should frontier AI be allowed to grade its own homework?
In a recent Science editorial, Thorsten Holz makes an important point: Claims about the capabilities and safety of frontier AI should not rest solely on assurances from the laboratories developing these systems. Some degree of independent verification is reasonable. The more difficult question is how much oversight is justified by the evidence. Comparisons are increasingly being made between frontier AI and biological containment or gain-of-function research. That analogy is powerful, but it is also more complicated than it first appears.
Supporters of gain-of-function research have argued that knowing in advance what a virus might become capable of doing could help us prepare, somewhat like a firefighter demonstrating how to use a fire extinguisher before there is an actual fire. Critics respond that deliberately creating or enhancing dangerous capabilities may itself amount to playing with fire, with no guarantee of an easy way out if something goes wrong. Does frontier AI present the same kind of risk? We have introduced other powerful technologies before fully understanding how they might be misused. When the internet, mobile phones, social media, and large-scale data systems emerged, did we anticipate the dark web, large-scale cybercrime, extensive harvesting of personal information, loss of privacy, or the ability of algorithms to shape what people see and read? Probably not. Many of these concerns became apparent only after the technologies were widely adopted.
Anxiety and dire predictions also seem to accompany every major technological change. When computers were introduced into Indian banking in the early 1980s, bank unions strongly resisted the change, largely out of fear that mechanization would lead to job losses. Can we imagine banking without computers today? In the early 2000s, I used to boast that I did not subscribe to the “cell phone culture.” Now I may forget my lunch box, but not my cell phone.
Real physician voices, twice a week
Free, and one click to unsubscribe.
AI may change universities, workplaces, medicine, and scientific research profoundly. But change is not the same as extinction. Humans are remarkably adaptable. So far, we have survived and learned to live with every technology we have created. That history does not mean that we should wait for serious harm before establishing safeguards around AI. Nor does it mean that every theoretical capability should be treated as an imminent danger. It suggests that we should distinguish among what a system might conceivably do, what it can demonstrate under specially designed experimental conditions, and what it actually does under ordinary conditions of use.
I am not an expert in AI or in how these systems process information. I am an ordinary end user. My questions arise from what I see when I use AI and from what happened with earlier technologies.
Where does the information behind an AI response come from? How is that information updated? When users ask questions, does the system merely answer them, or does it also learn something from what they enter? I do not know the answers, and I suspect that many ordinary users do not know them either. This leads me to privacy. We usually ask whether AI might divulge confidential information. But could privacy already be weakened when we enter information into the system, even if that information is never shown to another user?
People may ask AI about their health, finances, employment, family, unpublished research, or business plans. What happens to those conversations? Are they retained? Can developers or others see them? Could an outsider gain access? Could the information be combined with other information about the same person?
These are questions, not accusations. I am not suggesting that a particular company is selling private conversations. I am asking what an ordinary user can reasonably know about what happens after information is entered.
The questions become particularly important in health care. I have spoken with many practicing physicians about AI, and most seem excited about it. That impression is not mine alone. Recent surveys by the American Medical Association also show a rapid increase in physicians’ use of AI and growing confidence in its benefits.
Perhaps the excitement is understandable. Medicine has already passed through earlier changes in how information is handled. Medical transcription was often sent abroad, while coding and billing were increasingly handled by outside companies. Much of that work took place in the background, away from the physician.
AI feels different. Physicians can use it themselves and see the result almost immediately. Perhaps that direct access also gives them the feeling of being at the forefront of something new. That excitement should not be dismissed. AI may help physicians with documentation, summarizing information, reviewing medical literature, and reducing some of the administrative work that takes time away from patients. At the same time, excitement about what AI can do should not prevent us from asking what happens to the information placed in it.
A physician may remove a patient’s name before entering a question. But could the remaining details still identify the patient when combined with other information? Could a diagnosis, age, location, occupation, unusual medical history, or combination of treatments make someone recognizable? Who is responsible for deciding what information is safe to enter?
There is also a commercial question. Some AI systems require a subscription, while others provide substantial access without charge. Yet none of these systems is free to develop or operate. They require enormous investments in computing power, energy, information, infrastructure, and skilled personnel. Someone is paying for all of this. Why are such vast resources being made available free to millions of users? Perhaps free access is necessary at this stage to attract users, improve the systems, establish market share, and become part of people’s daily lives. But companies and investors will eventually expect a return. What will the business model look like then?
Search engines, Facebook, LinkedIn, and much of the commercial internet began by attracting large numbers of users. Advertising, subscriptions, commercial partnerships, and the value of information about users later became important sources of revenue. Could AI follow a similar path? A service supported mainly by subscriptions today could later introduce advertisements, sponsored recommendations, product placement, or a combination of these approaches. If that happens, would the advertisement simply appear beside the answer, or could commercial interests influence the answer itself? If an AI system recommends a medication, medical device, laboratory test, insurance plan, or health service, how would the user know whether it is the most appropriate answer or a sponsored one? Would a product be recommended because it is the best choice, because a company paid for visibility, or because the system had learned enough about the user to know when the suggestion might be effective?
Conversations with AI may reveal more than ordinary internet searches. They may reveal what people are planning, fearing, considering, purchasing, researching, or hoping to do. They may contain questions about illnesses, employment, investments, relationships, legal problems, unpublished research, or business decisions. Would that information eventually become commercially valuable? I am not suggesting that this is already happening in every AI system. I am asking what may happen as companies seek to recover the enormous investments being made today.
Paying a subscription does not necessarily answer these questions. A subscription model can change. At the same time, a free service should not automatically be assumed to misuse information. What users need is clarity about what is collected, how long it is kept, who can access it, and how it may be used now or in the future. I also wonder whether AI companies are still in a phase of acquiring information for training. Will that process ever end?
Probably not completely, because science, medicine, law, language, and world events continue to change. But will future systems continue gathering information in the same way, or will they depend more on selected, licensed, or specialized sources? Again, these are matters that most ordinary users cannot easily determine.
Different AI companies may use different strategies to market their products and capabilities, sometimes emphasizing what separates them from competitors. That is not entirely new. Claims made to investors, customers, and the public are already subject to securities, advertising, consumer-protection, privacy, and other laws. Those rules will continue to evolve as the technology develops.
One other question comes to mind. Electronic systems eventually fail or become obsolete. Solar storms can disrupt satellites, communications, and power grids. Cyberattacks, natural disasters, and prolonged power failures could do the same. As vast sums are invested in AI and it becomes increasingly central to health care, banking, education, and scientific research, are we investing enough in backup systems and in preserving our ability to function when AI is unavailable?
To me, the larger issue is not whether AI is uniquely dangerous. Almost every powerful technology can be used for good or misused. The more important question is how much access, autonomy, and authority we allow these systems to have, and what safeguards we put around them as their capabilities increase. An AI system that helps edit a paragraph does not present the same risk as one allowed to access medical records, conduct financial transactions, operate laboratory equipment, communicate with patients, or make decisions affecting employment, insurance, education, or public safety. Oversight should reflect those differences.
Independent verification, outside evaluation, incident reporting, privacy protections, and meaningful human accountability are appropriate where the possible consequences are serious. But oversight should grow with the evidence of risk. It should distinguish between experiments designed to probe extreme capabilities and what AI systems actually do under ordinary conditions of use. None of these concerns is entirely new. What may be new is that so many of them are coming together in one technology.
AI oversight should therefore be neither complacent nor driven primarily by analogy and speculation. It should follow the evidence and increase with the access, autonomy, authority, and possible consequences we give these systems.
Rao M. Uppu is professor of environmental toxicology and chemistry at Southern University and A&M College in Baton Rouge, Louisiana, where he has served since 2002. He is also an adjunct professor of chemistry and pathobiological sciences at Louisiana State University.
Trained in biochemistry, physical organic chemistry, and free radical chemistry, his research spans bioanalytical methods, biomarker discovery and validation, chemical toxicology, environmental chemistry, computational genomics, reactive intermediates, and the molecular mechanisms of disease. His scholarship is indexed on ORCID, ResearchGate, and Google Scholar.
Dr. Uppu is a Fellow of the American Association for the Advancement of Science (AAAS), the Royal Society of Chemistry (RSC), the Royal Society of Biology (RSB), the Royal Society of Medicine (RSM), the Royal Society for Public Health (RSPH), and the Academy of Toxicological Sciences (ATS).
In addition to his scientific research, he writes reflective essays on scientific culture, mentorship, peer review, ethics, and the human dimensions of academic life. He shares updates on LinkedIn.
