AI Will Improve Healthcare
But Beware the Eventual Enshittification
In 2023, the writer Cory Doctorow coined the term “enshittification” to describe the slow degradation of tech services and user experience over time. He described the premise like this:
“Here is how platforms die: first, they are good to their users; then they abuse their users to make things better for their business customers; finally, they abuse those business customers to claw back all the value for themselves. Then, they die.”
Right now, we are still in the honeymoon phase of large language model (LLM) artificial intelligence, where the major companies are still being abundantly generous just like the early days of Facebook, Uber, Google, Amazon, and TikTok.
These original 21st century tech companies offered their services for free or with steep discounts (the Millennial subsidy); the UI was intuitive and seamless, customer service was responsive, and they delivered exactly what they promised and sometimes more.
Eventually, these companies captured market share, locked in users, acquired their competitors, stopped improving their services, and increased the price of services by raising fees or inundating us with a never ending stream of ads.
If we don’t expect this to happen with artificial intelligence, then we are just being naive.
Enshittification of Healthcare
Healthcare has long been enshittified. This is undisputed. In fact, was it ever really efficient? The consolidation and closure of hospitals by private equity firms, the growing glut of administrators in healthcare, the incessant denial of claims or coverage by insurance companies, the slashing of basic science investment, the self-sabotage of government organizations like the CDC and the FDA, the health professional shortage, and on and on we go.
Rather than focus on first principles, reducing inefficiencies, and root cause analysis, the government and the healthcare industry have succumbed to patchwork legislation and solutions that vary by state. The involvement of middlemen and administrative superfluity has led to fragmented care, siloed information, and increased costs. Meanwhile, patients are left unsatisfied, sicker, and disempowered.
Utah Allows AI Prescription Renewals
But now, the promise of AI to fill in the gaps in the healthcare system is coming to fruition. Just this past week, Politico reported that Utah has established a partnership with the healthcare startup Doctronic to allow an AI system to refill prescriptions for patients with chronic conditions. So far, the FDA has not given an opinion on this new system, but the state of Utah believes that this endeavor can lower the costs for patients and ease the burden on patients.
Patients can visit Doctronic’s website, where it verifies that they are located in Utah via IP address and that they live in Utah via personal identification. It then pulls the patient’s existing prescriptions and gives the patient the opportunity to select which prescriptions to renew.
The chatbot prompts the patient with a series of questions to determine if the patient is eligible for a refill and if it is safe to do so by asking about symptoms, medication adherence, side effects, and recent changes in medical history. If there are any red flags (like medication interactions or contraindications), it will escalate to a human clinician. If everything checks out, it refills the order electronically at a pharmacy. Doctronic’s chatbot is covered by unique malpractice insurance that places the AI under regulatory oversight..
Doctronic supplied data to Utah regulators that demonstrated their AI-generated treatment plans were identical to physicians 99.2% of the time. This was based on 500 urgent care cases. Utah’s Office of Artificial Intelligence Policy will continue to track and publish results on medication adherence, patient satisfaction, safety outcomes, workflow efficiency, refill timeliness, and other metrics in order to determine effectiveness of the system.
As expected, physician groups, like the American Medical Association (AMA), have urged restraint and recommend that physicians remain involved in prescribing decisions. The AMA argues that some potential side effects or underlying health issues may not be effectively captured by the Doctronic. And even if the chatbot triggers human clinician oversight, how does that provide continuity of care for the patient from a provider that knows him well?
Will it work?
On first evaluation, this actually seems like a great idea. According to Doctronic and the Utah Department of Commerce, 80% of all medication activity are for prescription renewals. Further, it often takes weeks to obtain a medical appointment amid a national physician shortage. This may lead to reduced compliance, missed doses, and avoidable ER visits. This system might also allow for physicians to save time on low risk visits and spend more time with patients who have complex needs.
But it would be prudent to remind ourselves that we are still early stages of the AI transformation of medicine. There are a number of issues that patients who use Doctronic must contend with:
Technological barriers - For older patients, using a chatbot may not come naturally. They may also not be forthcoming about all their symptoms or they may not know if it is relevant. Even if the patient has a family member help with interacting with the chatbot, how can we be sure the patient is fully involved in their care?
Hallucinations - It has already been well established that LLMs hallucinate and supply false information. How often will Doctronic’s system be audited? How will it detect errors in judgement or hallucinations? This has not been well elucidated.
Security - Healthcare is rife with information leaks and system hacks. By partnering with Doctronic, Utah is allowing another third part, another middleman, to obtain and retain patient data. How will the state respond if Doctronic is hacked?
Prompt injection - Something as random as poetry has been shown to circumvent safety features on chatbots. It is too soon to tell if Doctronic is vulnerable to prompt injection, but people will inevitably try to bypass the chatbot. How is Doctronic protecting against this?
The role of pharmacists - One of the more interesting aspects of the prescription renewal debate that I discovered is that even pharmacists are questioning whether the system will detract from their authority. Some say that pharmacists should be given prescription refill authority, because they are in a better position to answer patient questions in person and catch medications interactions. Will anyone listen to them?
Liability - If Utah is partnering with a third party company, will they, as the government, allow themselves to be liable for an adverse reaction to medication? What are the consequences if Doctronic’s AI system is liable? Will any human face consequences? Is it just a slap on the wrist and a fine?
Access to healthcare based on class and status - One of the most consequential aspects of AI technology is the premium it will place on human interaction. With AI’s ability to mimic digital art and workflows, live performance and live interactions will become more expensive. We might be headed for a future where the poor or uninsured must interact with a digital AI doctor, while the wealthy and insured have access to real physicians in person because they can pay extra. What does that say about our healthcare system and our society if we dehumanize the patient-physician interaction? Does it matter?
Not the Only Game in Town
Doctronic is the not the only company exploring AI healthcare. Mass General Brigham (MGB) in Boston recently debuted their solution to the physician shortage: Care Connect, which was developed by the company K Health. As opposed to Doctronic, Care Connect focuses on patient’s medical history and chief complaint for common urgent care requests (colds, rashes, simple injuries, mental health issues, and chronic disease concerns). Care Connect’s AI tool then sends a suggested diagnosis and treatment plan to one of 12 employed physicians who are logged in remotely from around the country and available 24/7.
In this case, Care Connect is not making clinical decisions. Rather it is gathering history and forwarding the information to a physician who will ultimately sign off on the treatment plan. K Health has also partnered with the Mayo Clinic and Cedar-Sinai Medical Center in Los Angeles, giving it an impressive roster of partners.
Conclusion
Obviously it is still too early to predict the success of the partnership between Utah and Doctronic. My hope is that it does make a positive impact on the lives of patients and doctors, lowers costs, and improves efficiency.
But don’t we say that about every new service or technology, only to be eventually disappointed?
A recent NY Times op-ed makes the case that eventually, the AI companies, on which so many services rely on, will feel the pressure to turn a profit for their investors. At this moment, many users are not paying for their relatively simple AI needs. But we should not be surprised when the AI developers begin to charge for subscriptions, start running ads, and demand higher premium for agentic AI services like preparing our taxes or buying airline tickets. The same enshittification will undoubtedly happen with AI in healthcare.
Don’t say you weren’t warned.