Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me?
I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone much have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had.
Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it.
Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for.
I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
You can cure Google's AI (but not bad results) with &udm=14
Or you can take the alternative approach, where Microsoft's own "Merl" support agent says it knows anything to do with Minecraft, and then replies to basically any gameplay question with "I don't know that".
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them...
https://deadstack.net/cluster/google-removes-ai-overviews-fo...
Why does healthcare "need disruption"?
Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
Society pretends that human doctors are better than they really are, and AI is worse than it really is.
It's the self-driving cars debate all over again.
It's inefficient and not living to its potential
And "disruption" (a pretty ill-defined term) is the solution to that?
The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here.
And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
It's inefficient and not living to its potential
Yeah, because we saw what a great job the tech bros did making government more efficient.
The inefficiency is the buying of yachts for billionaires.
Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
If we could get healthcare to that level, it would be great.
For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.
Wal-Mart and Amazon have reduced wages for employees and the quality of purchased goods more than they have improved prices for consumers.
And Google search, a service on the level of a public utility, has been degrading noticeably for years in the face of shareholders demanding more and more returns.
Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
I sympthatise with the argument. We should test it against real world data.
Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
The real source of high medical costs is the entity that sets the hospital bill in the first place.
The explanation is much simpler than people want to admit, but emotionally uncomfortable: doctors and hospitals are paid more than the free market would otherwise justify. We hesitate to say this because they save lives, and we instinctively conflate moral worth with economic compensation. But markets don’t work that way.
Economics does not reward people based on what they “deserve.” It rewards scarcity. And physician labor is artificially scarce.
The supply of doctors is deliberately constrained. We are not operating in a free market here. Entry into the profession is made far more restrictive than is strictly necessary, not purely for safety, but to protect incumbents. This is classic supply-side restriction behavior, bordering on cartel dynamics.
We see similar behavior in law, but medicine is more insidious. Because medical practice genuinely requires guardrails to prevent harm and quackery, credentialing is non-negotiable. That necessity makes it uniquely easy to smuggle in protectionism under the banner of “safety.”
The result is predictable: restricted supply, elevated wages, and persistently high medical costs. The problem isn’t mysterious, and it isn’t insurance companies. It’s a supply bottleneck created and defended by the profession itself.
Insurance companies aren't innocent angels in this whole scenario either. When the hospital bill fucks them over they don't even blink twice when they turn around and fuck over the patient to bail themselves out. But make no mistake, insurance is the side effect, the profession itself is the core problem.
> This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
They absolutely do not.
They have their profit levels capped at 15% by law and regulation. That means if the insurer wants more absolute dollars of profit, prices must go up.
It also means that if they push prices down they necessarily have less funding to administer those plans, even if the needs are the same (same number of belly buttons, same patient demographics and state of health).
As you note there's also other variables, but this claim: "Insurance companies are structurally incentivized to minimize payouts across the board" is absolutely and categorically not so.
because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?
Profit isn't even a big part of the overall revenue.
> Mandate at least decent minimal coverage standards
I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.
> and large insurance pools that must span age groups and risk groups.
Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
> Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.
> large insurance pools that must span age groups and risk groups.
What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.
> Mandate at least decent minimal coverage standards
I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)
Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.
There are a lot of not-for-profit insurance companies and they aren't noticably cheaper, though I'm not in HR and they may well be cheaper for the employer.
My fiance was in hospital recently for a fairly common disease. She arrived at 2200 Wednesday night and was discharged 1000 Saturday morning.
Her bill before "insurance negotiated prices" was $59,000. Effectively $1,000/hr, 24/7.
Disruption, yes, in the sense that the current system needs to be overhauled. But this is a space that's frequented by the SV and VC space and "disruption" has very different connotations, usually in the realm of thought that suggests some SV-brained solution to an existing problem. In some edge cases like Uber/Lyft, this upending of an existing market can yield substantial positive externalities for users. Other "heavy industry" adjacent sectors, not so much. Healthcare and aviation, not so much.
Even SpaceX's vaunted "disruption" is just clever resource allocation; despite their iterative approach to building rockets being truly novel they're not market disruptors in the same way SV usually talks about them. And their approach has some very obvious flaws relative to more traditional companies like BO, which as of now has a lower failure-to-success ratio.
I don't think you'll find many providers clamoring for an AI-assisted app that hallucinates nonexistent diseases, there are plenty of those already out there that draw the ire of many physicians. Where the industry needs to innovate is in the insurance space, which is responsible for the majority of costs, and the captive market and cartel behavior thereof means that this is a policy and government issue, not something that can be solved with rote Silicon Valley style startup-initiated disruption; that I would predict would quickly turn into dysfunction and eventual failure.
Enshittification has done a lot of damage to the concept of "disrupting" markets. It's DOA in risk-averse fields.
The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
Profit is a small part of overall revenue.
> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added.
Notice how little this sentence says about whether anything is any good.
The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.
So interesting to see the vastly different approaches to AI safety from all the frontier labs.
Why vastly different?
Aren't they both searching various online sources for relevant information and feeding that into the LLM?
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
s/hallucinated/fabricated/, please.
arguably: incorrectly guessed*
in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be
"confabulation", though, seems perfect:
“Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.”
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk"
Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"
[deleted]
They AI summary is total garbage. Probably most broken feature I saw being released in a while.
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?
Being gutted by DOGE and the Trump Administration under RFK Jr.
huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy.
Also try "health benefits of circumcision"...
I agree with your point.
Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications.
I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
[dead]
[flagged]
What's that?
lol you must be fresh to the internet
chatGPT told me, I am the healthiest guy in the world, and I believe it
If an app makes a diagnosis or a recommendation based on health data, that's Software as a Medical Device (SaMD) and it opens up a world of liability.
https://www.fda.gov/medical-devices/digital-health-center-ex...
Not surprised. Another example is minecraft related queries. Im searching with the intention of eventually going to a certain wiki page at minecraft.wiki, but started to just read the summaries instead. It will combine fan forums discussing desired features/ideas with the actual game bible at minecraft.wiki - so it mixes one source of truth with one source of fantasy. Results in ridiculous inaccurate summaries.
I find its tricky with games, especially ones as updated as frequently as Minecraft over the years. I've had some of this trouble with OSRS. It brings in old info, or info from a League/Event that isn't relevant. Easier to just go to the insanely curated wiki.
What's interesting to me is that this kind of behavior -- slightly-buffleheaded synthesis of very large areas of discourse with widely varying levels of reliability/trustworthiness -- is actually sort of one of the best things about AI research, at least for me?
I'm pretty good at reading the original sources. But what I don't have in a lot of cases is a gut that tells me what's available. I'll search for some vague idea (like, "someone much have done this before") with the wrong jargon and unclear explanation. And the AI will... sort of figure it out and point me at a bunch of people talking about exactly the idea I just had.
Now, sometimes they're loons and the idea is wrong, but the search will tell me who the players are, what jargon they're using to talk about it, what the relevant controversies around the ideas are, etc... And I can take it from there. But without the AI it's actually a long road between "I bet this exists" and "Here's someone who did it right already".
I run a small business that buys from one of two suppliers of the items we need. The supplier has a TRASH website search feature. It's quicker to Google it.
Now that AI summaries exist, I have to scroll past half a page of result and nonsense about a Turkish oil company before I find the item I'm looking for.
I hate it. It's such a minor inconvenience, but it's just so annoying. Like a sore tooth.
You can cure Google's AI (but not bad results) with &udm=14
https://google.com/search?q=parkas&udm=14
Or you can take the alternative approach, where Microsoft's own "Merl" support agent says it knows anything to do with Minecraft, and then replies to basically any gameplay question with "I don't know that".
"Dangerous and Alarming" - it tough; healthcare is needs disruption but unlike many places to target for disruption, the risk is life and death. It strikes me that healthcare is a space to focus on human in the loop applications and massively increasing the productivity of humans, before replacing them... https://deadstack.net/cluster/google-removes-ai-overviews-fo...
Why does healthcare "need disruption"?
Many people don’t get it, it’s really expensive, even in countries with non broken healthcare systems (not the us) costs increase rapidly and no one is sure how the systems will remain solvent with the same level of care given today. The way things are currently done are entrenched but not sustainable, that’s when disruptions are apt to appear.
Society pretends that human doctors are better than they really are, and AI is worse than it really is.
It's the self-driving cars debate all over again.
It's inefficient and not living to its potential
And "disruption" (a pretty ill-defined term) is the solution to that?
The solution is single payer. Any attempt to solve this with technological band aids is completely futile. We know what the solution is because we see it work in every other developed nation. We don't have it because a class of billionaire doners doesn't want to pay into the system that allowed them to become fabulously wealthy. People who are claiming AI is the solution to healthcare access and affordability are delusional or lying to you.
There are good reasons to think single payer systems are not the answer. The numerous documented inefficiencies and inconveniences they suffer from don't need repeating here.
And many single payer systems around the world only appear to work as well as they do because the US effectively subsidizes medical costs through its own out of control prices.
It's inefficient and not living to its potential
Yeah, because we saw what a great job the tech bros did making government more efficient.
The inefficiency is the buying of yachts for billionaires.
Compare: Google's founders can buy all the yachts they could possibly eat, yet Google Searches are offered for free.
If we could get healthcare to that level, it would be great.
For a less extreme example: Wal-Mart and Amazon have made plenty of people very rich, and they charge customers for their goods; but their entrance into the markets have arguable brought down prices.
Wal-Mart and Amazon have reduced wages for employees and the quality of purchased goods more than they have improved prices for consumers.
And Google search, a service on the level of a public utility, has been degrading noticeably for years in the face of shareholders demanding more and more returns.
Because insurance companies incentivize upward price momentum. The ones who innovate and bring the prices down are not rewarded for their efforts. Health inflation is higher than headline inflation because of this absence of price pressure
I sympthatise with the argument. We should test it against real world data.
Eg your argument would predict that healthcare price inflation is not as bad in areas with less insurance coverage. Eg for dental work (which is less often covered as far as I can tell), for (vanity) plastic surgery, or we can even check healthcare price inflation for vet care for pets.
This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
The real source of high medical costs is the entity that sets the hospital bill in the first place.
The explanation is much simpler than people want to admit, but emotionally uncomfortable: doctors and hospitals are paid more than the free market would otherwise justify. We hesitate to say this because they save lives, and we instinctively conflate moral worth with economic compensation. But markets don’t work that way.
Economics does not reward people based on what they “deserve.” It rewards scarcity. And physician labor is artificially scarce.
The supply of doctors is deliberately constrained. We are not operating in a free market here. Entry into the profession is made far more restrictive than is strictly necessary, not purely for safety, but to protect incumbents. This is classic supply-side restriction behavior, bordering on cartel dynamics.
See, for example: https://petrieflom.law.harvard.edu/2022/03/15/ama-scope-of-p...
We see similar behavior in law, but medicine is more insidious. Because medical practice genuinely requires guardrails to prevent harm and quackery, credentialing is non-negotiable. That necessity makes it uniquely easy to smuggle in protectionism under the banner of “safety.”
The result is predictable: restricted supply, elevated wages, and persistently high medical costs. The problem isn’t mysterious, and it isn’t insurance companies. It’s a supply bottleneck created and defended by the profession itself.
Insurance companies aren't innocent angels in this whole scenario either. When the hospital bill fucks them over they don't even blink twice when they turn around and fuck over the patient to bail themselves out. But make no mistake, insurance is the side effect, the profession itself is the core problem.
> This argument doesn’t make sense to me. Insurance companies are structurally incentivized to minimize payouts across the board. They want hospital bills lower, physician compensation lower, and patient payouts as small as possible. If insurers had unilateral power, total medical spending would collapse, not explode.
They absolutely do not.
They have their profit levels capped at 15% by law and regulation. That means if the insurer wants more absolute dollars of profit, prices must go up.
It also means that if they push prices down they necessarily have less funding to administer those plans, even if the needs are the same (same number of belly buttons, same patient demographics and state of health).
As you note there's also other variables, but this claim: "Insurance companies are structurally incentivized to minimize payouts across the board" is absolutely and categorically not so.
because in america at least, the supply of doctors is kept artificially low. that combined with exploding administrative headcount, means patients are getting pretty terrible, expensive service.
Seriously? Spending a night in a hospital results in a $10,000 bill (though the real out of pocket is significantly cheaper. God help you if you have no insurance though). Healthcare in the US is the thing that needs the biggest disruption.
But no business is going to fix it. The market is captured. Only a radical change of insurance laws is going to have any impact. Mandate that insurance must be not for profit. Mandate at least decent minimal coverage standards and large insurance pools that must span age groups and risk groups.
Many hospitals are already non-profit. That doesn't seem to bring down prices. Why would you think that this would work for insurance?
Profit isn't even a big part of the overall revenue.
> Mandate at least decent minimal coverage standards
I assume you want higher coverage standards than what currently exists? Independently of whether that would be the morally right thing to do (or not), it would definitely increase prices.
> and large insurance pools that must span age groups and risk groups.
Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
> Why does your insurance need a pool? An actuary can tell you the risk, and you can price according to that. No need for any pooling. Pooling is just something you do, when you don't have good models (or when regulations forces you).
Wuh? The more diverse the pool, the lower the risk. Your way of thinking will very quickly lead to "LiveCheap: the health insurance for fit, healthy under 30s only" for dollars a month, and "SucksToBeYou: the health insurance for the geriatric and chronically disabled" for the low low cost of "everything you have to give".
These solutions are often proposed as easy fixes but I'm skeptical that they actually will do much to reduce healthcare costs. Healthcare is fundamentally expensive. Not-for-profit hospitals and for-profit hospitals don't really substantively differ in terms of out-of-pocket expenditures for patients; I find it difficult to imagine that forcing insurance companies to be nonprofit would do much to reduce costs.
> large insurance pools that must span age groups and risk groups.
What you describe (community rating) has been tried and it works. But it requires that a lot of young, healthy people enroll, and seniors receive most of the care. In an inverted demographic pyramid like most Western economies have, this is a ticking time bomb, so costs will continue to rise.
> Mandate at least decent minimal coverage standards
I think a better solution is to allow the government to threaten in negotiating prices with companies as Canada does; it greatly reduces rent-seeking behavior by pharmaceutical companies while allowing them to continue earning profits and innovating. (I understand a lot of the complaints against big pharma but they are actually one of the few sectors of the economy that doesn't park their wealth and actually uses it for substantive R&D, despite what the media will tell you, and countless lives have been saved because of pharma company profits)
Essentially the gist of what I'm saying, as someone who has been involved with and studied this industry for the better part of five years, is that it's much more complex than what meets the eye.
There are a lot of not-for-profit insurance companies and they aren't noticably cheaper, though I'm not in HR and they may well be cheaper for the employer.
My fiance was in hospital recently for a fairly common disease. She arrived at 2200 Wednesday night and was discharged 1000 Saturday morning.
Her bill before "insurance negotiated prices" was $59,000. Effectively $1,000/hr, 24/7.
Disruption, yes, in the sense that the current system needs to be overhauled. But this is a space that's frequented by the SV and VC space and "disruption" has very different connotations, usually in the realm of thought that suggests some SV-brained solution to an existing problem. In some edge cases like Uber/Lyft, this upending of an existing market can yield substantial positive externalities for users. Other "heavy industry" adjacent sectors, not so much. Healthcare and aviation, not so much.
Even SpaceX's vaunted "disruption" is just clever resource allocation; despite their iterative approach to building rockets being truly novel they're not market disruptors in the same way SV usually talks about them. And their approach has some very obvious flaws relative to more traditional companies like BO, which as of now has a lower failure-to-success ratio.
I don't think you'll find many providers clamoring for an AI-assisted app that hallucinates nonexistent diseases, there are plenty of those already out there that draw the ire of many physicians. Where the industry needs to innovate is in the insurance space, which is responsible for the majority of costs, and the captive market and cartel behavior thereof means that this is a policy and government issue, not something that can be solved with rote Silicon Valley style startup-initiated disruption; that I would predict would quickly turn into dysfunction and eventual failure.
Enshittification has done a lot of damage to the concept of "disrupting" markets. It's DOA in risk-averse fields.
The part that needs disrupting is the billionaires who own insurance companies and demand profit from people's health .
The profit in insurance is the volume, not the margin. Disrupting it will not dramatically change outcomes, and will require changes to regulation, not business policy.
Agreed. I'd also argue that there will always be the issue of adverse selection, which in any system that doesn't mandate that all individuals be covered for healthcare regardless of risk profile, will continue to raise costs regardless of whether or not margins are good or bad. That dream died with the individual mandate, and if the nation moves even further away from universal healthcare, we will only see costs rise and not fall as companies shoulder more and more of the relative risk.
Profit is a small part of overall revenue.
> Google … constantly measures and reviews the quality of its summaries across many different categories of information, it added.
Notice how little this sentence says about whether anything is any good.
The fact that it reached this point is further evidence that if the AI apocalypse is a possibility, common sense will not save us.
... at the same time, OpenAI launches their ChatGPT Health service: https://openai.com/index/introducing-chatgpt-health/, marketed as "a dedicated experience in ChatGPT designed for health and wellness."
So interesting to see the vastly different approaches to AI safety from all the frontier labs.
Why vastly different?
Aren't they both searching various online sources for relevant information and feeding that into the LLM?
Good. I typed in a search for some medication I was taking and Google's "AI" summary was bordering on criminal. The WebMD site had the correct info, as did the manufacturer's website. Google hallucinated a bunch of stuff about it, and I knew then that they needed to put a stop to LLMs slopping about anything to do with health or medical info.
s/hallucinated/fabricated/, please.
arguably: incorrectly guessed*
in a way, all overconfident guessing is a better match for the result than hallucination or fabrication would be
"confabulation", though, seems perfect:
“Confabulation is distinguished from lying as there is no intent to deceive and the person is unaware the information is false. Although individuals can present blatantly false information, confabulation can also seem to be coherent, internally consistent, and relatively normal.”
https://en.wikipedia.org/wiki/Confabulation
* insofar as “guess” conveys an attempt to be probably in the zone
Ars rips of this original reporting, but makes it worse by leaving out the word "some" from the title.
‘Dangerous and alarming’: Google removes some of its AI summaries after users’ health put at risk: https://www.theguardian.com/technology/2026/jan/11/google-ai...
Removing "some" doesn't make it worse. They didn't include "all" AI titles which it would. "Google removes AI health summaries after investigation finds dangerous flaws " is functionally equivalent to "Google removes some of its AI summaries after users’ health put at risk"
Oh, and also, the Ars article itself still contains the word "Some" (on my AB test). It's the headline on HN that left it out. So your complaint is entirely invalid: "Google removes some AI health summaries after investigation finds “dangerous” flaws"
They AI summary is total garbage. Probably most broken feature I saw being released in a while.
Google is really wrecking its brand with the search AI summaries thing, which is unbelievably bad compared to their Gemini offerings, including the free one. The continued existence of it is baffling.
It's mystifying. A relative showed me a heavily AI-generated video claiming a Tesla wheelchair was coming (self-driving of course, with a sub-$800 price tag). I tried to Google it to quickly debunk and got an AI Overview confidently stating it was a real thing. The source it linked to: that same YouTube video!
Yeah. It's the final nail in the coffin of search, which now actively surfaces incorrect results when it isn't serving ads that usually deliberately pretend to be the site you're looking for. The only thing I use it for any more is to find a site I know exists but I don't know the URL of.
How could they even offer that without a Medical Device license? where is the FDA when it comes to enforcement?
Being gutted by DOGE and the Trump Administration under RFK Jr.
huh.. so google doesn't trust it's own product.. but openai and anthropic are happy to lie? lol
Google for "malay people acne" or other acne-related queries. It will readily spit out the dumbest pseudo science you can find. The AI bot finds a lot of dumb shit on the internet which it serves back to you on the Google page. You can also ask it about the Kangen MLM water scam. Why do athletes drink Kangen water? "Improved Recovery Time" Sure buddy.
Also try "health benefits of circumcision"...
I agree with your point.
Going offtopic: The "health benefits of circumcision" bogus has existed for decades. The search engines are returning the results of bogus information, because the topic is mostly relevant for its social and religious implications.
I am related with the topic, and discussion is similar to topics on politics: Most people don't care and will stay quiet while a very aggresive group will sell it as a panacea.
[dead]
[flagged]
What's that?
lol you must be fresh to the internet
chatGPT told me, I am the healthiest guy in the world, and I believe it