> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue.
This is incredibly good for science. arXiv is free, but it's a privilege not a right!
As a certain doctor once said: the whole point of the doomsday machine is lost if you keep it a secret!
> This is incredibly good for science.
I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life. That's punitive instead of rehabilitative.
> It's just one darn hallucinated citation for heaven's sake, not fraud or something.
It is fraud.
> It doesn't account for the substance or quality of their work at all.
References are part of the work. If you're making up the references, what else are you making up?
> People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life.
A one year ban is not permanent. Having a negative consequence for making poor decisions seems like an inducement to learn from the mistake?
In an ideal world, one would be keeping notes on references used while doing the research that lead to writing the paper. Choosing not to do that is one poor decision.
Having a positive outlook, if asking an AI to provide references that may have been missed, one should at least verify the references exist and are relevant. Choosing not to do that is also a poor decision, even if one did take notes on references used while researching.
> It is fraud.
No, it is emphatically not.
Fraud requires intent to deceive.
> A one year ban is not permanent.
...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that with a permanent penalty. Did you completely miss where they said "...followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue"?
Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference”for the veracity of the statement asserted.
If you are using AI-hallucinated references in scientific papers then there is some obvious intent to deceive there
> No, it is emphatically not. D Fraud requires intent to deceive.
I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here.
Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature. If you're posting things to arxiv that are not sincere engagements with the literature, you are attempting to deceive.
it's very silly, but not a big deal. Arxiv is becoming irrelevant these days anyways.
In fact would be better if they just banned AI, so we could just get off the luddite platforms.
Automated research is the future, end of story. And really it couldn't have come out at a better time, given the increasingly diminishing returns on human powered research.
Poe's law striking hard.
If you cannot be bothered to check your references when writing academic quality papers then you have no place writing them in the first place. The punishment is not chopping off a finger, it is a polite reminder to do the bare minimum.
Well, in the good old days, when we have refereed journals, it would be part of the publishing process.
A "mistake" would be a typo in a real citation. A hallucinated citation is evidence of just plain laziness and negligence, which taints the entire submission.
No it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research or substance of the paper. The lack of any sympathy or imagination here is astounding.
There are no deadlines for journal submissions. Even if you felt you were running close to your revisions being due, an email to an editor will probably fix this for you. And what you described is still negligent, not verifying the garbage output bot did not in fact output garbage.
The lack of understanding that you are responsible for the content you create, no matter what tools you use, is what's astounding.
It's not the kind of mistake that is possible unless you're engaging in fraud anyway.
> It's not the kind of mistake that is possible unless you're engaging in fraud anyway.
Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing?
If you find evidence of fraud by all means lay down the hammer. Using a single hallucinated citation like it's some kind of ironclad proxy just because you think they must be committing fraud is insane.
I would not necessarily go as far as calling it fraud, but if you cannot even verify that the reference you are citing actually exists, you are not ready to publish research papers.
Deadlines are not an excuse here. Checking whether a cited book, paper, or passage exists is the absolute minimum standard for scientific work, not an optional extra. I have written academic papers myself, and I find it astonishing that people are trying to justify this as if it were some understandable workflow mistake. At that point it is simply slop with academic formatting.
A one-year ban is not a lifetime ban. Maybe six months would also have been enough, but the point is that the author gets time to think about whether they should verify references next time. They can also use that time to manually check every other citation.
if you're not checking citations in the paper youre publishing AND trusting a non SOTA, hallucination prone ai model to come up with sources for it, its probably for the best of everyone that this paper isn't published.
yes there will be rare exceptions but in general i feel like this is a really good addition.
Why would you ask the ai to find a citation you know exists? Just reach for that citation.
If you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.
Yes, it is fraud
A citation is where you derived knowledge... If you haven't checked it and you are submitting something that should represent a ton of labour (and which will consume labour to review), you don't understand what you're doing. It is not just crossing T's and dotting I'd.
Your being set behind is less important than the fact that your publishing is setting everyone else behind.
Such a banned person is being helped to "step out of the way", and someone more competent will assuredly step forward to consume the limited maintenance labour more thoughtfully
> Your being set behind is less important than the fact that your publishing is setting everyone else behind
One hallucinated citation does not in any way imply anyone is being left behind. All it means nobody is checked that particular line of the manuscript after it was written. The rest of the paper could still be solid and treated accordingly. If you find evidence of the contrary, of course treat it accordingly, but this is so obviously not that.
> One hallucinated citation does not in any way imply anyone is being left behind.
The parent said “setting” others behind, which refers to lost time.
Being “left” behind implies a degraded trajectory, which is defined not by time lost, but by the final destination.
Different but related things (e.g. lost time can indeed affect your final destination, for instance, after growing old correcting a scourge of hallucinated citations - which should have been table stakes all along).
No. It's fraud.
You clearly misunderstand. You cite a work in your paper because you have read that work, and build upon it or want to refer to it to back up a specific claim. Generating references is fraud period, because you are implying that you have read a work when in fact you just asked an AI "please insert some reference-shaped text here" to make it look like a proper paper. It is sadly not a necessary, but certainly a VERY sufficient, reason to conclude a paper is fraudulent.
> Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated (Dieterrich, T. G.)
There needs be to a careful vetting before such adverse actions. If somebody includes a name and pushed it without express permission, does everyone get the ban? I agree that implemented the right way, this is good.
Had a colleague submit a paper with literal AI slop left in the text, got hit with a nasty revision request. Check your drafts before you submit, people. The reviewers will find it.
Also check your LaTeX comments, Arxiv makes those publicly visible!!!
I'm a screen reader user and usually read papers as raw TeX. I've seen everything: slurs, demeaning comments towards reviewers and professors, admissions of fraud, instructions to coauthors to commit further fraud before paper submission to mask the earlier fraud... it's all there. There's far less of it than I would think, definitely <1% of papers, but it's there.
I think it would be useful to run an LLM anti-fraud pass on the TeX source of all new arxiv papers. It wouldn't catch everything, but it would catch some of the dumbest fraudsters.
On the positive side, you can also find stronger claims that didn't survive review, additional explanations that didn't make the cut due to the conference's page limit, as well as experimental results that the authors felt weren't really worth including. Those need to be approached with an abundance of caution, but are genuinely useful sometimes.
Sad the suggestion here is to just disguise the slop to make it harder for reviewers to spot rather than not submitting slop to begin with.
Good; academic literature is in crisis because of all of the slop. Forcing some consequences on easily-detectable hallucinations can only be a good thing
It's not just AI, though. I did a doctorate in physics about 40 years back, and bad references were a problem back then.
Doesn't matter if it is AI hallucinations or entirely human scientific fraud, the problem is the same, and the solution works fine for both cases.
If you can't validate that your bibliography is full of real articles, you shouldn't get published.
LLMs have just poured gasoline on the fire.
"Bad", like, you literally just made them up? I hope that would have been a problem.
In what way? Surely something like the source not quite saying what was cited, or mixing up citations, rather than inventing them outright?
That, and mixing reference details from multiple sources and messing it up.
Let's say you read a paper on Arxiv but cite the version that was submitted to a journal or conference, without realizing that the authors made changes to the version they submitted and forgot to upload them to Arxiv.
Imagine how bad they are now then.
Yes and ffs arrows kill people too but we don't bring that up every time we talk about what to do with guns.
It seems a good idea to ban cheating, but how hard is it, especially in new reasoning/agents contexts to validate references?
The deeper question is whether legitimate AI generated results are allowed or not?
Test - In the extreme - think proof of Riemann Hypothesis autonomously generated (end to end) formally proven - is it allowed or not?
This is not about banning cheating, it’s about banning inaccurate information.
It is allowed as long as it’s verified.
The thread specifically points out that if authors can’t be arsed to simply proofread their text the rest can not be trusted either.
It’s a simple heuristic against low quality submissions, not an anti-ai measure.
If you use AI correctly, nobody should be able to tell that it was used at all.
You don’t need to solve everything, catching a few thousand non existent citations with such a policy is on its own a net benefit.
[deleted]
There already exists multiple tools for automatically verifying references. This measure will likely only filter out the laziest and most incompetent of AI slop submissions. It's a very modest raising of the bar, but comes at zero cost to honest researchers.
I expect arXiv will still have problems with slop submissions but, at least, their references should actually exist going forward.
It isn't "cheating" they're concerned with, it's sloppiness. This dictum isn't some sort of AI ban, but instead simply that if there is evidence that it was so low effort that the work includes such blatant problems, it's just adding noise.
> think proof of Riemann Hypothesis autonomously generated (end to end) formally proven - is it allowed or not?
Sorry to be rude, but this seems like a dumb question. I want science to progress. A primary purpose of these journals is to progress science. A full proof of the Riemann Hypothesis progresses science. I don't care how it was produced, if Hitler is coauthor, etc, I just care that it is correct. Whether the authors should be rewarded for whatever methods they used can be a separate question.
Terence Tao had a nice talk from the Future of Mathematics conference posted yesterday [0] that shapes a lot of my own feelings on this matter.
The short of it is he argues how first to correctness shouldn't be the only goal / isn't a great optimisation incentive. Presentation and digestibility of correct results is a missing 1/3 when you've finished generation and verification. I completely agree with him. You don't just need an AI generated proof of the Reimann Hypothesis. You would really like it to be intentional and structured for others to understand.
A really beautiful quote I learned of in the talk is this:
> "We are not trying to meet some abstract production quota of definitions, theorems, and proofs. The measure of our success is whether what we do enables people to understand and think more clearly and effectively about math." - William Thurston
> The penalty is a 1-year ban from arXiv followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue.
This is incredibly good for science. arXiv is free, but it's a privilege not a right!
I'm not seeing this clearly listed on https://info.arxiv.org/help/policies/index.html so it's possible this is planned but not live yet - or perhaps I'm not digging deeply enough?
As a certain doctor once said: the whole point of the doomsday machine is lost if you keep it a secret!
> This is incredibly good for science.
I disagree. It's just one darn hallucinated citation for heaven's sake, not fraud or something. It doesn't account for the substance or quality of their work at all. A one-year ban seems plenty sufficient for a minor first time mistake like this. People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life. That's punitive instead of rehabilitative.
> It's just one darn hallucinated citation for heaven's sake, not fraud or something.
It is fraud.
> It doesn't account for the substance or quality of their work at all.
References are part of the work. If you're making up the references, what else are you making up?
> People make mistakes and a good fraction of them can learn from those mistakes. There's no need to permanently cripple someone's ability to progress their life or contribute to humanity just because an AI hallucinated a reference one time in their life.
A one year ban is not permanent. Having a negative consequence for making poor decisions seems like an inducement to learn from the mistake?
In an ideal world, one would be keeping notes on references used while doing the research that lead to writing the paper. Choosing not to do that is one poor decision.
Having a positive outlook, if asking an AI to provide references that may have been missed, one should at least verify the references exist and are relevant. Choosing not to do that is also a poor decision, even if one did take notes on references used while researching.
> It is fraud.
No, it is emphatically not. Fraud requires intent to deceive.
> A one year ban is not permanent.
...what text are you reading? Nobody was calling the one-year ban permanent, or even against it. I was literally in favor of it in my comment. I explicitly said it is already plenty sufficient. What I said is there's no need to go beyond that. My entire gripe was that they very much are going beyond that with a permanent penalty. Did you completely miss where they said "...followed by the requirement that subsequent arXiv submissions must first be accepted at a reputable peer-reviewed venue"?
Fraud requires intent to deceive _or_ reckless disregard, sometimes called, “conscious indifference”for the veracity of the statement asserted.
If you are using AI-hallucinated references in scientific papers then there is some obvious intent to deceive there
> No, it is emphatically not. D Fraud requires intent to deceive.
I'm about as pro AI-as-a-research--and-writing-assistant and anti AI-witchhunt as they come, but I simply cannot parse what I've quoted here.
Posting slop to arxiv is blatant deception. Posting an article is an attestation that the article is a genuine engagement with the literature. If you're posting things to arxiv that are not sincere engagements with the literature, you are attempting to deceive.
it's very silly, but not a big deal. Arxiv is becoming irrelevant these days anyways.
In fact would be better if they just banned AI, so we could just get off the luddite platforms.
Automated research is the future, end of story. And really it couldn't have come out at a better time, given the increasingly diminishing returns on human powered research.
Poe's law striking hard.
If you cannot be bothered to check your references when writing academic quality papers then you have no place writing them in the first place. The punishment is not chopping off a finger, it is a polite reminder to do the bare minimum.
Well, in the good old days, when we have refereed journals, it would be part of the publishing process.
A "mistake" would be a typo in a real citation. A hallucinated citation is evidence of just plain laziness and negligence, which taints the entire submission.
No it is not. Seriously. All you need for this to happen is for your lab partner to ask AI to add a missing citation that they are already familiar with at the last minute before a midnight submission deadline, and for the AI to hallucinate something else, and for them to honestly miss this. It does not even imply any involvement on your part, let alone that either of you were lazy or negligent on the actual research or substance of the paper. The lack of any sympathy or imagination here is astounding.
There are no deadlines for journal submissions. Even if you felt you were running close to your revisions being due, an email to an editor will probably fix this for you. And what you described is still negligent, not verifying the garbage output bot did not in fact output garbage.
The lack of understanding that you are responsible for the content you create, no matter what tools you use, is what's astounding.
It's not the kind of mistake that is possible unless you're engaging in fraud anyway.
> It's not the kind of mistake that is possible unless you're engaging in fraud anyway.
Seriously? You can't fathom an honest researcher asking for AI to find a citation they know exists, and the AI inserting or modifying a citation incorrectly without them realizing?
If you find evidence of fraud by all means lay down the hammer. Using a single hallucinated citation like it's some kind of ironclad proxy just because you think they must be committing fraud is insane.
I would not necessarily go as far as calling it fraud, but if you cannot even verify that the reference you are citing actually exists, you are not ready to publish research papers.
Deadlines are not an excuse here. Checking whether a cited book, paper, or passage exists is the absolute minimum standard for scientific work, not an optional extra. I have written academic papers myself, and I find it astonishing that people are trying to justify this as if it were some understandable workflow mistake. At that point it is simply slop with academic formatting.
A one-year ban is not a lifetime ban. Maybe six months would also have been enough, but the point is that the author gets time to think about whether they should verify references next time. They can also use that time to manually check every other citation.
if you're not checking citations in the paper youre publishing AND trusting a non SOTA, hallucination prone ai model to come up with sources for it, its probably for the best of everyone that this paper isn't published.
yes there will be rare exceptions but in general i feel like this is a really good addition.
Why would you ask the ai to find a citation you know exists? Just reach for that citation.
If you are citing a work you paste a citation to that work. If you are bullshitting you ask an AI to come up with a citation. Jesus, there is zero reason to ever "generate a citation" if you are not, in fact, commiting fraud.
Yes, it is fraud
A citation is where you derived knowledge... If you haven't checked it and you are submitting something that should represent a ton of labour (and which will consume labour to review), you don't understand what you're doing. It is not just crossing T's and dotting I'd.
Your being set behind is less important than the fact that your publishing is setting everyone else behind.
Such a banned person is being helped to "step out of the way", and someone more competent will assuredly step forward to consume the limited maintenance labour more thoughtfully
> Your being set behind is less important than the fact that your publishing is setting everyone else behind
One hallucinated citation does not in any way imply anyone is being left behind. All it means nobody is checked that particular line of the manuscript after it was written. The rest of the paper could still be solid and treated accordingly. If you find evidence of the contrary, of course treat it accordingly, but this is so obviously not that.
> One hallucinated citation does not in any way imply anyone is being left behind.
The parent said “setting” others behind, which refers to lost time.
Being “left” behind implies a degraded trajectory, which is defined not by time lost, but by the final destination.
Different but related things (e.g. lost time can indeed affect your final destination, for instance, after growing old correcting a scourge of hallucinated citations - which should have been table stakes all along).
No. It's fraud.
You clearly misunderstand. You cite a work in your paper because you have read that work, and build upon it or want to refer to it to back up a specific claim. Generating references is fraud period, because you are implying that you have read a work when in fact you just asked an AI "please insert some reference-shaped text here" to make it look like a proper paper. It is sadly not a necessary, but certainly a VERY sufficient, reason to conclude a paper is fraudulent.
Don't use AI? Problem solved?
You are being ironic right?
https://xcancel.com/tdietterich/status/2055000956144935055
> Our Code of Conduct states that by signing your name as an author of a paper, each author takes full responsibility for all its contents, irrespective of how the contents were generated (Dieterrich, T. G.)
There needs be to a careful vetting before such adverse actions. If somebody includes a name and pushed it without express permission, does everyone get the ban? I agree that implemented the right way, this is good.
Had a colleague submit a paper with literal AI slop left in the text, got hit with a nasty revision request. Check your drafts before you submit, people. The reviewers will find it.
Also check your LaTeX comments, Arxiv makes those publicly visible!!!
I'm a screen reader user and usually read papers as raw TeX. I've seen everything: slurs, demeaning comments towards reviewers and professors, admissions of fraud, instructions to coauthors to commit further fraud before paper submission to mask the earlier fraud... it's all there. There's far less of it than I would think, definitely <1% of papers, but it's there.
I think it would be useful to run an LLM anti-fraud pass on the TeX source of all new arxiv papers. It wouldn't catch everything, but it would catch some of the dumbest fraudsters.
On the positive side, you can also find stronger claims that didn't survive review, additional explanations that didn't make the cut due to the conference's page limit, as well as experimental results that the authors felt weren't really worth including. Those need to be approached with an abundance of caution, but are genuinely useful sometimes.
https://xcancel.com/leaksph
Sad the suggestion here is to just disguise the slop to make it harder for reviewers to spot rather than not submitting slop to begin with.
Good; academic literature is in crisis because of all of the slop. Forcing some consequences on easily-detectable hallucinations can only be a good thing
It's not just AI, though. I did a doctorate in physics about 40 years back, and bad references were a problem back then.
Doesn't matter if it is AI hallucinations or entirely human scientific fraud, the problem is the same, and the solution works fine for both cases.
If you can't validate that your bibliography is full of real articles, you shouldn't get published.
LLMs have just poured gasoline on the fire.
"Bad", like, you literally just made them up? I hope that would have been a problem.
In what way? Surely something like the source not quite saying what was cited, or mixing up citations, rather than inventing them outright?
That, and mixing reference details from multiple sources and messing it up.
Let's say you read a paper on Arxiv but cite the version that was submitted to a journal or conference, without realizing that the authors made changes to the version they submitted and forgot to upload them to Arxiv.
Imagine how bad they are now then.
Yes and ffs arrows kill people too but we don't bring that up every time we talk about what to do with guns.
It seems a good idea to ban cheating, but how hard is it, especially in new reasoning/agents contexts to validate references?
The deeper question is whether legitimate AI generated results are allowed or not? Test - In the extreme - think proof of Riemann Hypothesis autonomously generated (end to end) formally proven - is it allowed or not?
This is not about banning cheating, it’s about banning inaccurate information.
It is allowed as long as it’s verified.
The thread specifically points out that if authors can’t be arsed to simply proofread their text the rest can not be trusted either.
It’s a simple heuristic against low quality submissions, not an anti-ai measure.
If you use AI correctly, nobody should be able to tell that it was used at all.
You don’t need to solve everything, catching a few thousand non existent citations with such a policy is on its own a net benefit.
There already exists multiple tools for automatically verifying references. This measure will likely only filter out the laziest and most incompetent of AI slop submissions. It's a very modest raising of the bar, but comes at zero cost to honest researchers.
I expect arXiv will still have problems with slop submissions but, at least, their references should actually exist going forward.
It isn't "cheating" they're concerned with, it's sloppiness. This dictum isn't some sort of AI ban, but instead simply that if there is evidence that it was so low effort that the work includes such blatant problems, it's just adding noise.
> think proof of Riemann Hypothesis autonomously generated (end to end) formally proven - is it allowed or not?
Sorry to be rude, but this seems like a dumb question. I want science to progress. A primary purpose of these journals is to progress science. A full proof of the Riemann Hypothesis progresses science. I don't care how it was produced, if Hitler is coauthor, etc, I just care that it is correct. Whether the authors should be rewarded for whatever methods they used can be a separate question.
Terence Tao had a nice talk from the Future of Mathematics conference posted yesterday [0] that shapes a lot of my own feelings on this matter.
The short of it is he argues how first to correctness shouldn't be the only goal / isn't a great optimisation incentive. Presentation and digestibility of correct results is a missing 1/3 when you've finished generation and verification. I completely agree with him. You don't just need an AI generated proof of the Reimann Hypothesis. You would really like it to be intentional and structured for others to understand.
A really beautiful quote I learned of in the talk is this:
> "We are not trying to meet some abstract production quota of definitions, theorems, and proofs. The measure of our success is whether what we do enables people to understand and think more clearly and effectively about math." - William Thurston
[0] https://www.youtube.com/watch?v=Uc2zt198U_U