TMLR Editor Asked 10 Desk-Rejected Authors About Their Own Papers; 3 Could Not Answer Basic Questions
- Of 10 TMLR submissions slated for desk rejection, authors of 3 could not answer basic questions about their own paper, 3 more handled high-level ideas but stumbled on technical details, 1 withdrew, 1 cited other commitments, 1 scheduled a meeting and did not show, and 1 answered every question (though the paper still had a major flaw).
- TMLR editor-in-chief Nihar B. Shah ran the meetings himself over two weeks and says the exercise took 20 to 25 hours for 8 papers, making the format hard to scale against current submission volumes.
- Shah frames the result as a check on papers that may be AI-generated or heavily AI-assisted: if authors cannot explain the technical content, they likely could not have verified it.
- The meetings also undercut authorship as a credit signal and raise doubts about these same authors being asked to review other people's papers, a common requirement at journals and conferences.
- TMLR reads the outcome as evidence its desk rejection process works, alongside author submission quotas tied to recent submission patterns and added emphasis on clear writing.
Hacker News opinions
Peer review always had problems, but LLMs writing good-sounding papers is a new one. I'm fine with AI in science, I'm not fine with authors who don't understand their own work. High-repute journals might need to add an oral exam to acceptance.
Only one out of ten authors could answer questions properly. Authenticity and trust are going to command a premium in the age of slop.
Sure, but how many authors of papers slated for desk rejection ten years ago could answer questions either? You need that baseline before you blame AI instead of just bad papers.
I'd rather see this run on published papers. What fraction of accepted papers have authors who can't answer basic questions about them?
Heads up to the author, that greCAPTCHA link on cs.cmu.edu is dead. There's a live version on arxiv.
Funny, I proposed a CAPTCHA for scientific publications back in June. Everyone treated it as a joke.
The real value of a paper is rarely the knowledge it adds, it's the process of doing the research and training grad students. AI papers shortcut all of that. Academia made papers the currency of success, so it has a lot to answer for.
I disagree. Papers are just an easy medium for verification and sharing knowledge. LLMs short-circuit everything, so what's your fix for academia?
The Atlantic has been covering this. Journals are the pipes knowledge flows through, and they're getting clogged with AI slop. COVID plus AI plus funding cuts point to a real disruption coming.
I'm outside academic publishing, so someone explain: how does the industry check submissions aren't partly AI-generated? Is defending your paper in an interview standard? If not, why not? It's worrying that anyone would even try.
Look at publication rates, they went up. If everyone weren't using AI the numbers wouldn't move like that.
No, interviews aren't standard, but maybe they should be. Postgrad degrees require an oral defense and I never minded defending my thesis.
I've published a few papers and this sounds extremely unusual to me. It's clearly a special thing this editor chose to do. Also brutally time-consuming. Journals don't have to be perfect, peer review is just the entry barrier, science sorts itself out through citations and talks.
The Medium comments are on point too. Running this on accepted papers would be a good control. I'd keep a private blacklist of authors who waste reviewer hours proving they're not legit, though the list itself would be a problem.
That blacklist fails immediately. Humans and agents can spin up new author accounts, or pay for them.
The problem isn't the paper, it's the author's understanding of it. A great AI-written paper is still a good paper, just not the named author's paper. Cited work doubles as proof and as credit, and treating citation as currency is what drives most academic fraud.
This looks like plagiarism to me. If you can't answer basic questions about the paper, how can you claim you wrote it? If you used AI and then actually learned the content cold, fine, that's a different thing.
The genie is out of the bottle. Use LLMs to review too: summarize, find the biggest weaknesses and strengths, hand that to a human reviewer. I'd genuinely go to a conference that encouraged LLM-written papers reviewed by LLMs.