Peer reviewers and journal editors have a new tool to automatically check the references in manuscripts, developed by Jennifer Kleiman at the University of Georgia. You copy the list of references from the manuscript, paste it into the tool’s web page, and it searches the internet for each of the cited works. If it finds the work, then it corrects the reference format as needed to conform to APA style. If it cannot find the work, then it informs you so you can resolve whether the reference is genuine.

I tried out the tool on a manuscript that was recently assigned to me as Associate Editor. The tool is at https://jenkleiman.com/reference-checker/  I copied the list of references page by page from the .pdf of the manuscript and pasted them into the Paste References box on the tool’s web page. In a few cases the tool mistook the last line of a citation for the start of a new one. In all the other cases it found the work the citation referred to. It corrected the format of the citations to comply with APA style, and in several cases it inserted information such as DOI number that had been left out of the reference in the manuscript.

As a further test, I gave the tool some fake citations that I made up. I based the fake citations on correct citations to real works, but changed some of the information such as title or author. In many cases, the tool found the real work that I had based the fake citation on, and showed me a corrected citation that reversed the errors I had inserted. 

But if I made the fake citation different enough from a real one, such as by changing the author and title and leaving out the DOI number, then instead of returning a corrected version the tool labeled the citation as “hallucinated”; that is, the kind of reference to a nonexistent work that an Artificial Intelligence program might make up. Kleiman notes that there is a grey area between a citation that is close enough to matching a real work that is should be corrected, and one that is different enough from any real work that it should be flagged as a hallucination. https://jenkleiman.com/reference-checker/ 

The tool uses an Artificial Intelligence program, which is called Gemini and operated by Google. As Kleiman explains, the tool:

“…uses Gemini 3.0 Pro with search grounding. Instead of just asking ‘is this reference real?’, it performs web searches to verify each citation exists. This mimics how humans do it- we google the paper title, check authors, verify the journal and year, maybe search multiple times to find all the details. The AI does exactly that, but systematically for every reference…search grounding nearly eliminates AI hallucinations. The AI isn’t relying on training data, just what it found on the web.” https://jenkleiman.com/reference-checker/ 

Kleiman does not charge to use her tool, but you need a Gemini API key for Google to run your queries from the tool. The tool’s website includes a link and instructions to get an API key, which is not complicated. You need to give Google a payment method such as a credit card, but when I signed up the API key came with $300 of free usage, so I have not paid anything so far.

As powerful as the tool is, it does not address another problem with references that I have seen as an editor and peer reviewer – citations that point to a work that is real but does not substantiate the assertion that the manuscript cites it for. This still requires human judgement.

Jennifer Kleiman’s tool for verifying and correcting references is very impressive, and I think it should be a routine part of the workflow for evaluating manuscripts that are submitted to journals.