This blog post from someone named David Oks pines for a time before citations became a central pillar in the effort to advance scientific publishing. Thank you Mr. Oks for sharing your views, which got me thinking. I am not sure I agree 100%, because an important part of civilization and science is our ability to document the state of knowledge so that each successive generation can build on it. But the point is taken that the extreme focus on gaming this one metric (number of citations, citation scores of individual academics and journals) has become an end in itself, rather than a means for advancing civilization and science.

I do think AI can be very good at improving the state of the “literature review”. Every scientific article starts with a summary of literature on the topic, which the authors then typically build on (although, some articles are just literature review). A human author can spend years sifting through a vast amount of literature relevant to a topic, particularly a novel or interdisciplinary topic, trying to find those few needles in a haystack that are really the most, say, 100 highly relevant papers, and synthesize them into a foundation that can be built on it. I have done this, and it is actually a very fun thing to do (for my personality type, I suppose, not for everyone, but there are many like me…) Through this process, you gradually build and refine your own unique mental model on a subject which then can become the foundation for your personal unique contribution to human progress. However, even spending years, you can’t come close to looking at everything possibly relevant, and you can miss some of those needles. An AI should be able to look at literally the entire haystack, find and synthesize the needles, in a matter of minutes or at most hours, compared to months or years for a person. The only sad thing here is that the best mental models in the brains of the smartest humans might not be built in the same way. Overall though, I suspect the rate of progress can be increased significantly.

What is an alternative to the citation regime according to Mr. Oks?

So I suspect that we’ll have to fundamentally rethink the institutions of scientific life for the age of strong AI. Perhaps, as AI makes it possible to do much more science much more quickly, the culture of science will become more like the culture of engineering—faster, more collaborative, less interested in priority claims. In such a world, the most efficient unit of scientific contribution might be a living document, perhaps even just a GitHub repo: something with data, code, analysis, and a thin narrative layer that AI scientists could read, regenerate, or update as needed. And citations, in this world, could ultimately become obsolete. Journal articles would survive, though perhaps they’d become something closer to definitive pronouncements on major breakthroughs or on the state of knowledge in a given domain—a bit like what scientific books were before the rise of journals. In a world where science is much more productive than it is today, legitimacy will be the scarce factor in the production of useful scientific knowledge.

I am not sure I have experienced the culture of engineering described here in my own engineering career, but this may be referring to something more like “product development” rather than the public infrastructure and environmental planning type work I do. Anyway, my idea of AI-backed knowledge synthesis plus this idea of open data and code may be on to something.