How do you keep up with AI research?

Keeping up with AI research means building a routine for throwing papers away, not a routine for reading them. More than a thousand papers a week land in the three AI categories on arXiv alone, and nobody reads that. The routine that works runs three filters in sequence — source, abstract, figures — and ends with one paper read end to end each week. Everything else you close in seconds, on purpose.

Where the papers are has a short answer, because almost all of AI is open and free: finding papers without paying is an afternoon’s work. The hard part is the second one, which is what to do with a thirty-page PDF once it’s open, and that’s the subject of reading an AI paper without being a researcher.

This piece is the map of the territory. The benchmark score, which is the weakest available signal that a result matters, has a piece of its own in saturated benchmarks.

Where AI research gets published

In most of science the path runs submit, wait for peer review, publish. In AI you publish first, and review happens afterwards, in the open, if it happens at all.

The default channel is arXiv. An author uploads a PDF, a moderator checks that the text is scientific and filed under the category it claims, and a day or two later the work is live with a permanent identifier. Moderation is not peer review. Nobody checked whether the experiment runs, whether the comparison is fair, or whether the number in the table is right.

Volume is the number that organises everything else. In June 2026 arXiv as a whole took in 32,040 submissions¹. In the three categories that matter to anyone working with language models, that month’s listings held 5,845 entries in cs.AI, 5,229 in cs.LG and 2,718 in cs.CL, checked in July 2026. Those three overlap, since a paper usually appears under more than one category. Even netting the overlap out, it comes to something like 250 a weekday.

The conference comes later, when it comes. NeurIPS, ICML and ICLR cover machine learning broadly; ACL and EMNLP cover language; CVPR and ICCV cover vision. Months pass between submission and presentation, so the paper you read on arXiv today only counts as published the following year. And a good share of the work that changed practice most never went near a conference: lab technical reports, signed by an institution, with no outside reviewer.

There is a fourth source almost nobody uses and everybody should. OpenReview publishes the ICLR reviews, the scores and the authors’ rebuttal, next to the submission itself. Reading the reviews of a paper that impressed you is the fastest exercise there is for calibrating your own reading.

The funnel: from a thousand a week to one

Nobody keeps up with that volume by reading. You keep up by filtering, and the filter has to discard more than 99% without guilt.

Source. You don’t follow arXiv, you follow people. Twenty to forty names across authors, labs and two or three human aggregators cut the day from 250 down to about 40. It’s the crudest filter and the highest-yield one.

Title and abstract. Ten seconds per item, with a single question: does this change a decision I’ll make in the next three months? “This is interesting” doesn’t count, because almost everything is interesting. Of the 40 a day, roughly 10 survive the week.

Figure and table. Two minutes. Skip the prose, go to Figure 1, which is usually the diagram of the method, and to the first results table. If the figure doesn’t explain the method on its own and the table has no recognisable baseline, close it. Two or three survive per week.

The full read. Forty minutes, one paper, once a week. Fifty a year.

Reading a paper in three passes

The most useful method here didn’t come from AI. Keshav laid out a three-pass approach to reading any computing paper in a two-page 2007 article in SIGCOMM Computer Communication Review, and it has aged well².

The first pass takes five to ten minutes. Read the title, abstract and introduction, the section and subsection headings and the conclusion, then run your eye over the references ticking off what you’ve already read. By the end you can answer Keshav’s five Cs: category, context, correctness of the assumptions, contributions and clarity. Nearly every paper dies here, and that is what the pass is for.

The second takes up to an hour. Read carefully and skip the proofs. The most valuable instruction concerns the graphs: are the axes labelled, do the results carry error bars, does the conclusion survive the variance? Keshav says mistakes of that kind separate rushed work from excellent work, and it’s still the check that eliminates the most papers per minute spent. After the second pass you can explain the argument and its evidence to someone else without having understood the details.

The third pass means re-implementing the paper in your head, making the same assumptions the authors made and comparing your reconstruction against what they wrote. Keshav puts it at four or five hours for a beginner and about an hour for an experienced reader.

This is where the adaptation for practitioners goes: the third pass becomes running the code. Almost every AI paper worth the time ships a repository, and cloning it, running the example and changing one hyperparameter buys the same understanding in less time. The question that pass answers never changes: what would have to be true for their result to hold in my setting?

The papers that make up the base

There’s a small set of papers that nearly every AI text cites without explaining. Grouping them by layer works better than ordering them by date, because chronology mixes contributions of completely different kinds.

What the model is covers architecture and scale. Attention Is All You Need proposed in 2017 an architecture built on attention alone, dropping recurrence and convolution entirely, and measured it on translation: 28.4 BLEU on WMT 2014 English-to-German, more than 2 BLEU above the previous best including ensembles, and 41.8 BLEU on English-to-French after three and a half days of training on eight GPUs³. The founding paper of modern generative AI was a machine translation paper on a modest budget.

How it gets shaped covers everything after pre-training: instruction following, human preference, alignment, and the cheap adaptation techniques. It’s the layer that explains why two models with nearly identical architectures behave so differently.

How you use it covers what runs at inference time without touching a single weight: chained reasoning, tool use, retrieval. It’s the layer closest to anyone building a product, and where prompt engineering lives.

The reading order that works runs backwards. Start from the paper that changes what you’d do tomorrow, then follow the citations down when you hit missing context.

Signal and noise

Lipton and Steinhardt catalogued four recurring patterns in the machine learning literature in 2018: failing to separate explanation from speculation; failing to identify where an empirical gain actually came from, crediting a new architecture for what came from hyperparameter tuning; mathematics that impresses rather than clarifies; and terms of art borrowed with their colloquial baggage attached. They put all four down less to bad faith than to the community’s rapid expansion and the thinning of the reviewer pool⁴.

The second pattern is the one you meet most often. Dodge and colleagues showed in 2019 that a test-set score on its own doesn’t support a claim about which model is better: they proposed reporting expected validation performance as a function of compute budget, and found published comparisons where the authors would have concluded the opposite with more or less hyperparameter search. In the same work, the compute needed to reach published results ranged from hours to weeks across papers⁵. A table without that information doesn’t let you compare anything.

The third signal is the choice of benchmark. Dehghani and colleagues named this the “benchmark lottery”: the relative ranking of methods can shift substantially just from which tasks make it into the evaluation, and they demonstrated it across subfields from natural language processing to information retrieval and reinforcement learning⁶. A result that exists on one benchmark, particularly one that looks chosen after the experiment, is not a result.

On the other side, what says the forty minutes are worth spending: public code and weights, a baseline tuned with the same effort as the proposal, and a limitations section that names a concrete cost.

Where the routine fails

Peer review is not the filter you think it is. In the NeurIPS 2021 consistency experiment, 10% of submissions went to two independent committees: the committees disagreed on accept or reject for 23% of the papers, and about half of the accept list would change if the process were rerun from scratch. The authors also note that making the conference more selective would increase the arbitrariness⁷. Revisiting the equivalent 2014 experiment, Cortes and Lawrence estimated that 50% of the variation in quality scores was subjective, and seven years on found no correlation between the scores of accepted papers and their citation impact. The process identified weak papers well and strong papers badly⁸.

Reproducibility is still your job. NeurIPS built a programme in 2019 around a code submission policy, a community reproducibility challenge and a mandatory checklist in the submission form⁹. That improved what gets declared. The distance between “the repository exists” and “the table’s number comes out of the repository” is still where most results die.

Null results never reach you. The technique that didn’t work doesn’t become a paper, so the literature shows a biased sample of what was tried. You will reimplement something other teams already discovered is a dead end.

The scale of the result may not be your scale. A gain measured on a 70B model says little about what happens on the model you call through an API. It cuts both ways: techniques that only emerge at large scale, and techniques that only pay off at small scale.

The paper that will matter doesn’t look important today. Citations take years to accumulate, and on publication day the available signal is weak by construction. No triage routine fixes that, and insisting on fixing it is the fastest route to reading more and understanding less.

A two-hour-a-week routine

1. Build the sources before the habit. Subscribe to the cs.CL and cs.LG feeds, follow twenty authors whose work you have already used, and one or two human aggregators. Bad sources collapse the rest of the routine within a fortnight.
2. Fifteen minutes a weekday, not an hour on Saturday. Letting seven days pile up turns ten seconds per abstract into a lost afternoon.
3. Log what you throw away. One line per closed paper, with the title and the reason. After a month that file shows you which filter is miscalibrated.
4. One full read a week, on the calendar. Without a fixed slot it doesn’t happen, and triage without reading is just headline consumption.
5. Run the code before you believe the table. Twenty minutes here beats a third reading of the text.
6. Write three sentences. What the authors claim, on what evidence, what changes for you. If the third sentence comes out empty, the filter let something through.
7. Review your sources quarterly. People change fields and labs change focus. A source list from two years ago is delivering you the field as it was two years ago.

Step 3 is what separates people who get better at triage from people who merely read more. Without a record of what you discarded, you never find out that you’ve been reflexively rejecting the one subfield that would change your work.

What it costs

Two hours a week comes to roughly a hundred hours a year. Keshav estimated in 2007 that a typical researcher spends hundreds of hours a year reading²; a hundred hours is the slice of that which fits around another job, and it’s enough to stay current on anything that affects an engineering decision.

In money, close to nothing. arXiv, OpenReview’s reviews and the NeurIPS, ICML, ICLR and ACL proceedings are all open. What’s left is the closed journal literature, which in AI is small and almost always has a matching preprint; given a DOI, Unpaywall locates the legal open version. A budget for access to AI papers is a problem other fields have and this one doesn’t.

The real cost is attention: every hour reading is an hour not building. Read too much and it turns into scholarship with no outlet; read too little and you spend three weeks reimplementing what a 2023 paper settles in an afternoon.

Footnotes

1. arXiv publishes its monthly submission count going back to 1991. June 2026 closed at 32,040 submissions across all fields, against 21,206 in December 2024. ↩

2. Keshav (2007) sets out the three-pass method and estimates each pass: five to ten minutes, up to an hour, and four to five hours for beginners on the third. ↩ ↩²

3. Vaswani et al. (2017) proposed an architecture built on attention alone, with no recurrence or convolution, measuring 28.4 BLEU on English-to-German and 41.8 on English-to-French for WMT 2014, after three and a half days of training on eight GPUs. ↩

4. Lipton and Steinhardt (2018) catalogue four patterns: conflating explanation with speculation, failing to identify the source of empirical gains, mathematics that obscures, and misuse of technical terms. ↩

5. Dodge et al. (2019) showed that a test-set score alone doesn’t support a comparison between models, and found published cases where the conclusion flips under a different hyperparameter search budget. ↩

6. Dehghani et al. (2021) showed that relative performance between methods shifts significantly from the choice of evaluation tasks alone, across several machine learning subfields. ↩

7. Beygelzimer et al. (2023) report 23% disagreement between two independent committees at NeurIPS 2021, and estimate that half the accept list would change on a rerun. ↩

8. Cortes and Lawrence (2021) estimated that half the variation in the 2014 review scores was subjective, and found no correlation between score and citation count among accepted papers. ↩

9. Pineau et al. (2020) describe the NeurIPS 2019 reproducibility programme: a code submission policy, a community reproduction challenge and a mandatory checklist. ↩