2023 research did not focus on predicting future LLM software

In short: Despite claims about a 2023 study making many assumptions about “anticipated LLM-powered software,” widely cited 2023 research mostly measured how these AI models work in the present.

What's going on

Search results for 2023 research that matches the idea of “making a lot of assumptions” about future LLM-powered software do not turn up a clear match. Instead, the most visible 2023 papers focus on what large language models, or LLMs, can do and how they do it.

One example is a February 2023 paper from researchers at MIT, Google, and Stanford about “in-context learning.” That is when an LLM can pick up a new task from a few examples in the prompt (like showing a student two solved problems and then asking them to solve a third). The researchers argued the model may be doing something closer to internal learning than simple pattern replay.

Other 2023 writing is more about practical uses and careful surveys. Reviews in statistics and data science discussed using LLMs for things like generating code and fixing errors, and they described the transformer design (the structure behind many chatbots) in a general way. A separate survey of 102 papers looked at how LLMs were being tried in software testing, such as generating unit tests, and it included a roadmap of open problems, but it leaned on evidence from existing work rather than big guesses about the future.

What to watch

It is common for newer articles to mention future AI software while citing older studies for background. When you see a claim that a study “assumed” a future full of LLM-powered tools, it is worth checking whether the study actually made forecasts, or whether it simply explained today’s capabilities and limits.

Source: Arstechnica