Probably unrelated, but I tried to get ChatGPT to write me some code to programmatically control the details of a column filter in an Excel spreadsheet in PowerShell.
Nothing it tried worked, it got close, but it didn't work.
Finally I found some C# code that fixed the problem, and I pasted that code into ChatGPT, asked it to read it, and then fix the problem in PowerShell.
It said it understood the solution, updated the script, and it worked perfectly.
For some reason that behavior was pretty eye opening. Providing material in the question that it wasn't trained on made it solve it.
It's understandable how it did it from language training, it just felt very cool that LLM's can do that.
Interesting anecdote. I think there's a common theme with current LLMs, that people focus unreasonably much on "knowledge retrieval" from the models (1) and under-hype and under-appreciate the "language model" part.
These things are really easy to anthropomize, partly because they are good at "talking" and "articulating". So good that we tend to just accept that magical, enormous feat of statistical engineering as a trivial building block. But it's a brick made of gold.
Translating (from natural language to code, from text to audio, from image to image, one natural language to another), editing, summarizing, expanding/extrapolating is what these models do.
The inherent "knowledge" is just context.
(1) Vector embedding is in my view a little different - it's a form of semantic cataloging (akin to Dewy decimal) - and certainly enables search.
But "data retrieval" (who was us president in 1984) directly from the models isn't really all that interesting IMNHO.
It's why the "hallucination" concern is IMO not a helpful way for people to conceive of the remaining challenges. These things aren't meant to be search engines, search engines already exist, and I don't understand the utility of using the model itself as a search engine (I do understand having the model search for you and summarise what it finds, like an integrated assistant). The model is better conceived of as the part that does the thinking, and to work on knowledge that you want to be reliable you have to have that knowledge accessible to the model in some other format. We know how to store information, generally. What is interesting and useful about this model is not their ability to off-the-cuff recall facts without access to any resources, that's a party trick in humans and AI. What is interesting about them is their ability to be given a piece of information, understand it, and use that information for logical reasoning. That provides the ability to answer questions about the information, use the information in conjunction with other information, etc. That is new for a natural language interface, and it has really interesting implications for what we can build with it.
Nothing it tried worked, it got close, but it didn't work.
Finally I found some C# code that fixed the problem, and I pasted that code into ChatGPT, asked it to read it, and then fix the problem in PowerShell.
It said it understood the solution, updated the script, and it worked perfectly.
For some reason that behavior was pretty eye opening. Providing material in the question that it wasn't trained on made it solve it.
It's understandable how it did it from language training, it just felt very cool that LLM's can do that.