>Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel)
Would it?
Why would "creating a novel" by a human not itself be text generation based on prediction on what are the next good choices (of themes, words, etc) based on a training data set of lived experience stream and reading other literature?
What is the human predicting there? Why would it need to be a prediction task at all? How about a dada-ist poem? Made-up words and syntax? If it is prediction but the criterion for "what is a good next choice" can totally be made up on the fly - what does the word "prediction" even mean?
>Why would it need to be a prediction task at all?
What else would it be?
Note that prediction in LLM terminology doesn't mean "what is going to happen in the future" like Nostradamus. It means "what is a good next word given the input I was given and the words I've answered so far".
>How about a dada-ist poem? Made-up words and syntax?
How about it? People have their training (sensory input, stuff they're read, school, discussions) and sit to predict (come up with, based on what they know) a made-up word and then another.
That is a meaningless definition of prediction if "what is a good next word" has an ever changing definition in humans (as everything would fulfill that definition).
That's the very definition of production in an LLM.
What does "has an ever changing definition" mean?
And why "everything would fulfill that definition"?
At any time whats the "good next word" is based on the state created by our inputs thus far (including chemical/physiological state, like decaying memories, and so on). And not only not "everything fullfil it", but it can be only a single specific word.
(Same as if we include the random seed among an LLM output: we get the same results given the same training and same prompt).
"it can be only a single specific word" - that is incorrect as a human can change the process to generate the next word, up to and including, using a random process to create or select the next word (i.e., any word would be fine).
You could say the process chosen is somehow predetermined (even if the choices then are all made by using randomness), but then really the word "prediction" has very little meaning as the criteria to what is a "good next word" have a nearly unlimited and ever changing range as the generating process changes.
>"it can be only a single specific word" - that is incorrect as a human can change the process to generate the next word, up to and including, using a random process to create or select the next word (i.e., any word would be fine).
That's also exactly what an LLM does.
It's still only a single specific word if (as I wrote above) you take the seed into account too (i.e use the same input, including same random seed value).
If you mean to answer "yes, but LLMs use a random number generator, whereas humans can actually pick a word at random" I'd answer that this is highly contested. Where would the source for such randomness be in the universe (exept if you beg the question, and attribute it to an "soul" that is outside the universe)?
Claiming that the universe has no randomness is a very strong claim and moves beyond our ("standard") understanding of quantum mechanics. For example, a human could be using radioactive decay to sample randomness (and such devices are available).
An LLM is bound to what an LLM can, while humans can construct and use tools to go beyond what humans can do. Being a universal function approximator does not give access to all processes in the natural world.
> Why would "creating a novel" by a human not itself be text generation based on prediction on what are the next good choices (of themes, words, etc) based on a training data set of lived experience stream and reading other literature?
Unless you're Stephen King on a cocaine bender, you don't typically write a novel in a single pass from start to finish. Most authors plan things out, at least to some degree, and go back to edit and rewrite parts of their work before calling it finished.
Sure, but if we're talking about editing an entire book eventually the fine details do matter. That, and presumably human authors' abstraction/memories of their books are stored in some more compact form than language tokens. Though we can't be sure about that.
Would it?
Why would "creating a novel" by a human not itself be text generation based on prediction on what are the next good choices (of themes, words, etc) based on a training data set of lived experience stream and reading other literature?