I feel like the LLM is the invention, but next-word prediction, or the fundamental concept beneath it is the discovery.
Listen to Ilya Sutsksver and you’ll notice that he is fascinated by the fact that something that looks a lot like intelligence appears just from training a model to predict its own inputs--no special objective function required.
This same principle may explain the mechanism for learning in the human brain too (see Karl Friston).
This is a great contribution. Thanks for sharing the link. It is a more erudite break down than Bezos offered and the analogy is stronger. I particularly like the emphasis of inputs over processing. That sounds right. The only problem I have is semantic. A discovery suggests the outputs already exist. Just because they can exist given a set of inputs does not mean they are there to be discovered. To discover something that does not yet exist, even if the elements that compose it do exist, is paradoxical.
With that said, this is where the camera angle (or photography) is much stronger. Angles, lighting, arrangement, are techniques that bring something that does not exist into existence, at least for a moment when the variables align. But we can set that all aside. Whether Bezos is precisely correct (he is not) or or Rao is precisely correct (he is much closer) in the use of the term discovery is unimportant compared to the larger point. The camera (or more broadly, photography of data) offers the capability to assemble something that did not exist and may never again exist AND to record it.
Generative AI in particular is a tool for invention that may also yield discoveries.
Well, as a next word predictor, we can all go astray! 😀
I differentiate between the potential to exist and existence. If the training data was never processed just so, and the RLHF done the same way, and the prompt entered in that way, the output would not exist. It's kind of a silly semantic argument about the definition of the term discovery, but we need some tools to anchor ideas and semantic meaning is very useful.
For Chess, there are positions that may have never been achieved. Therefore they never existed. You can still discover them, but only after they are achieved, either by others or by you. For art, David or the Mona Lisa didn't exist before Michelangelo and Da Vinci created them. The materials and techniques existed previously (analogy to training data). The artwork didn't exist until the artists created it. In an indirect way they discovered something that could exist, but the word for that is invented!
Granted, I'm not quite sure why I predicted the previous word sequence.
I feel like the LLM is the invention, but next-word prediction, or the fundamental concept beneath it is the discovery.
Listen to Ilya Sutsksver and you’ll notice that he is fascinated by the fact that something that looks a lot like intelligence appears just from training a model to predict its own inputs--no special objective function required.
This same principle may explain the mechanism for learning in the human brain too (see Karl Friston).
The discovery may be that the next word prediction actually resembles intelligence and the LLM with next-word prediction is the invention!
This is a great contribution. Thanks for sharing the link. It is a more erudite break down than Bezos offered and the analogy is stronger. I particularly like the emphasis of inputs over processing. That sounds right. The only problem I have is semantic. A discovery suggests the outputs already exist. Just because they can exist given a set of inputs does not mean they are there to be discovered. To discover something that does not yet exist, even if the elements that compose it do exist, is paradoxical.
With that said, this is where the camera angle (or photography) is much stronger. Angles, lighting, arrangement, are techniques that bring something that does not exist into existence, at least for a moment when the variables align. But we can set that all aside. Whether Bezos is precisely correct (he is not) or or Rao is precisely correct (he is much closer) in the use of the term discovery is unimportant compared to the larger point. The camera (or more broadly, photography of data) offers the capability to assemble something that did not exist and may never again exist AND to record it.
Generative AI in particular is a tool for invention that may also yield discoveries.
Well, as a next word predictor, we can all go astray! 😀
I differentiate between the potential to exist and existence. If the training data was never processed just so, and the RLHF done the same way, and the prompt entered in that way, the output would not exist. It's kind of a silly semantic argument about the definition of the term discovery, but we need some tools to anchor ideas and semantic meaning is very useful.
For Chess, there are positions that may have never been achieved. Therefore they never existed. You can still discover them, but only after they are achieved, either by others or by you. For art, David or the Mona Lisa didn't exist before Michelangelo and Da Vinci created them. The materials and techniques existed previously (analogy to training data). The artwork didn't exist until the artists created it. In an indirect way they discovered something that could exist, but the word for that is invented!
Granted, I'm not quite sure why I predicted the previous word sequence.