The Bitter Lesson
If you're building anything that involves AI, then you have to know about the bitter lesson.
The bitter lesson is an article by Rich Sutton from March 13, 2019. This sounds like ancient AI lore, but it is only six years old.
Rich Sutton, a pioneer of reinforcement learning and a Turing Award winner, wrote a short essay on what 70 years in AI has taught him, and the best quote from it is also the first sentence.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin."
In other words, as computation gets cheaper, you can throw more computing power at the problem, and it will outperform any of your optimisations.
It doesn't matter if you've found this crazy new prompting technique for improving your AI app.
Time spent optimising will ultimately be a waste.
Time that you could have spent waiting for the models to improve.
This is the bitter lesson.
We are all just wasting our time. Leveraging domain knowledge is pointless. Nothing really matters in AI research, apart from improving the power of the machines.
This is controversial, to say the least.
If you see someone on the internet talking about the bitter lesson, now you know what they're talking about.
Here's the original essay.