Ashish Tulsian

A lot of custom post-training work being done today will be deprecated in favor of smarter models and continual learning 10-100x intelligence with continual learning is around the corner Compute, time and effort to heavily customize models is not a great use of resources while these capabilities are emerging. They will disrupt the assumptions people are building around today. It analogizes to adapting open source BERT variants to build models for language in 2019. LLMs disrupted this by solving the same problem through in-context learning with much less data and effort. 1. The base model will become fundamentally smarter, there are 10-100x improvements to be made in pre-training alone. 2. Continual learning algorithms fundamentally change the interface by which models are updated, how easy they are to maintain and use, and how much effort is required to adapt them to your domain and context. 3. The context and data sources from the domain, and the judgment of what good looks like will of course continue to matter. But even there, the level of data curation and feedback needed will shrink dramatically. Of course, it's not possible for folks to sit around and do nothing, but it's good to understand the challenges the current stack might be up against in the near future.