DeepSeek launched a free, open-source large language model in late December, claiming it was developed in just two months at a cost of under $6 million.
But I feel like that will just lead to more training with the same (or more) hardware with a more efficient model. Bitcoin mining didn’t slow down only because it got harder. However I don’t know enough about the training process. I assume more efficient use of the hardware would allow for larger models to be trained on the same hardware and training data?
They’ll probably do that, but that’s assuming we aren’t past the point of diminishing returns.
The current LLM’s are pretty basic in how they work, and it could be that with the current training we’re near what they’ll ever be capable of. They’ll of course invest a billion in training a new generation, but if it’s only marginally better than the current one, they won’t keep investing billions into it if it doesn’t really improve the results.
But I feel like that will just lead to more training with the same (or more) hardware with a more efficient model. Bitcoin mining didn’t slow down only because it got harder. However I don’t know enough about the training process. I assume more efficient use of the hardware would allow for larger models to be trained on the same hardware and training data?
They’ll probably do that, but that’s assuming we aren’t past the point of diminishing returns.
The current LLM’s are pretty basic in how they work, and it could be that with the current training we’re near what they’ll ever be capable of. They’ll of course invest a billion in training a new generation, but if it’s only marginally better than the current one, they won’t keep investing billions into it if it doesn’t really improve the results.