China is overtaking the West in the open AI race
November 30th marked the first birthday of ChatGPT, but in future may be remembered as the day the Chinese open AI community overtook the West. It saw the release of multiple new large language models (LLMs) reaching the seventy billion weight milestone, all with weights available. What's the significance of this? With language models, generally the more weights they have, the "smarter" they are. It's possible for a smaller model to be smarter if trained on significantly more data than a larger one, but if the larger one was trained on a similar volume of data, it almost always wins. Seventy billion weights is the size of the largest competitive "open" model in the West: Meta's Llama2 70B, and is enough to be competitive with GPT3.5 in benchmarks.
Now, the West has other large competitive models, but they are all proprietary: Google's Bard, Anthropic's Claude, OpenAI's GPT and Phind's latest CodeLlama model. Their weights are not publicly available, meaning you (or your company) can't run the models locally on a beefy computer, instead they're only accessible though an API/website. The West also has some older, large open models like PALM that are not competitive with even much smaller models due to being extremely undertrained (not fed enough data), as at the time they were trained it wasn't well-understood how much data such models need. There is the 180-billion-weight Falcon 180B, which performs respectably and can run on the Mac Pro with 190GB of RAM, but that was developed by the UAE, which is only "Western" in the loosest sense of the word. There are also some promising smaller models, like the French Mistral 7B and Microsoft’s Orca 13B, but neither’s creators have stated an intention to create a 70B model and release its weights.
China in contrast now has at least five open1 models around 70 billion weights in size that perform competitively on benchmarks (and in subjective human evaluation amongst the community). There is DeepSeek 67B, from a Chinese startup "dedicated to making AGI a reality" funded by a large Chinese hedge fund, which allows free commercial use subject to some restrictions against various forms of "unethical use". There is Qwen 72B, the Alibaba Intelligence that Ma Yun once promised the world, boasting a 32k context size. The context size is essentially the "memory" of a LLM, and 32k is almost ten times larger than the 4096 context of Llama2 70B. There is XVerse-65B from Shenzhen Yuanxiang Technology, a company whose website I can’t find, with a 16k context size. There is Aquila2 70B, also with a 16k context, from the Beijing Academy of Artificial Intelligence, a non-profit. And finally, there is Yuan2.0 102B, which claims superior performance to GPT3.5, from a Chinese hardware company. An honourable mention also goes to the upcoming Yi 100+B from 01.ai, an AI startup, which is highly anticipated as their Yi 34B model with 200k context was competitive with 70B models nearly twice its size.
These Chinese models now dominate the open AI leaderboards. While they still fall well behind GPT4, it's quite possible that the competition and cross-pollination of ideas among those open models will lead to one of the teams making a breakthrough that brings them to GPT4 level, especially since they seem to have no shortage of compute (properly training a seventy billion weight model costs millions to tens of millions of USD in hardware time). If this happened, the West would be dethroned from the cutting edge of AI, in spite of deliberate effort (via sanctions) to prevent that. While China does have a reputation for copying more than innovating, it has already demonstrated the ability to take the lead in some sectors, such as the drone industry and video recommendation (TikTok). It also surprised the world with Huawei's development of its own 3nm chip technology, a feat previously assumed to be many years away.
Why does China have so many more large open models than the West? It's presumably not due to an excess of GPUs (necessary for training), which due to sanctions are in very short supply there. Or due to an excess of investment capital; America's capital and VC markets are still larger. One possibility is just that expenditure and investment in the West has been more return-focused: it's not clear how to profit from the development of an open model, while with a proprietary model you can just put it behind an API and charge people to access it. Another possibility is the proliferation of the Effective Altruism/LessWrong ideology, which opposes the release of AI to the public for fear of facilitating the emergence/dominance of a godlike superintelligence. A third possibility is fear of the AI being used for spam and misinformation, or the production of politically incorrect content. Aligning an LLM to Chinese requirements is relatively easier, as it primarily needs to just avoid any negativity about the government/party and its leaders, and stick to a certain interpretation of China's land and sea borders, while Western political orthodoxy encompasses a broader range of things it's unacceptable to say, and models must be trained not to fall victim to certain biases present in the data.
If the relative paucity of large open LLMs in the West is due to the ethical concerns of Western developers, this might be counter-productive in the long run. A surplus of competitive Chinese LLMs makes it more likely that one of them will become the basis of a popular product, exposing Westerners to a non-Western-aligned LLM promoting non-Western values. This was already seen to some degree with TikTok, a Chinese product, recently found to be promoting the development of anti-Zionist views in Western youth. The release of more 70B+ Western LLMs would increase the chance that the first open LLM to achieve widespread usage is Western-aligned.
It's entirely possible that China's increasing dominance in the open model space won't lead to dominance in the overall commercial AI space, or to overtaking OpenAI. But given the consequences if China overtakes the West, complacency is risky. We'll see over the next year whether Chinese AI is indeed able to catch up with GPT4 or stays stuck in second place. Feel free to share your predictions in the comments.
Note the use of the word “open” here, not “open source”. Most large LLMs weights are not under truly open source licenses, as they don’t allow the user to use them however they wish.