Thinking Isn’t AI’s Hardest Problem

The big talk in artificial intelligence (AI) is about the race for faster compute, what you might call faster “thinking.” Lately, some attention has focused on the multiplying power needed to drive it. In truth, the AI world is hosting three races at once:

• faster compute
• faster communication
• faster synchronization

Compute gets the most headlines. Communication—especially the bandwidth connecting memory to processor—has gotten increasing attention, including from investors. Synchronization gets the least, though it may prove to be the hardest and most crucial challenge.

Training modern AI systems requires thousands—sometimes tens of thousands—of processors working simultaneously. Each processor calculates part of the model’s learning. At every step, those results must be shared and aligned across the entire system.

In technical language, the processors must “synchronize their gradients”—the mathematical updates that tell the model how to improve. If even a few processors fall behind, the whole system slows down.

Imagine a racing shell with eight powerful rowers. Each athlete may be extraordinarily strong. But if even one rower falls out of rhythm, the boat slows immediately. Modern AI systems face the same problem. Thousands of processors may be individually powerful, but they must move in lockstep. If some finish their calculations early while others lag, the entire system waits. Massive computing power sits idle waiting for the slowest participant.

Firms across the industry are attacking the synchronization problem in different ways.

• Speed communication between processors. When Nvidia (NASDAQ: NVDA) bought Mellanox Technologies, the goal was not merely networking hardware but a faster “data fabric” that allows thousands of GPUs to exchange updates quickly enough to stay in rhythm. Nvidia’s CEO Jensen Huang says AI breakthroughs at his company would be impossible without Mellanox.
• Change the architecture so synchronization becomes easier. Cerebras Systems builds enormous wafer-scale processors that place vast numbers of cores on a single chip. By reducing the number of separate devices that must coordinate, the system reduces synchronization overhead.
• Increase communication bandwidth. Startups such as Ayar Labs and Lightmatter are experimenting with optical interconnects—moving data between processors using light rather than electrical signals.

These companies are pursuing different technologies, but they are all confronting the same problem: keeping thousands of processors moving in lockstep.

The synchronization challenge is not going away. It’s getting bigger because it grows roughly with the square of system size. As clusters expand, the number of interactions that must remain aligned explodes.

The history of computing is a history of collapsing constraints. For AI, first, the bottleneck was computation. Then it became memory. Today, it is coordination.

Investors naturally focus on the most visible parts of technological revolutions—faster chips, larger models, and bigger data centers. But the technologies that ultimately determine winners are often less obvious. In the case of AI, these are architectures that allow vast systems to function as a single machine. In the coming phase of the AI era, companies that enhance those architectures will capture a lion’s share of value.

At the Gilder Technology Report, we spend far less time chasing the obvious stories and far more time studying the often-unremarked technologies that actually confer success.