Google Takes Aim at Nvidia With New AI Chips Built for Privacy and Massive Memory

The TPU 8tx and TPU 8i Max join Google’s eighth-generation Tensor Processing Unit lineup, giving businesses four configurations for training and running AI models. Google says it serves nearly 100 TPU customers, from AI labs to universities.

Unlike Nvidia’s widely used graphics processing units, TPUs are purpose-built AI accelerators. Google developed them for its own services but increasingly sells access to outside companies, including on-premises customers.

The TPU 8tx targets a problem that many serious businesses have when it comes to adopting AI: protecting sensitive information while a model is using it. Data is encrypted in storage and during transmission, but processing typically requires decrypting it in memory.

Google says the configuration uses hardware-level confidential computing to isolate customer data and proprietary model parameters even during operation. The system combines TPU 8t silicon with fifth-generation AMD EPYC processors and Google’s low-latency Boardfly network.

Google claims it can cut latency by up to half versus TPU 8t, with almost no performance penalty for the added protection. In other words, tasks that require privacy won’t sacrifice speed or accuracy.

Companies could fine-tune and run models on the same system instead of transferring sensitive information between clusters. However, confidential computing cannot prevent a poorly designed AI agent from mishandling data it is allowed to access.

Breaking AI’s Memory Bottleneck

TPU 8i Max addresses another limitation: memory. Autonomous agents analyzing data, codebases, documents or hours of video, etc., need fast access to far more information than conventional AI tasks.

Google combines two TPU 8i chips into one package, doubling computing power, on-chip SRAM and high-bandwidth memory throughput versus TPU 8i. The design packs twice as much silicon into the same rack footprint.

Google is open-sourcing TPU Sync, software that moves AI working memory between chips, servers and storage. It claims the system can reduce certain data-transfer delays and response startup times by more than 90%.