Google announces two new TPU chips for training and running AI

In short: Google has unveiled two new Tensor Processing Unit chips, one built to train AI models and one built to run them.

What happened

Google announced its eighth generation of TPUs, which are the company’s custom chips for AI work. Unlike earlier generations, this release is split into two separate chips with different jobs.

The TPU 8t is designed for training, which is the long, expensive phase where an AI model learns from large amounts of data. Google says TPU 8t is meant to cut training time for very large models from months to weeks. Google’s updated TPU 8t “pods” (big groups of chips that work together like a single machine) can include 9,600 chips and share 2 petabytes of memory.

The TPU 8i is designed for inference, which is what happens when a trained model answers questions or generates text, step by step. (You can think of training as teaching a student, and inference as the student taking a test.) Google says inference does not need the same level of raw power, so TPU 8i focuses more on efficiency and handling many AI “agents” (software helpers that can carry out tasks). TPU 8i pods can include 1,152 chips, and the chip includes more fast on-chip memory to help with longer prompts and longer conversations.

Why it matters

This is part of a wider push to lower the cost of building and using AI, especially the electricity and cooling needed in data centers. Google claims these new TPUs deliver more performance per watt than the prior generation, although that does not automatically mean data centers will use less total power.

Source: Arstechnica