French startup Kog says software can speed up AI on GPUs

In short: French startup Kog says it can make common data center GPUs run AI responses faster by using software that better uses the hardware.

What happened

Kog, a startup from France, is working on ways to get more AI speed out of GPUs, the powerful computer chips often used for graphics and AI. The company argues that GPUs are not necessarily “bad” for advanced AI setups, and that better software can unlock more performance.

In May, Kog shared a technical preview showing fast “inference,” which is the step where an AI model produces an answer (like the moment a cashier totals your bill). In its demo, Kog reported about 3,000 tokens per second for a single request, using standard data center GPUs like AMD’s MI300X and Nvidia’s H200.

Kog says it received “200 tangible business leads” after the demo. The company expects early customers to include software engineering teams and other businesses that lose time, and money, when AI tools take too long to respond. Kog also says some partners want faster AI so people can generate apps or games from a text prompt more quickly.

Why it matters

Many companies already own GPUs, and new AI chips can be expensive and hard to get. If Kog’s approach works on larger AI models, it could reduce wait times and costs without forcing businesses to replace their hardware, similar to making an old car drive farther on the same tank by tuning the engine.

Source: TechCrunch AI