Nvidia pitches AI factories as GPUs sit idle - VentureBeat
A new survey reveals that 81% of companies running AI in production use half their GPU capacity or less, yet most plan to adopt new AI infrastructure within 12 months.
According to VentureBeat Intelligence's September VB Pulse survey of 102 qualified respondents, 68 companies were running AI in production and provided GPU utilization figures. Fifty-five of those (81%) reported average utilization of 50% or less, yet 71% of this group still planned to adopt new AI infrastructure within the next year.
These findings come as Nvidia CEO Jensen Huang is actively promoting the concept of AI factories, urging companies to move beyond merely consuming AI to producing it. At his October 21 GTC Berlin keynote, Huang stated, 'To be part of this AI economy, you don't want to just be consuming the AI; you want to be producing the AI.'
Infrastructure decisions without ROI validation
Across all 102 respondents, 63% planned to acquire a new, additional or replacement AI platform within a year. However, only 10% had an infrastructure ROI figure reviewed and signed off by finance. Among planned adopters, an overwhelming 91% lacked such financial validation.
This suggests companies are making infrastructure decisions before properly validating returns on their existing AI investments. Among respondents whose primary platform was in production, overall satisfaction averaged 4.0 out of 5.0, ease of implementation averaged 3.9, and value for money averaged 3.6.
Memory as architectural challenge
The survey identified key infrastructure considerations: 48% prioritized integration with existing cloud and data stacks, 42% focused on performance, and 35% on GPU availability. The architectural challenge lies in identifying where workflows lose time, repeat work, or consume resources without improving outcomes.
'Memory' presents three connected design decisions: institutional knowledge (organization's durable information), agent task memory (state needed to continue assignments), and inference memory (resources used during model serving). Teams must determine what an agent retains, retrieves, shares, and when results become part of the organization's permanent record.
Economic evaluation needed
The survey found that 23% selected cost per million tokens as a buying factor, up from 11% in August—a statistically significant increase. However, the final cost of a task also depends on token consumption, retries, tool use, and human intervention.
For agentic applications, I would make cost per successfully completed task a central operating measure. That calculation should include inference, retrieval, storage, repeated attempts, and human review.
Seniority results reveal different priorities: 52% of vice presidents and directors selected total cost of ownership as a buying factor, compared to just 4% of individual contributors, who most frequently prioritized integration (58%). This disconnect suggests leaders and builders need a shared economic metric to connect technical decisions with business outcomes.
Related
