Amid a shortage of AI chips, the GPU rental market is booming

GPUs, or graphics processing units, have become increasingly difficult to acquire as tech giants like OpenAI and Meta buy mountains of them to power AI models. Amid an ongoing chip shortage, a host of startups are stepping up to increase access to highly sought-after AI chips by renting them out.

The GPU rental market is part of a niche, existing industry known as GPU-as-a-Service, where chip owners use an online marketplace to sell computing power to customers for set periods of time through the cloud Businesses typically turn to major cloud providers such as Amazon Web Services, Microsoft Azure, and Google Cloud, which collectively have a 63 percent market share of the global cloud computing market:to run AI workloads in their on-premises data centers.

However, GPU as a service offers a more decentralized approach. Vendors in this space partner with data centers and GPU owners globally to lease their chipsets to customers on an as-needed basis. Renting compute power allows organizations with tight budgets, such as startups and academic institutions, to access high-performance GPUs for specific projects, said David Bader, director of the Data Science Institute at the New Jersey Institute of Technology.

“GPU as a service has significantly leveled the playing field in AI and high-performance computing,” Bader told the Observer. “Instead of making substantial upfront investments in hardware that quickly depreciates and becomes obsolete, businesses can now access GPU power on demand.”

While supply chain constraints around GPUs are beginning to ease, the rental market continues to grow. The GPU-as-a-Service market, valued at $3.79 billion as of 2023, is expected to grow 21.5 percent annually to $12.26 billion by 2030 as demand for analytics increases advanced data, such as running machine learning algorithms, according to data from Search Grand View.

Generative AI has sparked interest in GPU rental

Some startups in the GPU rental space they have seen demand increase ever since ChatGPT came out in November 2022 as companies seek computing power to build AI

Jake Cannell, founder and CEO of Vast.ai, said his company’s customers were primarily cryptocurrency miners before the generative AI hype began. today, more than half of the projects running on Vast.ai’s GPU rentals are AI-related. Customers include AI entrepreneurs, startups, and academics who build large custom language models using foundational models like OpenAI’s GPT. and deploying LLM in AI-related workloads like the Stable Diffusion AI imager, Cannell said.

The launch of ChatGPT, combined with high demand from major cloud providers and GPU shortages, pushed more customers to look for alternative options, which has partly accelerated demand for Vast.ai’s GPU rentals, according to cinnamon “It’s probably relaxed a bit now that production has caught up, but demand still looks very high and growing,” the CEO said.

CEO of Nvidia (NVDA). Jensen Huang recently said Demand for Nvidia’s new Blackwell chips has been “insane” and that the company, which owns about 90 percent of the GPU market, plans to ramp up Blackwell production this year through 2026.

Launched in 2017, Vast.ai is behind an online marketplace that connects owners of Nvidia and AMD GPU clusters with organizations looking to rent computing power. As of the end of October, the market offers 109 GPU clusters, including Nvidia’s popular H100 chips, housed in data centers and, in some cases, owners’ garages across Europe, Asia and Australia in the US, according to Cannell.

By offering GPU clusters with different capacities, speeds and system requirements for different periods of time, Vast.ai aims to give tenants the freedom to choose the GPUs they need for specific projects and scale up and down based on their needs. For example, a customer developing an AI chatbot may initially rent 100 GPUs to train their model. If their product takes off, the customer could increase their computing capacity by renting thousands of GPUs. The flexibility to access different amounts of computation at different stages of product development, the company says, is what makes renting GPUs attractive to buying chips.

“The purchase would only make sense if you have a much more predictable and consistent demand for GPUs over a very long period of time,” Cannell said. “Only the hotheads can formulate it,” referring to industry giants like OpenAI.

While startups like Vast.ai that launched before ChatGPT are seeing a surge in interest, new startups have sprung up since the chatbot’s launch to take advantage of the growing GPU rental market.

Foundry, a marketplace for GPUs designed specifically for AI workloads, says it has attracted “dozens” of customers since launching its cloud platform in August and can significantly reduce computing costs by leveraging excess power from pre-existing chips, according to CEO Jared Quincy Davis.

The startup, which raised $80 million from investors including Sequoia and Lightspeed Ventures as of March, leases GPUs using a combination of computing clusters the company owns and “underutilized clusters” from data center partnerships.

Foundry’s clients include companies in the technology, telecommunications, media and healthcare industries. Foundations and academic labs also use Foundry services. Common use cases include fine-tuning models like Meta’s Llama to display desirable properties, building neural networks from scratch, and performing sentiment analysis, a deep learning technique used to analyze text to determine its emotional tone Foundry even has customers renting out GPUs to predict protein sequences for drug discovery, train models to translate rare languages, and build AI agents that can control websites without human intervention.

“Much of the cutting-edge development that was previously only possible for labs like OpenAI and DeepMind will now be available to others as Foundry makes GPU computing more accessible and affordable,” said Davis, who previously worked at Google DeepMind as an engineer. Observer

Some organizations are seeing the benefits of renting GPUs materialize. Bader, the New Jersey Institute of Technology professor, said he has seen his university use the GPU rental approach to “free up resources” for “critical activities” such as research and development. The GPU rental model, it claims, is ideal for projects with “temporary” or “temporary” computational needs and “removes the burden” of expensive hardware management and maintenance. Bader said he has, too looking at small businesses, the university collaborates with access to the same GPU power as larger businesses.

“I’ve witnessed countless startups benefit from this,” Bader said. “They no longer need millions in upfront investment for specialized hardware. Instead, they can prototype, test, and iterate their algorithms using leased GPUs, ensuring that funds go toward development rather than infrastructure.”

Renting GPUs may not save you that much money in the long run

Still, Bader noted that renting GPUs to buy them comes with some advantages.

Performance on the shared infrastructure can be inconsistent, which could slow down the execution of tasks such as AI model training if there are service outages. Renting the GPU could also be expensive despite the initial cost savings. According to Bader, the costs of transferring data between the cloud and the enterprise could “add up quickly,” and for workloads that require real-time processing, customers who continually experience latency issues may end up paying more than if were GPU owners. Lack of control over infrastructure could also be “problematic” for companies with strict security and compliance protocols.

The future of the GPU rental market could also depend on how the chip industry evolves. After all, major cloud providers such as Amazon Web Services are expected to continue to expand their product lines and are likely to absorb smaller companies, which could lower prices in the short term and limit consumer choice in the long term, according to Bader. Also, supply chain delays could make it difficult for cloud giants to get their hands on GPUs.

Despite these concerns, the startups that spoke to the Observer remain confident that their services will still be needed in the years ahead as AI continues to grow. Vast.ai continues to improve its GPU matchmaker service and is getting more directly involved in use cases like LLM inference, especially for AI agents. Foundry plans to release additional features to increase the accessibility of its platform and make it more useful for AI developers to build advanced models.

“Nvidia is still a leader, and I don’t see that changing overnight, but there is more and more competition,” said Cannnell, CEO of Vast.ai.