Podcast Summary
From crypto mining to AI infrastructure: CoreWeave grew from supplying GPUs for crypto mining to becoming a major player in the AI infrastructure market, recognizing the potential in GPU-oriented mining for Ethereum and expanding into other areas as demand for AI hardware grew.
The rapid adoption of AI technology is outpacing the infrastructure built for it, leading companies to rebuild and expand their physical infrastructure at an unprecedented pace. CoreWeave, a hosting company for AI and GPUs, identified this opportunity early and grew from supplying GPUs to crypto miners to becoming a major player in the market. The founders, who have a background in finance and data, saw an arbitrage opportunity in the crypto space and recognized the potential in GPU-oriented mining for Ethereum. However, they soon realized that this wasn't a large enough market and sought to expand into other areas as the demand for AI hardware continued to grow. The company, founded in 2018, has since grown significantly, with the value of NVIDIA's market cap increasing over 7 times during this period due to the high demand for these chips. CoreWeave's role as a CDO (Chief Development Officer) at the company involves raising capital and interfacing with equity and debt participants to fuel the growth of the business in this capital-intensive industry.
From Crypto to AI: Transitioning Businesses: A company that started in the crypto space pivoted to AI and cloud infrastructure, recognizing the potential and demand shift. The hyperscalers face challenges in adapting their infrastructure to compete.
While Bitcoin ASICs are specialized for Bitcoin mining and have limited use cases, GPUs can be used for various applications including crypto and AI workloads. The speaker's company started in the crypto space but soon realized the potential of using GPUs for AI and cloud infrastructure. The demand for cloud gaming is not substantial, and the crypto market fizzled out as AI began to boom. The speaker's company transitioned into the cloud infrastructure market in 2019 and built a proprietary orchestration solution for running AI workloads, differentiating it from the hyperscalers. The innovator's dilemma applies, as the hyperscalers would face significant challenges in changing their infrastructure to compete. NVIDIA's H100, for instance, is not just a GPU but a platform for many GPUs, costing around $30,000-$40,000 each. The speaker emphasizes the challenges the hyperscalers face in changing their infrastructure to compete in this market.
Scaling up GPU usage in data centers creates scarcity and high costs: Scaling up GPU usage in data centers offers cost savings for businesses, but energy consumption and data center capacity are major challenges
By scaling up the use of GPUs in data centers, companies like 3five zero six and zero zero six:fifty four are able to offer access to powerful computing resources at a lower cost per hour, while also creating a scarcity in the market due to the engineering complexity involved in building large-scale fabric networks. However, this comes with a significant energy consumption, making data center capacity and power availability the next major bottlenecks in the GPU cloud infrastructure market. For instance, a single server with 8 GPUs can cost upwards of $1,000,000, and renting out an individual GPU for $4 an hour results in substantial hourly costs for large-scale jobs. Yet, the ability to use thousands of GPUs at once makes this a cost-effective solution for many businesses, leading to the operation of some of the world's largest supercomputers on their platforms. This scaling approach de-commoditizes the market by making it less about individual GPUs and more about access to large numbers of GPUs, which requires unique engineering solutions. For businesses, OpenPhone offers a simpler solution for managing communications through a single, easy-to-use app, starting at just $13 a month for Twist listeners. However, the energy consumption of these data centers is a significant concern, and the availability of data centers with sufficient power will be a crucial factor in the future growth of the GPU cloud infrastructure market.
GPU's power efficiency unlocks value from data: GPUs are more power-efficient than CPUs for data processing, but their high energy consumption raises concerns about availability of energy sources. Solutions include co-locating data centers with nuclear power plants and utilizing base load power sources.
While GPUs for data processing consume a significant amount of power, they are more power-efficient when it comes to running workloads compared to CPUs. This efficiency, despite the high energy consumption, makes GPUs a transformational technology with the potential to unlock unprecedented value from data. However, the growing demand for these GPUs is raising concerns about the availability of energy sources to power them. Some solutions being explored include co-locating data centers with nuclear power plants and utilizing base load power sources like natural gas and nuclear energy. Additionally, the issue of heat dissipation is being addressed through natural cooling methods in colder regions, reducing the need for air conditioning. Overall, the integration of these advanced technologies requires careful planning and a focus on sustainable energy solutions.
The future of GPU cooling is direct-to-chip liquid cooling: Direct-to-chip liquid cooling increases energy efficiency and reduces the need for large-scale air circulation, improving operational efficiency and reducing downtime in GPU infrastructure.
As the demand for GPU infrastructure continues to grow, especially in the realm of AI and cryptocurrency mining, the need for advanced cooling solutions becomes increasingly important. Currently, the most reliable and efficient cooling solution is through large-scale, forced air cooling in tier 4 or 5 data centers. However, the future lies in direct-to-chip liquid cooling, which will increase energy efficiency and reduce the need for large amounts of air circulation. This method involves running coolant directly to the chips, avoiding the need for draining and refilling large pools of liquid as with immersion cooling. By focusing on direct-to-chip liquid cooling, companies can improve operational efficiency and reduce downtime, making their infrastructure more reliable and cost-effective.
Exponential demand for compute power in AI sector outpacing infrastructure: The shift from model training to inference is driving massive growth in the compute market, with inference being massively compute-intensive and requiring significant physical infrastructure investment to keep up.
The demand for compute power in the AI sector, particularly for inference, is growing exponentially and outpacing the current infrastructure. Companies like 3five zero six are seeing a massive increase in revenue and a fully booked build schedule for their compute capacity, which is being driven by the shift from model training to inference. Inference, which involves processing user queries, is massively compute-intensive and will be the primary driver of growth in this market. The existing cloud infrastructure was not built for this use case and requires significant physical infrastructure investment to keep up with the rapid adoption of AI software. As a founder, focusing on building products and meeting with partners is important, but handling HR and payroll efficiently is crucial. Gusto is a solution designed for small business owners to make these tasks easier, ensuring accurate payroll calculations, tax handling, and access to HR experts.
A diverse range of hardware and software solutions will shape the future of AI technology: From GPUs for model training to specialized hardware for inference, the future of AI technology will involve a mix of hardware and software solutions catering to various model types and objectives.
The future of AI technology will involve a diverse range of hardware and software solutions, rather than relying on a single dominant technology. Gusto, as discussed, simplifies HR tasks for businesses, allowing them to focus on their products and customers. In the AI industry, GPUs will continue to be the go-to solution for training the most demanding and complex models, while specialized hardware like Groq's inference engines will cater to specific model types. Nvidia's continuous improvement of GPUs makes them a strong contender for training the latest generation models. However, the dominance of Nvidia's CUDA software solution creates a significant barrier for competitors, making it challenging for other hardware providers to gain traction. Despite this challenge, there have been attempts to adapt CUDA for use with different infrastructure, but the performance and configurability losses make it an unattractive option for most clients. The potential impact of open-source chips and chip architecture on the space is still uncertain. Overall, the AI landscape will be characterized by a diverse range of hardware and software solutions, each catering to different model types and objectives.
Nvidia's Infrastructure and Software Dominance in Compute Market: Nvidia's performant GPUs and proprietary InfiniBand fabric give them a significant advantage in the large-scale compute market, making it difficult for competitors to gain substantial market share.
Nvidia's infrastructure and software dominance in the compute market, particularly for large consumers, is significant due to their performant Nvidia GPUs and proprietary InfiniBand fabric for data throughput. This infrastructure's integration and the lack of a comparably performant training fabric in AMD's offerings make it challenging for competitors to gain substantial market share. The vast majority of consumers with tens of thousands of GPUs are sticking with Nvidia, making it a formidable player for the foreseeable future. InfiniBand, a high-speed data transfer network solution, is crucial for moving large amounts of data efficiently between GPUs and clusters, and Nvidia's acquisition and integration of this technology into their DGX solution have given them a significant advantage. While there may be room for smaller players or specific use cases, the current demand and scale of consumers indicate a prolonged Nvidia-dominated market.
Building AI infrastructure at scale is a complex challenge: Companies are investing in building AI infrastructure to test new ideas and create organic, scalable products, but the availability of cloud infrastructure limits their ability to do so
Building and managing large-scale infrastructure for AI workloads is a complex physical problem that requires a significant amount of capital and resources. This infrastructure, which can consist of thousands of nodes or servers with multiple GPUs each, needs to be connected through a high-speed fabric with a large number of discrete connections. This is a new challenge that wasn't present when dealing with Ethernet or hosting websites. Companies like Microsoft, Meta, and Google, with their massive amounts of capital, are turning to building infrastructure as an alternative to M&A. This infrastructure can lead to new jobs and commercial products, as well as the ability to test and experiment with new ideas. The most promising areas for AI adoption at scale are within existing products that feel organic and don't require users to learn something new. One example is Copilot solutions, which integrate AI into apps to assist users with pre-existing processes. However, the ability for these products to scale is limited by the availability of cloud infrastructure capable of handling large numbers of users and their corresponding GPU queries.
AI is disrupting advertising with personalized ads: Businesses must invest in AI for personalized ads to stay competitive, while the infrastructure investment provides strategic advantages.
AI is becoming a mandatory expense for businesses, especially in sectors like advertising, rather than an optional add-on or a potential source of incremental revenue. The ability to provide highly personalized and effective ads through generative AI is expected to disrupt the advertising sector significantly. Companies that fail to adopt this technology may lose market share and users to their competitors. Additionally, the infrastructure required to support AI at scale is a strategic advantage, decommoditizing compute and enabling businesses to offer more effective products and services. The discussion also highlighted the example of search engines, where the integration of AI is essential for maintaining market dominance, as seen with Google. However, the infrastructure demands for AI can be a significant financial investment, making it a tax for some businesses. Nevertheless, the potential benefits, such as increased user engagement and market share, often outweigh the costs.
The Future of Advertising: Customized, Mind-Reading Ads: The future of advertising lies in customized, mind-reading ads, driven by AI and massive data infrastructure, with businesses like Amazon, Uber, and Instacart leading the charge. However, the current infrastructure may not be sufficient, and the shift towards generative AI and GPUs is necessary for growth.
The future of advertising is headed towards highly customized, mind-reading ads that will bridge the gap between aspirations and reality. This level of customization comes with immense infrastructure demands, making it a significant business opportunity for companies like Copilot and those in the e-commerce sector. The convergence of data and frequent transactions in businesses like Amazon, Uber, and Instacart make them prime targets for this AI revolution. However, the current infrastructure may not be sufficient to meet the demand, and it could take until the end of this decade for the supply and demand to balance out. The shift towards generative AI and the need for more powerful computing like GPUs are essential components of this transformation. While there are concerns about overbuilding infrastructure, the current trajectory suggests heavy growth for the foreseeable future.
Disruption in Commoditized Cloud Market: The cloud market is shifting from commoditized offerings to more differentiated services, leading to operational complexities and unique offerings.
While different cloud platforms like AWS, GCP, and Azure may seem similar for hosting websites, the lack of fungibility is becoming more apparent due to differences in software and infrastructure. This commoditization is being disrupted, leading to unique offerings and operational complexities. For instance, setting up an h100 at CoreWeave is different from AWS due to their distinct approaches. This process involves hundreds of people and semi trucks delivering equipment to data centers across the US, making it an operational feat. The hiring of high-level system operations people is ongoing to configure this infrastructure. This disruption signifies a shift from a commoditized cloud market to one with more differentiated offerings.
