General Compute Secures $100M for Inference

General Compute Secures $100M for Inference MTS, the technology‑focused YouTube channel, released a new episode on Oct. 15, 2024, featuring host Jason Goodison and guest Jason Goodison, co‑founder and chief technology officer of General Compute. The interview, recorded at the company’s Palo Alto headquarters, announced a $100 million Series A funding round led jointly by Blackbird and Abstract and outlined the startup’s plan to launch a cloud‑based inference service built on purpose‑designed chips rather than traditional training hardware. Goodison said the capital will fund the deployment of several 5‑ to 10‑megawatt inference clusters slated for the first quarter. “We are using the money to secure energy assets, chips, capacity, networking and cooling so we can do real‑world hardware deployments,” he said. He added that General Compute has already signed contracts with early customers and expects additional announcements in the coming weeks. The discussion turned to a technical challenge that has slowed the adoption of large language models: the split between “prefill” and “decode” workloads. In the prefill stage, the system processes the input prompt, a task that General Compute says runs well on NVIDIA GPUs. “NVIDIA does a great job on the prefill side,” Goodison noted. “The problem shows up when you get to decode, where you generate one token at a time.” To bridge that gap, General Compute is partnering with Cerebras, a company that specializes in slicing massive models across many wafers to achieve up to 2,000 tokens per second. “Cerebras handles the decode bottleneck,” Goodison explained, “so latency‑sensitive applications get the speed they need.” He emphasized that the combination of NVIDIA for prefill and Cerebras for decode creates a hybrid pipeline that delivers near‑instant responses for AI agents, code‑generation tools and other real‑time services. Goodison argued that the market is moving from a “slow‑AI” phase, where developers tolerate seconds of latency, to a “broad‑band AI” era in which applications must feel instantaneous. “Users won’t accept a loading spinner when a voice assistant is handling a support call,” he said. “If you can charge three to five times more for a fast token, the economics drive rapid adoption.” Industry observers see General Compute’s approach as a possible turning point for open‑source models, which have lagged behind commercial offerings in inference speed. By offering a cloud that purchases, deploys and manages specialized hardware, the startup aims to give developers control over data and cost while delivering the low‑latency performance traditionally reserved for proprietary systems. The interview concluded with Goodison’s outlook that the next six months will see a surge in demand for ultra‑fast inference. “We expect customers to start asking for high‑throughput, low‑latency tokens within a few months, and we’re positioning ourselves to meet that need,” he said. The announcement marks a significant infusion of capital into the niche of inference‑only hardware and underscores a broader industry shift toward accelerating the decode phase of large language models.