Ultra-low-latency LPU inference. Routinely the fastest tokens-per-second numbers you can buy.
Groq is an AI inference platform built on custom silicon called an LPU — Language Processing Unit — designed specifically to run large language models fast. Rather than using general-purpose GPUs like every other cloud provider, Groq built hardware that handles the sequential nature of token generation natively, delivering speeds of 500+ tokens per second on 70B-parameter models. For developers building voice agents, real-time chat systems, or multi-step AI pipelines where latency compounds across each call, Groq offers speeds that GPU-based inference consistently cannot match. The hosted platform is called GroqCloud and exposes an OpenAI-compatible API, so switching from another provider requires minimal code changes.
Standard GPU inference is a mismatch for language model generation. GPUs are optimised for parallel matrix operations — perfect for training, but language models generate one token at a time in a sequential loop. Each token depends on all the previous ones, which means the parallelism GPUs excel at is not available for most of the generation process. GPUs compensate with high memory bandwidth and clever batching, but the fundamental constraint remains.
Groq’s LPU architecture addresses this directly. The key difference is in how weights — the parameters that define a model’s behaviour — are stored. On an LPU, weights live in on-chip SRAM (fast memory built directly into the processor) rather than off-chip HBM (the high-bandwidth memory used by GPUs). Fetching weights from on-chip SRAM is orders of magnitude faster than loading them from external memory on every generation step.
Groq’s compiler also pre-schedules every operation down to individual clock cycles before the model runs. There is no dynamic scheduling overhead at inference time — every compute step is predetermined, which eliminates the scheduling delays that accumulate at scale on GPU clusters.
The result: where a GPU-based API typically returns 50–100 tokens per second for a 70B model, Groq routinely delivers 500+ tokens per second, with consistent latency rather than the variability common on shared GPU infrastructure.
The GroqCloud API is OpenAI-compatible, meaning you can point an existing application at Groq’s endpoint by changing a base URL and API key. The supported model names differ, but the request and response formats are identical.
Groq’s primary users are AI engineers and developers building latency-sensitive applications where the speed of inference directly affects user experience or system throughput. Voice agent builders use Groq because real-time conversation requires responses in under two seconds — generating a 200-token answer at Groq speeds takes around 400ms, leaving room for other pipeline stages. Multi-agent system developers use Groq because agents that call LLMs repeatedly in a loop compound latency across every step, so faster inference means meaningfully faster end-to-end task completion. High-volume batch processing teams use the Batch API to run large classification or summarisation jobs at half the standard cost.
Real-time voice agent: A conversational AI that listens, processes, and responds quickly enough to feel like a natural conversation. Groq’s low-latency generation on models like Llama 3.3 70B keeps the response generation step inside the sub-second window needed for fluid voice interaction.
Multi-step research agent: An agent that breaks a question into sub-tasks, fires off LLM calls at each step, synthesises results, and returns a final answer. At Groq speeds, a 10-step reasoning chain that might take 30 seconds on slower inference completes in 3–5 seconds.
Document classification pipeline: A batch job that processes thousands of documents — classifying them, extracting structured data, or flagging anomalies — using the Batch API at 50% of real-time pricing. Results land in your system within 24 hours.
Low-latency coding assistant backend: A developer tool that generates code suggestions fast enough to keep pace with typing, without the variability that comes from shared GPU inference under load.
Prototype with open-source models: An evaluation environment for comparing Llama, Mixtral, Gemma, and DeepSeek outputs on your data. The free tier and low per-token pricing make it cheap to run comparisons at scale.
Groq has a free tier that requires no credit card. It allows 30 requests per minute and is genuinely usable for prototyping and building proofs of concept. Paid usage is billed per token. Most models are under $1 per million tokens: Llama 3.1 8B starts at $0.05 per million input tokens, and the flagship Llama 3.3 70B Versatile runs at $0.59 input / $0.79 output per million tokens. The Batch API cuts these prices in half for non-real-time workloads. There are no monthly subscriptions — you pay for what you use.
Sign up at console.groq.com — no credit card required. Generate an API key and make your first request. The base URL is https://api.groq.com/openai/v1 and the request format matches the OpenAI chat completions spec, so you can drop it into an existing LangChain, LlamaIndex, or OpenAI SDK integration by changing one environment variable. Start with llama-3.3-70b-versatile for general-purpose tasks or llama-3.1-8b-instant for the fastest possible responses. Test your latency against your current inference provider and compare. The free tier is enough to run a meaningful benchmark before committing any budget.
Yes, Groq has a free tier that requires no credit card and allows 30 requests per minute. It is genuinely usable for prototyping and testing. Paid usage is billed per token with most models under $1 per million tokens.
Groq's catalog is entirely open-source models. Supported models include Llama 3.x (including the 70B and 405B variants), Mistral, Mixtral 8x7B, Gemma, DeepSeek R1 Distill Llama, and Qwen. Groq does not host proprietary models like GPT-4o, Claude, or Gemini.
Groq's LPU architecture is purpose-built for sequential token generation and achieves 10–20x lower latency than equivalent GPU inference services. The tradeoff is that Groq only hosts open-source models. If you need a proprietary frontier model, you will use the provider's own API. If you need maximum speed on an open model for a voice agent, real-time chat, or multi-step agent loop, Groq is the faster and often cheaper option.
Replicate is a cloud API for running and deploying thousands of open-source AI models without managing GPUs or infrastructure.
Inference / HostingvLLM is an open-source LLM inference engine from UC Berkeley that maximises GPU throughput using PagedAttention and continuous batching, with an OpenAI-compatible serving API.
Inference / HostingModal is a serverless cloud platform that lets developers run GPU inference, training, and batch jobs directly from Python code.
Inference / HostingA cloud platform offering fast inference, fine-tuning, and GPU compute for over 200 open-source AI models, used by developers building production AI apps.
Inference / HostingUpdates from the AI world — what shipped, what we’re using in production, and what’s worth your attention. Two emails a month, no spam.
