Over the past few months, Parity engineers have been testing open-weight AI models and self-managed inference to understand how much of our AI engineering stack we should control ourselves. Here's what we learned from the first month.

What we did, in short

• A 20%-time Parity engineering initiative exploring open-weight models and self-hosted inference.
• A trial on DeepSeek V4 Flash meant to run for one afternoon. It ran for three days, then kept going: 25 engineers, ~12.9 billion tokens, ~$1200 in GPU compute over a month.
• Seven of the eight engineers who responded to our survey said the model could serve as their primary model, although most felt it was only just good enough. 
• The aim is not to replace hosted providers but to make sure we can choose.

AI coding assistants became part of everyday engineering at Parity faster than any of us expected. Codex, Cursor, Claude, OpenCode, Pi, and others are now used across much of the engineering organization. As adoption increased, we started asking a more fundamental infrastructure question: if these systems are becoming part of how we build software, how much of that capability should depend entirely on external providers?

Over the past few months, a small group of Parity engineers has been exploring an alternative. Parity AI is a 20%-time engineering initiative exploring open-weight models, self-hosted inference, and the infrastructure needed to make different models available through the tools engineers already use.

The aim isn't to replace every hosted AI service, but to understand where owning more of the stack gives us useful control over cost, availability, model choice, data, and future capabilities.

Our first experiment gave us some useful answers. We made a cheap open-weight model available; engineers kept using it after the original trial was supposed to end. What began as an afternoon test became a month of real-world use. 

Making AI models replaceable without changing developer tools

As generative AI models continue to change quickly, engineers shouldn't have to change how they work every time we want to try a different model.  Our current architecture is therefore deliberately simple. LiteLLM provides a common interface between developer tools and the models behind them. vLLM serves the models. GPU capacity is currently rented from RunPod.

Technically, this is self-managed inference on third-party GPUs rather than in-house hardware. Engineers' code and prompts still leave our infrastructure, but we now control retention and logging behavior rather than a vendor.

The current Parity AI architecture routes requests through a common interface, allowing different models and inference providers to sit behind the same developer tools.

Engineers can continue using familiar clients and coding environments while the model serving the request changes behind a common endpoint. Instead of coupling a developer workflow to one AI provider, we can test both open-weight and hosted models, compare them against our own requirements, and progressively move workloads between external and internally controlled infrastructure.

The model becomes another part of the engineering stack we can evaluate and replace. This was the original purpose of the exercise.

From an afternoon trial to nearly 13 billion tokens

In mid-August, we ran a short trial using DeepSeek V4 Flash, an open-weight model. It was supposed to run for an afternoon. Usage kept growing, so we left it running for three days. Then engineers kept using it. Between 16 August and 12 September, the system handled more than 144,000 requests and almost 12.9 billion tokens from 25 engineers.

The first three days account for ~3.2 billion of those tokens and roughly $300 in GPU compute, about $0.10 per million tokens. That figure covers GPU compute only. It excludes engineering time, which is easily the higher cost: this was a 20%-time project running for a couple of weeks across a handful of engineers.

The bet is that the infrastructure, benchmark suite, and operational knowledge pay back over the next N workloads, and that if we never build them, we never find out.

What the AI inference data showed

We also learned something about the workloads our engineers actually generate. The distribution of context lengths is the useful shape here:

• 99% of requests needed less than 500k of context
• 55% of total time was spent idle, 26% of time was spent serving 1 request, 9% of time was spent serving 2 parallel requests, 4% of time was spent serving 3 parallel requests
• 1 parallel request gets 345 tokens/sec on average
• 2 parallel requests get 251 tokens/sec on average
• 3 parallel requests get 209 tokens/sec on average
• 4 parallel requests get 152 tokens/sec on average
• at its peak, prefill went as fast as 168k tokens/sec
• mean time to first token: 3.34s
• mean queue time: 2.25s
• mean per-request decode rate: 182 tokens/sec
• prompt tokens: 770M tokens
• completion tokens: 5.55M tokens
• peak GPU KV cache usage: 9.4%
• median generation tokens (how many tokens were generated for a request): 349 tokens
• p99 time to first token: 70 seconds (those hiccups some people were reporting)

Context length is not, on its own, an infrastructure constraint: KV-cache memory, concurrency, and throughput matter more. But it is a useful planning signal. In practice, our engineers are not pushing against the advertised context window of current models.

Usage alone doesn't tell you whether a model is good enough for everyday work, so we also asked the engineers who used it. Twenty-five people used the system and eight responded to our survey. Seven of those eight considered the model capable of serving as their primary model, though most felt it was only just good enough.

What the numbers do show, independent of the survey, is that the model was useful enough for engineers to keep using without prompting.

Why Parity is taking a hybrid approach to AI infrastructure

Hosted and self-hosted AI are not opposing choices. Some models make sense to consume through external APIs, and others are better run directly. The right approach depends on the workload, so we're building a hybrid architecture.

A common inference layer lets us decide where each workload should run without rebuilding the tools around it. Over time, we want to make more of those decisions ourselves.

We're exploring dedicated inference hardware that would allow models to run directly on infrastructure controlled by Parity. We're also developing benchmarks based on our own engineering requirements rather than relying entirely on generic model leaderboards. An engineering organization has very different requirements from a general AI benchmark. Codebases, languages, long contexts, agentic workflows, and internal engineering conventions influence whether a model is really useful.

Why spare GPU capacity may matter more than cost

Cost was one result of the experiment, but capacity may matter more.

During the trial, the GPUs were idle roughly 55% of the time. Thatis one argument against dedicated hardware and one argument for it, depending on what you do with the slack. Better request packing and continuous batching would push effective cost per token lower than the $0.10 headline. But unused capacity also suggests another possibility: instead of treating inference as a scarce resource consumed one prompt at a time, spare compute can run continuous or opportunistic workloads.

That could include assistants with deeper knowledge of our codebases and conventions, organization-specific retrieval, and automated analysis of changes as they land. Workloads that are hard to justify when every token is billed as a separate external API call, and easy to justify when the hardware is already running.

AI infrastructure is becoming an  engineering decision

As coding assistants become part of everyday engineering infrastructure, their architecture matters. Which models we use, where they run, how easily we can replace them, what information they can access, and what becomes possible when an organization runs its own inference are engineering decisions.

We don't have final answers. After a month and thirteen billion tokens, however, we know a better way to investigate them: build, measure, and be specific about what worked and what didn't.

What Parity is testing next

We're testing more models, building benchmarks around our own workloads, expanding access internally, and investigating dedicated hardware. As those benchmarks mature, we intend to publish the methodology and results rather than keep them in-house.

The more fundamental question is what happens to an engineering organization when generating code is nearly free. We think the honest answer is that nobody knows yet. But for companies like Parity, the answer cannot simply be "more code." We write consensus, cryptographic, and blockchain infrastructure where a bug is not a ticket but an incident. In this environment, output is not the deliverable. Correctness is.

Unlimited generation does not change what we build, but it changes where the pressure lands: on review, on testing, on CI, on our ability to say with confidence why a change is safe. If anything, the verification problem scales faster than the generation problem because every line produced is a line someone has to defend. For Parity, the challenge is closing that loop end to end, quickly but safely. That requires owning enough of the stack to test models against our own definition of correctness, while also knowing when not to use them at all.