Josep Prat (@jlprat) header
Josep Prat (@jlprat) profile picture

Open Source Engineer Director at @Aiven_io working in #opensource, formerly at @springernature and @Gameduell.

Berlin
Joined October 2011
Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)Photo by Josep Prat (@jlprat)
Josep Prat retweeted
It’s time for a paradigm shift. We’re introducing Nevia, the cloud for the agentic era. Infrastructure that starts the second an agent asks, branches to test a change and costs nothing while idle. In early access today and live at Slush this week: nevia.cloud/?utm_source=twit…
2
4
5
268
Josep Prat retweeted
A Qwen3.8-27B model compressed from 55.6GB to 5.9GB, with 98.2% of its aggregate benchmark performance reportedly preserved. That’s the result PrismML, the team behind Bonsai, is reporting with its second-generation compression system, Bonsai 2. The numbers are worth looking at: Original Qwen3.8-27B: 55.6GB Bonsai 2 version: 5.9GB Size reduction: Approximately 89.4% Aggregate performance retained: 98.2% Previous Bonsai 27B version: Approximately 95% retention Evaluation coverage: 20 benchmarks The interesting part isn’t simply that the model is smaller. It’s how much of its measured capability reportedly survives the compression. How does Bonsai 2 achieve this? The underlying approach is ternary quantisation. Conventional models store weights using numerical formats such as FP16, BF16, INT8 or 4-bit representations. These formats allow weights to take values from a much larger numerical range. Ternary quantisation takes a more aggressive approach. Instead of storing each weight using a wide range of values, it represents weights using just three possible states: -1 0 +1 Scaling factors are then used to recover useful numerical ranges from these compact representations. This dramatically reduces the amount of information that needs to be stored for each weight. The difficulty is that model weights contain the numerical information behind a model’s behaviour. Compressing them too aggressively can remove distinctions that are important for reasoning, coding and instruction following. The challenge is to reduce storage requirements without destroying too much of the information the model needs to perform its tasks. What improved in Bonsai 2? PrismML reports that Bonsai 2 raises aggregate performance retention from approximately 95% in its previous 27B compression to 98.2%. The team also highlights improvements in three areas: Coding agents Multimodal reasoning Long-horizon tool use The agentic capabilities are particularly interesting. A coding agent working on a real repository needs to do more than generate code from a prompt. It has to inspect files, understand dependencies, modify multiple components, run tests, interpret errors and make corrections. These tasks require sustained reasoning across multiple steps. Preserving that behaviour after reducing a model’s reported size to 5.9GB could make compressed models considerably more useful for local AI development. What this could mean for local inference Reducing the model from 55.6GB to 5.9GB makes it much easier to distribute, store and experiment with. It could also bring a 27B model within reach of more developers using consumer hardware and systems with limited memory. However, the compressed checkpoint size is not the same as total inference memory. Running the model still requires additional memory for the KV cache, activations, temporary buffers and runtime overhead. Longer contexts and multiple concurrent requests can increase those requirements. There is also an important qualification around the 98.2% figure. That’s aggregate performance across 20 benchmarks, not a guarantee that every individual capability retains exactly 98.2% of its original performance. Independent testing will be useful for identifying where the remaining performance differences appear, especially in complex reasoning, repository-level coding and long-running agent workflows. Moving from roughly 95% retention in the previous Bonsai version to 98.2% with Bonsai 2 is a notable reported improvement. The question now is how consistently that retained performance holds up across real workloads, not just the aggregate benchmark score.
The RTX 3060 12GB is reportedly running Qwen3.8-35B-A3B at around 50 tokens/s, with prompt processing reaching approximately 550 tokens/s. And the machine has just 16GB of system RAM. No RTX 3090. No multi-GPU setup. No high-end inference server. Just a consumer GPU running a 35B-parameter model through llama.cpp The interesting part is how the setup works despite having far less GPU memory than the model would normally require The reported configuration Model: Qwen3.8-35B-A3B GPU: NVIDIA RTX 3060 (12GB VRAM) System RAM: 16GB Quantisation: Q4_K_M Inference engine: llama.cpp Generation speed: ~50 tok/s Prompt processing: ~550 tok/s Configured context: 262K tokens Attention: Flash Attention KV cache: Q8 CPU offloading: 26 MoE layers One distinction matters: 550 tok/s is prefill speed, while 50 tok/s is decode speed Prefill measures how quickly the model processes incoming text. Decode measures how quickly it generates new tokens. They describe different stages of inference How does a 35B model run on 12GB of VRAM? Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) architecture It has approximately 35 billion total parameters but activates around 3 billion parameters per token Instead of using every expert for every token, the model’s routing mechanism selects the experts needed for each computation.This reduces the computational workload during generation However, 3B active parameters does not mean the model only needs memory for 3B parameters The full collection of weights must remain accessible because different experts can be selected as generation progresses The solution here is to distribute the workload across GPU memory and system RAM The CPU and GPU share the workload The RTX 3060 provides 12GB of VRAM, while the system contributes another 16GB of RAM Using llama.cpp, the developer offloads 26 MoE layers to the CPU rather than forcing the entire model into GPU memory This allows the system to run a model that would otherwise exceed the GPU’s available VRAM There is a trade-off, though. Moving data between system RAM and GPU memory requires PCIe transfers. Too much CPU offloading can introduce latency and slow generation The goal is to keep the right components on the GPU while using system RAM to accommodate the rest Three optimisations make the setup possible 1. Q4_K_M quantisation The model’s weights are compressed into a four-bit GGUF format, significantly reducing their storage requirements This makes a 35B model more accessible on consumer hardware, although lower precision can affect output quality 2. Flash Attention Long-context inference creates additional memory pressure Flash Attention improves memory efficiency by reducing unnecessary memory transfers and avoiding the need to materialise large intermediate attention matrices in full 3. Q8 KV cache The KV cache stores information from previously processed tokens so the model can reuse it during generation Its memory requirements increase with context length. Using an 8-bit cache reduces the space required compared with higher-precision formats, helping the system accommodate longer conversations and documents Each optimisation addresses a different part of the memory problem The 262K context claim is worth examining The developer also reports configuring the model for up to 262K tokens That’s a substantial context window for a machine with 12GB VRAM and 16GB RAM However, a configured maximum doesn’t guarantee that the model will maintain 50 tok/s throughout the entire context As the context grows, the KV cache consumes more memory. The system must also accommodate model weights, activations, inference buffers and runtime overhead Actual performance will depend on how much context is being used and how the workload is distributed between the CPU and GPU What I want to see next The planned comparisons against Ornith-1.5-35B, Qwen3.8-27B and Qwen3.6-35B-A3B could make this experiment even more interesting
4
4
28
1,708
Josep Prat retweeted
Qwen3.8-27B has reportedly gone from 55.6GB to just 5.9GB, while retaining 98.2% of its aggregate benchmark performance. That’s an 89.4% reduction in checkpoint size. The result comes from PrismML’s Bonsai 2, the second generation of its model compression system. The team evaluated the compressed model across 20 benchmarks and reports a performance-retention improvement over the original Bonsai release. Here are the numbers: Original checkpoint: 55.6GB Bonsai 2 checkpoint: 5.9GB Storage reduction: 89.4% Aggregate performance retained: 98.2% Evaluation coverage: 20 benchmarks Previous Bonsai retention: approximately 95% Moving from roughly 95% to 98.2% retention while keeping the compressed checkpoint this small is particularly interesting. How Bonsai 2 achieves the compression Bonsai 2 uses ternary quantisation, which represents model weights using just three possible values: -1 0 +1 Conventional model weights can use formats such as FP16, INT8 or 4-bit integers, which represent a wider range of numerical values. Ternary quantisation takes a more aggressive approach. It reduces weights to three possible values and uses scaling factors to recover useful numerical ranges. The result is a much smaller representation of the model’s weights. However, reducing the number of bits is only half the challenge. Model weights contain numerical information that influences how a model reasons, writes code, follows instructions and generates responses. Aggressive compression can remove information that matters to these capabilities. Bonsai 2’s challenge is to reduce the storage footprint without significantly damaging the behaviour developers depend on. What improved over Bonsai 1? PrismML reports that aggregate performance retention for its 27B compression improved from approximately 95% with the first Bonsai generation to 98.2% with Bonsai 2. The team also highlights improvements across three areas: Coding agents This includes code generation, repository understanding and iterative development. These tasks require a model to maintain context and make consistent decisions across multiple steps. Multimodal reasoning The ability to interpret and reason across different types of input is another area the team highlights. Long-horizon tool use This is particularly relevant to autonomous agents. An agent may need to inspect a repository, modify several files, execute tests, interpret errors and revise its implementation. A small reliability loss at each stage can accumulate and affect the final result. Preserving performance across extended workflows is therefore worth examining separately from ordinary question-and-answer benchmarks. What does a 5.9GB checkpoint mean for local AI? The smaller checkpoint makes downloading, storing and distributing the model considerably easier. It could also make local experimentation possible on hardware that cannot accommodate the original 55.6GB checkpoint. But there’s an important distinction: 5.9GB is the checkpoint size, not the total memory required for inference. Running the model still requires additional memory for: KV cache: Stores information from previously processed tokens and grows with context length. Activations: Holds intermediate values generated during computation. Runtime buffers: Memory allocated by the inference engine for internal operations. Actual requirements will depend on the inference engine, context length, batch size and workload. So, a 5.9GB download does not automatically mean the model will run comfortably with 6GB of available VRAM. What needs independent verification? The reported 98.2% retention is an aggregate across 20 benchmarks. It does not mean every individual capability retains exactly 98.2% of its original performance. Some tasks could experience larger losses than others, even when the overall score remains high. I’d particularly like to see independent comparisons covering:
One RTX 3090 with 24GB VRAM is reportedly delivering 1,035 tokens/s across 64 concurrent Qwen3.8-27B sessions. The GPU is running at a 250W power limit, using HyperQwen, an open-source inference stack built around vLLM. What’s interesting here is the focus on serving multiple users simultaneously rather than maximising the generation speed of a single conversation. The reported setup GPU: NVIDIA RTX 3090, 24GB VRAM Power limit: 250W Model: Qwen3.8-27B Single-session decode: ~127 tok/s Aggregate throughput: ~1,035 tok/s Concurrent sessions: 64 Configured context: Up to 150K tokens Inference stack: HyperQwen + vLLM There are two important numbers to distinguish. The ~127 tok/s figure is the reported generation speed for one session. The 1,035 tok/s figure is the combined output throughput across 64 simultaneous sessions. It does not mean every user receives 1,035 tokens per second. All active requests share the GPU’s compute resources and memory bandwidth. Why serving 64 users is difficult Running one conversation is different from managing dozens of active requests. Each session has its own prompt, generation state and KV cache. Some users send short prompts, while others submit large documents or request lengthy responses. The inference engine must continuously manage these requests without wasting GPU capacity or exhausting available memory. This is where vLLM’s serving architecture becomes important. Two techniques help make this possible 1. Continuous batching Traditional batching groups requests together for processing. However, different users finish generating at different times. With fixed batching, the engine can end up waiting for slower requests while some GPU capacity remains unused. Continuous batching allows new requests to enter the active batch as existing requests finish. Instead of waiting for an entire batch to complete, the engine continually schedules available work. This helps maintain GPU utilisation and improve aggregate throughput. 2. PagedAttention Every active transformer session needs a KV cache to store information from previously processed tokens. As the number of concurrent users and their context lengths increase, KV-cache memory becomes a major constraint. PagedAttention manages this memory in smaller blocks rather than requiring each request to reserve one large, continuous allocation. This makes memory allocation more flexible, reduces fragmentation and helps the engine manage KV caches across multiple active sessions. Together, continuous batching and PagedAttention help vLLM serve concurrent requests more efficiently. What does 1,035 tok/s actually tell us? It measures the total amount of output the server produces across all active sessions. But aggregate throughput alone doesn’t tell us how responsive the experience is for each user. A more complete evaluation should include: Time to first token (TTFT): How long users wait before generation starts. Per-request latency: How long each response takes to complete. Individual throughput: How quickly each session generates tokens. Concurrency scaling: How performance changes as more users connect. Memory utilisation: How much VRAM is consumed by model weights and KV caches. A server can achieve impressive aggregate throughput while individual users experience different speeds depending on prompt length, scheduling and competition for GPU resources. What about the 150K context window? HyperQwen reportedly supports context lengths of up to 150K tokens. However, supporting a 150K context window and serving 64 users simultaneously at that context length are two very different things. Longer contexts require larger KV caches. When dozens of sessions are active, those memory requirements accumulate quickly. Actual concurrency depends on the context length of each request, KV-cache precision and available memory.
10
20
336
24,344
Josep Prat retweeted
Working in the open source community @jlprat joined Aiven for its open source vision and is now Senior Engineering Director. Next up: bringing diskless topics to Apache Kafka® Read his full story: aiven.io/blog/build-the-futu…
1
1
4
188
Josep Prat retweeted
Andrej Karpathy: “A 1B model can be much more effective than most current LLMs. 90% of current models waste a ton of capacity and money on remembering stuff that doesn’t matter.” here is his 5 core thesis about Small LLM’s: 1. the intelligence was never the knowledge. Strip the memorized internet, keep the reasoning. What’s left is the cognitive core, and it’s small. 2. the core might fit in ~1B params. Maybe less. “Plenty of room at the bottom.” We’ve been paying for a library when we needed a brain. 3. size stopped being the bottleneck. What makes an agent work is the tools it can call and how tight the feedback loop is, not parameter count. 4. intelligence lives in the system, not the weights. A raw LLM is a CPU with no OS. Small core + good harness > giant model alone. 5. you can own it. A model on your hardware is one no one can rate-limit, price-hike, or deprecate on you. in 12 months the gap between teams shipping on-device and teams paying per-token frontier bills won’t be capability. TwIL an 8.8B model at 90%+ of frontier isn’t a benchmark flex. It’s the thesis Karpathy’s been repeating all year, made real. study more on this below ↓
Modest intelligence is arriving faster than most people realize. Today, we're open-sourcing TwIL-8B. Just 8.8 billion parameters, yet approaching frontier-model scores on several difficult benchmarks: • 93% of frontier performance on coding • 92% on instruction following • 90% on contest mathematics All from a model small enough to run on your own hardware. But the benchmarks aren't the most important part of this announcement. For years, the AI industry has focused on building the most capable model, regardless of the cost required to run it. I believe that equation is changing. The next era of AI will be defined by the cost of completing work. Not simply the intelligence of the model, but how efficiently that intelligence can accomplish the task. If a model with 8 billion parameters can reliably complete the same work as one with hundreds of billions, why pay for the larger model? As smaller models become more capable, intelligence becomes something we can distribute, specialize, and run across the hardware we already own. That's the future we're building at @thewebAI TwIL-8B is available today. Open weights. And this is just the first of several announcements we're making today. HF Link: huggingface.co/webAI-Officia…
202
658
135
6,999
1,286,747
Josep Prat retweeted
An MIT Media Lab study used EEG to track what happens in the brain when people write with ChatGPT rather than on their own. Fifty four students completed essay tasks while wearing electrode caps. One group used ChatGPT, one used a search engine, and one used no tools. Across sessions, the ChatGPT group showed the weakest neural connectivity in networks linked to memory, attention, and analytical work. They also had more trouble recalling what they had just written and produced more generic prose. The authors describe this as a kind of cognitive debt: the tool does the synthesizing, and the brain stays quieter [1]. The same experiment points to a better order of use. Students who first wrote unaided and only later switched to ChatGPT showed stronger recall and more widespread brain activity than students who started with the model and then had to work alone. In other words, the model helped more when it scaffolded thinking that was already underway than when it replaced that thinking from the start. The practical lesson is not that large language models should be banned. It is that they work best as a second pass, after the writer has already done the hard part of forming and remembering an argument [1]. [Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X. H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task. arXiv. DOI: 10.48550/arXiv.2506.08872]
340
765
118
2,845
286,370
Josep Prat retweeted
Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.
435
1,009
490
9,793
2,695,107
Josep Prat retweeted
Never lose KRaft quorum. Until you do. Lose 2 of 3 voters and the metadata log freezes. AddRaftVoter can't fix it: removing dead voters needs the quorum you lost. The loop: ➡️Demote lost voters → commit a membership change ➡️Commit anything → need quorum ↩️Get quorum → remove lost voters Stock RPCs refuse, correctly. Raft requires a majority for a reason. Recovery: 1️⃣Find the longest __cluster_metadata log (brokers count) 2️⃣Stop controllers 3️⃣Offline-append VotersRecord, demoting every other voter 4️⃣Start that one controller as sole voter 5️⃣Add replacements via normal online changes This restores availability, not integrity. Whatever lived only on the lost disks is gone, so we page a human instead of automating it. Upstream KIP-1347 covers stale voter endpoints, not lost voters. Write-up: aiven.io/blog/kafka-never-lo…
2
3
277
Josep Prat retweeted
alibaba/open-code-review: Battle-tested at Alibaba's scale. Hybrid architecture code review tool: deterministic pipelines + LLM Agent, precise line-level comments, built-in fine-tuned ruleset (NPE, thread-safety, XSS, SQL injection), OpenAI & Anthropic compatible. #devopsish github.com/alibaba/open-code…
15
105
980
85,272
Josep Prat retweeted
New to Apache Kafka® and wondering if it's the right tool for you? Start with Kafka 101: a full walkthrough of core Kafka principles, getting a cluster running and your first messages flowing. Watch the recording now: youtu.be/fGD8sNbEVFU
1
1
142
Josep Prat retweeted
Dima Kan, leading search here at Aiven, came for the startup pace and got it right away. Hear about how he approaches leading his team through change, and about how he made a hands-on impact in his first week. Read the full blogpost: aiven.io/blog/build-the-futu…
1
1
151
Josep Prat retweeted
Oracle has opened governance for MySQL, and AWS is thrilled to be part of it. We are contributing upstream to performance, vector search, and the extensions framework. It's the same approach we've taken with OpenJDK, Valkey, and dozens of other projects: do the work and make the software better for everyone. Learn more: go.aws/4xUys3s #OpenSource #AWS #MySQL
2
19
2
38
16,323
Josep Prat retweeted
We removed an LM's ability to speak German by fine-tuning on only 4 German tokens. As part of a 1-day hackathon with our product Silico, we removed a 67M-parameter language model's ability to predict German text, by tuning only a scalar factor on one subcomponent of the weights. (1/6)
56
128
49
1,424
307,603
Josep Prat retweeted
As a result of a US government directive, we are suspending access to Claude Fable 5 for all users. You can continue to use all other Claude models. Here’s what this means for you: Across Claude products, new sessions will run on your selected default model or Opus 4.8, and existing Fable 5 sessions will end with an error. On the Claude Platform, requests to Fable 5 will also return an error. Please update your integrations to other Claude models. We know this is a disruption to your workflows; we appreciate your patience and support.
The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees. The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance. Access to all other Claude models is not affected. We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible. Read our full statement: anthropic.com/news/fable-myt…
3,516
7,036
4,658
43,733
13,447,819
Josep Prat retweeted
Inkless keeps expanding. After launching on GCP recently, we’re excited to bring Inkless to Azure, making it even easier to get started without BYOC. Spin up Inkless services on Azure in just a few clicks and focus on building. Learn more: aiven.io/inkless?utm_source=… #Aiven #Inkless #MicrosoftAzure #Cloud #DevTools
1
8
7
1,471
Josep Prat retweeted
Following our announcement a couple of weeks a go with Inkless Clusters being on Aiven Cloud. We're happy to share that Inkless is now live on GCP. This means that users can now create new Inkless services on GCP, without requiring to be on BYOC. Learn more here: aiven.io/inkless?utm_source=…
1
2
432
Josep Prat retweeted
KIP-1150: Diskless Topics has been officially approved for Apache Kafka. It enables storage and compute separation by writing messages directly to object storage (S3, GCS, Azure Blob), with broker disks acting as optional caches. A big step toward a more cloud-native Kafka architecture. Read more: aiven.io/blog/kip-1150-accep… #ApacheKafka #StreamingData #CloudNative #DistributedSystems #DataInfrastructure
2
2
402
Josep Prat retweeted
2025 gave us plenty to celebrate: we passed $100M ARR, launched Inkless Cloud on BYOC, introduced the PostgreSQL Developer Tier, and rolled out Free Kafka - with more to come in 2026.
1
1
2
247
Josep Prat retweeted
Benchmarking Diskless Topics after launching Inkless on BYOC: 144 producers + 144 consumers, 1 GiB/s in + 3 GiB/s out across 3 AZs. Spoiler: ~1s latency for ~94% lower cost. Read the full report here: aiven.io/blog/benchmarking-d…
2
5
232