Tech with Mak

Someone turned the hardest parts of LLM inference engineering into a course where you have to fix the performance problems yourself. And you can do it on a Mac. The project is called tiny-llm, an open-source course for building a miniature vLLM-style inference engine around Qwen3. It starts with a simple implementation using matrix operations. You build attention, normalization, positional encoding, the Transformer layers, and the loop that generates tokens. You have a working model. Now the interesting engineering begins. During decoding, repeatedly processing the previous tokens wastes computation. So you implement a KV cache and reuse previously computed attention state. Next, you look at model weights. Keeping everything in dense BF16 consumes memory bandwidth. So you retain packed 4-bit weights and implement specialized matrix operations. Then you move into Metal kernels, SIMD matrix operations, fused RMSNorm, RoPE, SwiGLU, and tiled prefill attention. Once the single-request engine works, you tackle a different problem. What happens when multiple requests arrive, each with different prompt lengths and different amounts of cached state? You implement continuous batching, chunked prefill, paged KV storage, and attention that reads directly from noncontiguous cache pages. The four-week curriculum eventually extends into a small coding-agent harness with validated tool calls, checkpoints, context management, and evaluation. Every stage comes with exercises, reference implementations, tests, and measurements to help you understand the behavior of the system you're building. The course uses MLX, C++, and Metal on Apple Silicon. You don't need an NVIDIA GPU, and the introductory path supports Qwen3-0.6B on an 8GB Mac. Some later checkpoints remain under development and review, but there's already a substantial amount of real inference engineering to learn from. The GitHub repo has crossed 4,800 stars. GitHub: github.com/skyzh/tiny-llm

Someone built a web browser that doesn't render webpages. It has passed 37K GitHub stars as a project written in Zig from scratch, rather than forked from Chromium. And in the team's benchmark, it used 16× less memory than headless Chrome. It's called Lightpanda, a browser designed specifically for AI agents, web scraping, and automation. Here's what makes the architecture interesting. A traditional browser has to fetch resources, parse HTML, execute JavaScript, maintain the DOM, calculate layouts, and render graphics. But an AI agent often needs only a subset of that functionality. It needs to navigate pages, inspect elements, execute scripts, click buttons, fill forms, and extract data. Lightpanda implements the browser machinery needed for those tasks without a graphical rendering engine. Under the hood: → Zig for the core browser implementation → V8 for JavaScript execution → html5ever for HTML parsing → DOM and Web APIs for interacting with pages → CDP and WebDriver BiDi for browser automation → Native MCP tools for AI agent integration It can work with existing automation clients, including Playwright, Puppeteer, and Selenium, subject to its supported browser APIs. In its published 100-page benchmark, Lightpanda completed the workload in roughly 5 seconds using 123 MB peak memory. Headless Chrome took 46 seconds and peaked at 2 GB. Those numbers are workload-specific, but the architectural idea is compelling. The project has now reached version 1.0 and passes more than 1.7 million Web Platform Tests subtests. Compatibility still isn't equivalent to Chrome, especially for visual rendering and layout. For automated systems that mostly need to interact with the DOM, this is a fascinating alternative to running a full graphical browser engine. Fully open source. Here's the GitHub Repo: github.com/lightpanda-io/bro…

The same mathematics that describes heat spreading through a metal plate also explains how Gaussian blur works in image processing. The connection has its roots in Joseph Fourier's work on heat conduction in the early 19th century. Fourier studied how heat moves through a solid when different regions have different temperatures. The resulting heat equation describes how a temperature distribution changes over time, gradually smoothing out sharp differences. There is a remarkable connection to computer vision. If we treat the brightness of an image as a quantity that can diffuse across space, the same equation describes how the image becomes progressively smoother. For a continuous, unbounded domain with constant, isotropic diffusivity, solving the heat equation is mathematically equivalent to convolving the original image with a Gaussian kernel. The connection becomes even more interesting in Fourier analysis. Fine details and sharp transitions correspond to higher spatial frequencies. The heat equation attenuates these frequencies faster than lower frequencies, which explains why Gaussian smoothing removes fine texture while broader structures remain visible. There is also a precise mathematical relationship between blur and diffusion time. The variance of the Gaussian kernel increases linearly with time. To double its standard deviation, we need four times the total diffusion time, starting from the same initial image. This mathematics forms the foundation of Gaussian scale-space, an important concept in computer vision. By examining images at different levels of smoothing, algorithms can study structures at different spatial scales. Gaussian pyramids and classical feature-detection methods make use of these ideas. Mathematics originally developed to understand how heat moves through physical materials later became part of the foundation for how computers process and analyze images.

Inference Engineering is becoming one of the most important skill sets in AI, and I think a lot of engineers are still underestimating it. Training gets most of the attention because that is where the model is created. But once a model has to serve real traffic, a different class of problems appears: queueing, batching, KV-cache pressure, GPU memory, scheduling, communication and tail latency. Inference is not one homogeneous workload. Prefill processes the prompt and builds the KV cache. Work across prompt positions can be parallelized efficiently, so prefill usually has higher arithmetic intensity and can use GPU compute. Decode is different. Each sequence advances autoregressively, usually one token per step, while repeatedly reading model weights and accumulated KV state. At practical serving batch sizes, decode can become heavily constrained by memory bandwidth. That split explains many inference optimizations. FlashAttention reduces memory traffic inside attention. PagedAttention manages KV-cache memory in blocks, reducing fragmentation. MQA and GQA reduce the number of key/value heads, cutting KV-cache storage and bandwidth. Continuous batching lets requests enter and leave the active batch as generation progresses. Chunked prefill breaks long prefills into smaller pieces so they can run alongside decode work. Prefix caching reuses KV state for repeated prefixes, saving prefill computation, but not making later decode tokens faster. Memory is another constraint. Model weights are only part of the footprint. For conventional attention, KV-cache size grows with active tokens and depends on layers, KV heads, head dimension and bytes per element. Quantization can reduce weight memory and bandwidth, while KV-cache quantization can reduce cache memory and traffic. Speculative decoding proposes tokens using a cheaper draft mechanism and verifies them with the target model. Speedup depends on draft cost, acceptance rate and verification efficiency. At larger scale, tensor parallelism adds communication, pipeline parallelism adds scheduling and bubbles, and MoE serving adds routing, communication and load-balancing problems. Some systems separate prefill and decode onto different GPU pools, paying the cost of KV transfer between them. This is why tokens per second is not enough. Production systems care about TTFT, ITL/TPOT, throughput, queueing delay, concurrency, KV-cache occupancy, GPU utilization, tail latency and goodput: how much request load still meets the latency objective. A server can show excellent aggregate throughput while users are still waiting too long for the first token. Inference engineering sits between ML systems, distributed systems, GPU architecture, networking, scheduling and performance engineering. The model decides what token should come next. Inference engineering determines how much serving that token costs, how quickly it reaches the user, and how many users the hardware can support at once.

Harness Engineering from Scratch 🚀 Found this excellent 21K+ star GitHub repo that teaches you how to build reliable AI coding agents, from the ground up. An entire open-source curriculum with 14 lectures, 8 hands-on projects, and 8 progressive learning phases. What makes it useful is the approach: instead of just learning concepts, you gradually build the engineering environment around an AI agent, using the same real application throughout the course. What's inside: → Agent instructions and repository structure → Context management and persistent memory → Multi-session continuity and task handoffs → Task planning, scope control, and progress tracking → Testing, verification, and observability → Automated coding loops and maker-checker workflows → Graph orchestration, parallel execution, and rollback Includes practical projects, reusable templates, a harness-creator skill, and technical breakdowns of how Claude Code, Codex, Pi, and DeepSeek build their harnesses. Available in 15 languages. Perfect for software engineers and AI developers who want to move beyond prompting and understand how reliable coding agents are actually built. Completely free and open source. Here's the GitHub Repo: github.com/walkinglabs/learn…

This is how physics should have been taught from day one.

This 12.5K-star GitHub repo is basically an entire LLM engineering course hiding inside one codebase. It starts with raw text and builds a Transformer in plain PyTorch, then keeps going through pretraining, supervised fine-tuning, reward modeling, preference optimization and reinforcement learning. The repository implements: → Tokenization, attention, Transformer blocks and pretraining → SFT with assistant-token loss masking → Bradley-Terry reward modeling from preference pairs → DPO, ORPO and KTO for preference optimization → PPO with a value head and Generalized Advantage Estimation → GRPO with group-relative rewards and GSM8K evaluation These aren't just disconnected implementations. The post-training methods build around the same underlying Transformer, so you can examine how the training objectives, reward calculations and optimization loops change at each stage. There's also a modern architecture implementation covering RoPE, RMSNorm, SwiGLU, GQA, MLA, optional Mixture-of-Experts and KV caching. The repository even includes a 369K-parameter model trained on TinyStories in approximately five minutes on the author's laptop CPU, with a small BPE tokenizer built from scratch. That's a small educational pretraining experiment. Larger models and the full post-training pipeline require considerably more compute. Everything comes with runnable training scripts, tests, evaluation tools, documentation and a Streamlit control panel. Fully open source. Here's the GitHub Repo: github.com/FareedKhan-dev/tr…

Where can I get this money from?

Reverse an audio recording, and its Fourier magnitude spectrum remains exactly the same. For a real-valued recording, reversing the samples changes the phase of the Fourier coefficients, but preserves their magnitudes. This is a fascinating property of Fourier analysis, and it reveals an important problem in signal processing. A magnitude spectrum tells us how strongly different frequencies are represented in a recording. But it cannot uniquely tell us when those components occurred. The complete complex Fourier transform retains enough information to reconstruct the original samples. The magnitude spectrum alone does not. This is where spectrograms become useful. A spectrogram is commonly computed as the squared magnitude of the Short-Time Fourier Transform (STFT), allowing engineers to examine how a signal's frequency content changes over time. Instead of analyzing an entire recording as one sequence, engineers divide the signal into short, usually overlapping windows and compute a Fourier transform for each section. But there is an important mathematical tradeoff. Short windows provide better time localization, making it easier to identify brief events. However, they provide poorer frequency resolution. Longer windows improve the ability to distinguish nearby frequencies, but make it harder to determine precisely when a change occurred. This is the fundamental time-frequency resolution tradeoff. It matters in real engineering applications. In speech processing, spectrograms help reveal consonant transitions and the formant structure of vowels. In mechanical engineering, they help reveal transient vibrations, changing rotational frequencies, and patterns that may be obscured in an average frequency spectrum. In radar and sonar, time-frequency analysis helps engineers study signals whose frequencies change over time. Even increasing the number of FFT points through zero-padding cannot recover the ability to resolve closely spaced frequencies lost by using a short observation window. The mathematics places a real constraint on what we can measure. A spectrogram is therefore more than a picture of frequencies changing over time. The choice of analysis window determines which events and frequency differences we can resolve. And that choice is an engineering decision, not simply a visualization setting.

Claude Code can now preview what a knowledge graph change will make true before it ships. Fabio Rovai has open-sourced Open Ontologies, a Rust-based engine that brings Terraform-style change management, semantic reasoning, and independently checkable proofs to knowledge graphs. The idea is to treat ontologies as production infrastructure rather than static files that AI agents can freely modify. Through MCP, agents such as Claude Code and Cursor can build and query ontologies, validate constraints, propose changes, inspect their consequences, and roll back updates. The repository also includes a pipeline for turning structured data into knowledge graphs. It can ingest CSV, Excel, JSON, Parquet, and SQL query results, map them into RDF, validate them with SHACL, and derive additional relationships through reasoning. But the change-management system is where the engineering becomes particularly interesting. A normal Git diff tells you which statements changed. It cannot tell you every new logical conclusion those statements make possible. The repository demonstrates this with a single RDF domain declaration. The structural blast-radius report shows zero affected triples and a low risk score. Yet OWL reasoning discovers 901 additional consequences. Open Ontologies detects that semantic change before the proposed update is applied. There's also a separate verification mechanism. For supported inferences, the engine produces derivation certificates containing the original premises, the applied rules, and the resulting conclusions. An independent Lean 4 checker verifies those certificates without trusting the Rust reasoning engine. The repository demonstrates this with 259 certified inferences and a deliberately forged conclusion that the checker rejects. The distinction matters. The proof establishes that a conclusion follows from the supplied facts and rules. It doesn't establish that the original facts are true. Open Ontologies also supports drift detection, rollback, historical evidence, and checks that determine whether a retrieved knowledge-graph subset still supports particular claims. The entire engine is open source, MIT licensed, and available as a standalone Rust binary. A pure-Python implementation is also available. For AI systems that maintain structured knowledge over time, the challenge isn't simply retrieving the right information. It's controlling what changes, understanding the consequences, and preserving evidence that another system can verify. Here's the GitHub Repo: github.com/fabio-rovai/open-…

AirLLM is a Python library that lets 70B-parameter language models run on a single 4GB GPU, without requiring quantization, distillation, or pruning. The problem it's solving is access. Large open-source models keep getting released, but running them with all their weights resident in GPU memory requires substantial hardware, which puts many of them out of reach for anyone without high-end or multiple GPUs. AirLLM's approach: during inference, the original model is first decomposed and saved layer-wise. That's how it avoids needing the full model resident in GPU memory at once. For supported MoE models, it goes further and streams only the experts selected by the router rather than loading every expert in a layer, since each token typically uses only a subset. It supports a wide range of popular models. You pass a Hugging Face repo ID through AutoModel, with support for Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM, and newer architectures. Recent releases have extended this considerably. AirLLM now supports Kimi K3 (2.8T parameters), with the developers reporting just 3.72GB of peak VRAM during generation on an RTX 6000 Ada. Qwen3.8 models are also supported, and v4.0 introduced streamed LoRA fine-tuning, with reported GPU memory requirements of roughly 2–6GB for selected Qwen configurations. There's also an optional compression feature using 4-bit or 8-bit block-wise quantization. The project reports up to 3x faster inference with limited accuracy loss, although the actual results depend on the model and configuration. The trade-off is speed. Streaming model weights from storage reduces GPU memory requirements, but large models can become extremely slow when data transfer dominates execution. It's Apache 2.0 licensed, has approximately 40.7K GitHub stars, and remains actively maintained, with v4.0 released in September 2026. Here's the GitHub Repo: github.com/lyogavin/airllm

A 135K-star GitHub repo is built around one painfully obvious idea: Your coding agent should map the codebase once, not grep it forever. Graphify turns an entire project into a persistent, queryable knowledge graph that AI coding agents can use to understand how different parts of the system connect. Functions, classes, files, SQL schemas, infrastructure configurations, documentation, PDFs, images and videos can become connected nodes instead of isolated pieces of context the agent has to rediscover. So instead of: → Search for authentication → Open twelve files → Follow imports and function calls → Lose the trail as the context fills up The agent can ask: > What connects authentication to the database? > Trace the path from UserService to DatabasePool. > Explain RateLimiter. > Which components are central to this architecture? Graphify returns relevant graph relationships and paths, rather than another collection of keyword matches. For source code, the implementation is particularly interesting. It uses tree-sitter to parse approximately 40 programming languages locally, extracting functions, classes, imports, calls, inheritance and cross-file relationships. No embeddings, vector database or LLM calls are needed to construct the code graph. Document and media processing works differently. PDFs, images and other non-code content can go through an optional semantic extraction step using a configured model. Every relationship carries a confidence label: → EXTRACTED: Explicitly present in the source → INFERRED: Derived through analysis or resolution → AMBIGUOUS: Uncertain and flagged for review Graphify also uses community detection to identify related subsystems and highlights highly connected nodes that may be important to the overall architecture. But the part that makes this useful for coding agents is what happens after the graph is built. Graphify integrates with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot and more than 20 other assistants. Depending on the platform, it installs hooks or persistent instructions that encourage the agent to query the existing graph before repeatedly searching and reading source files. The graph can be committed to Git, updated incrementally as files change, and exposed through MCP. There's also support for examining pull requests against the graph, including identifying PRs that touch related graph communities and may need closer review. The local tool produces three useful artifacts: an interactive HTML graph, a Markdown architecture report and a reusable graph.json. Long context windows help coding agents read more code. A persistent knowledge graph gives them a way to navigate what they've already analyzed. And as coding agents spend more time working inside large repositories, avoiding repeated discovery of the same architecture becomes increasingly useful. Here's the GitHub Repo: github.com/Graphify-Labs/gra…

50 AI terms to help you sound smart in meetings you were pretending to understand.

Any finite sequence of numbers can be represented exactly as a combination of complex oscillations, even when the original data looks nothing like a wave. This is the mathematics behind the Discrete Fourier Transform (DFT), one of the fundamental tools of digital signal processing. When engineers record sound, measure vibrations, or collect data from sensors, they obtain a sequence of samples. The DFT provides a different way to represent those same measurements, using complex exponentials at discrete frequencies. The mathematics relies on a remarkable property called orthogonality. For N samples, the DFT uses N orthogonal complex exponentials. Each frequency component has a corresponding complex coefficient, and the complete set of coefficients contains enough information to reconstruct every original sample. When a complex sinusoid aligns exactly with a DFT frequency bin, its contributions reinforce at that bin and cancel at the others. When its frequency falls between bins, the finite observation window causes its contribution to spread across multiple frequencies. This is known as spectral leakage. These properties have enormous practical value in engineering. In audio processing, Fourier analysis helps identify frequency components and design digital filters. In mechanical engineering, it helps detect periodic vibrations in rotating machinery. In wireless communications, OFDM systems use Fourier transforms to generate and recover signals carried by orthogonal subcarriers. The mathematics also simplifies certain computations. Circular convolution in the sample domain becomes pointwise multiplication in the frequency domain. The Fast Fourier Transform (FFT) makes these calculations efficient by exploiting mathematical symmetries, reducing the computational work from the quadratic cost of direct evaluation to roughly N log N for standard FFT algorithms. There is one important distinction. The DFT preserves all the information in the original sequence of samples, but it cannot recover information already lost during sampling. It is an exact change of mathematical representation, not a way to recover frequencies that sampling has made indistinguishable.