Gregg Furstenwerth (@Stealthwater) header
Gregg Furstenwerth (@Stealthwater) profile picture

Married to @Shayfurst, Autistic Madman, Florida Keys, Underwater Wonderland, Rebreather Diver, Cinematography, Photography, Developer, Elixir, C, Python, Go

Little Torch Key FL
Joined May 2012
Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)Photo by Gregg Furstenwerth (@Stealthwater)
KV cache silently completes while your conductor waits forever. Q reports done. K and V hand off. KV cache just… disappears. One missing events_mark_finished and your entire lockstep pipeline jams. Always instrument your DAG, not your intermediates. #AI #ConcurrentTransformer
59
Just debugged a fun one shared cublas handle across concurrent substreams causes silent corruption on layer N+1 even when streams are long-running. Each substream needs its own handle created and stream-bound at init. cublasSetStream after init is a race condition. #ai
2
98
RoPE and GQA done for modern models. Scalable inference engine is coming a long nicely. 100% custom kernels and memory management. So full MHA and GQA support. Runs GPT2 and Llama. The next step is to make sparse attention work with a switch.
1
1
134
Being able to enable and disable sparse attention based on resources to sequence length and desired results will be cool.
1
87
Llama implementation of QKV seems to be working well in FP32 even though it’s a FP16 model. Writing metal shaders that handle QKV for fused and non fused weights is tricky. Automatic detection and handling is now in place. Just jam in a model and it goes. Automatically parsing configuration and weights. Once the pipeline is aligned though logits, FP16 will be finalized. This requires some core changes to make it functional. That will lay the foundation to allow int8 and int4. Platform detection of Nvidia and Metal is in place. Model detection and parsing. Config detection and parsing. Finalizing the llama pipeline and then integrating the fp16, int8 and int4 will be the next big pushes. Building a custom transformer is no easy task.
102
RMS Norm kernel hacking now. Embeddings works great for llama. Config and model management is solid. The full pipeline adapted with small changes. Full integration of llama models underway. Metal shaders, let’s do this. 6000 lines of metal 🤘 So much duplication and nasty code lol. Gotta make it work then beautify it. #MachineLearning
1
113
Just a note, this is running on a 2010 Mac Pro too. Ancient hardware, as I’ve said before if it runs on this? It will run on anything. Want to do ML on a potato? 🥔 Let’s do it. #MachineLearning
1
76
Memory optimizations removed 1.2gb of memory allocations on gpt2. That’s a big win. 824mb for gpt2 load fp32 with full kv cache to sequence length. No quality loss. Building fp16 kernels now to see how these llama models do. Optimizations should make larger models run on hardware it’s never run on before. #MachineLearning #Transformers
1
1
74
Hack the planet? No Cache the Planet.
53
This is not just an inference engine, it’s distributed ML infrastructure platform. Multi precision concurrent execution (FP32/FP16, INT8, INT4) Per model isolation with no shared state Streaming disk loading of up to 70b models without OOM. Websocket with AES-256-GCM Request batching and orchestration Metal/CUDA backend abstraction memory management with GPU synchronization Metal buffer lifetime guarantees Streaming archive format with crash recovery Configurable loader strategies This is a ML serving infrastructure that happens to run efficiently on consumer hardware. It’s a 320kb binary for the C side. #MachineLearning #AI #Inference
GPT2 prompt that yielded stable attention across 540 tokens running on a 2010 Mac Pro with my inference engine. lol this thing is nuts. 100% custom memory management engine, novel attention management, math purity to 0.000000 no compromise zero tolerance for drift across all layers. 100% designed by me inference engine running on a potato (2010 Mac Pro). Runs on my Mac mini no problem all the metal shaders work for entire pipeline. Cuda support is almost complete. It speaks crazy, like how the hell is GPT2 124m doing this? Usually it’s like “Dr Chen is a doctor. Dr Chen is a doctor.” prompt = "Dr. Sarah Chen, a neuroscientist at Stanford, discovered something unexpected while analyzing brain scans from her latest study. The patterns she was seeing in the visual cortex didn't match any of the established theories about how humans process faces. After triple-checking her methodology and running the data through three different analysis programs, she realized she might have stumbled onto" #Ml #MachineLearning #Inference #AI
1
143
GPT2 prompt that yielded stable attention across 540 tokens running on a 2010 Mac Pro with my inference engine. lol this thing is nuts. 100% custom memory management engine, novel attention management, math purity to 0.000000 no compromise zero tolerance for drift across all layers. 100% designed by me inference engine running on a potato (2010 Mac Pro). Runs on my Mac mini no problem all the metal shaders work for entire pipeline. Cuda support is almost complete. It speaks crazy, like how the hell is GPT2 124m doing this? Usually it’s like “Dr Chen is a doctor. Dr Chen is a doctor.” prompt = "Dr. Sarah Chen, a neuroscientist at Stanford, discovered something unexpected while analyzing brain scans from her latest study. The patterns she was seeing in the visual cortex didn't match any of the established theories about how humans process faces. After triple-checking her methodology and running the data through three different analysis programs, she realized she might have stumbled onto" #Ml #MachineLearning #Inference #AI
1
1
295
[TEST] Decoded text: a new theory that is far more interesting than any previous study that has been done. Chen was a part of a team of neuroscientists that developed a new model of human brain structure that had a clear and consistent picture of the human brain. It was based on a single set of brain scans. The team, led by Dr. John D. Ziegler, a neuroscientist at the University of California, San Diego, had been studying how the human brain processes faces for decades, but the data they were seeing were so inconsistent with previous research that they were now using a new method of computer vision to analyze the brain. The team first came up with a model of the human brain that looked like a picture of a person's face. It was based on a series of images of a human face. As the researchers worked through the images, they created a new model of the human brain that was consistent with prior research that had not yet been done. The new model of the human brain, the model that they call an "architecture of the human brain," is based on the model that they used to explain how human brains are made. The team's new model of how the human brain processes faces was based on the same technique used to make models of humans' brains. The new model of the human brain is based on the same technique used to make models of humans' brains. The model that they used to model the human brain was based on the same method used to make models of humans' brains. The team then ran a series of test images of the human brain. When they looked at the images, they found that the human brain had a different pattern of neurons. It had a more prominent pattern of neurons in the left side of the brain, which is the part of the brain that processes visual information, and a different pattern of neurons in the right side of the brain. The team found that the human brain had a different pattern of neurons in the left side of the brain, which is the part of the brain that processes visual information, and a different pattern of neurons in the right side of the brain. The team then ran a series of test images of the human brain. When they looked at the images, they found that the human brain had a different pattern of neurons in the left side of the brain, which is the part of the brain that processes visual information, and a different pattern of neurons in the right side of the brain. The team then ran a series of test images of the human brain. When they looked at the images, they found that the human brain
112
QKV projection now exact match Attention now exact match Softmax now exact match Attention output exact match GPT almost integrated into my custom C inference engine. Metal kernels and cuda kernels standard. Runs on Mac Pro from 2010 with open core and AMD RX 580-Modern. 100% portable Mac x86_64, Apple Silicon, x86_64 + Nvidia. Model Agnostic with Llama and GPT automatic detection of model, config, pow 💥 done. #MachineLearning #AI
109
Wanted to run a language model. Fired up the ‘industry standard’ tools. Watched them immediately consume 10GB of my 12GB RAM just to import. On my Nvidia, Said what the actual fuck and deleted everything. Fine, I’ll build my own. #MachineLearning #AI #Performance
1
1
73
Currently stress testing by running 1.3B parameters on a 2010 Mac Pro because if it works here, it’ll work anywhere. Most engineers hit resource walls and give up. I hit walls and go through them. #SoftwareEngineering #Optimization #TechTwitter
1
61
The lesson isn’t ‘don’t use frameworks.’ It’s ‘question everything.’ When tools consume 10x the resources they should, when everyone says ‘that’s impossible,’ when the only answer is ‘buy bigger hardware’ - that’s when you know there’s a better way waiting to be built. #Engineering #Innovation #TechPhilosophy #FirstPrinciples #PerformanceEngineering #BuildBetter
52
How do I concatenate 3D tensors in Elixir binary format to feed custom Metal shaders for hot-swappable transformer inference? #MachineLearning #AI
1
1
2
152
Ever stop and think all these ML engines are trash and need to have. Defensive programming ✅ Atomic trust ✅ Fail Loud and fast ✅ Unit test all functions ✅ Cyclomatic Complexity of less than 8 ✅ Never hit OOM ✅ Validate all layers and shapes with size ✅ Platform and Model Agnostic? ✅ Automatically detect tokenizer ✅ Validate tokenizer ✅ Memory corruption checks ✅ Replay of step by step layer by layer ✅ Truth in the system. Is this what we want. Then distributed, platform and model agnostic belt fed ML? Orchestration, batching that is a supervisor? That’s exactly what this is. 100% custom designed ML infrastructure. Zero IP limitations. Owned by me my code my design. Coming soon. No fuck it ship it, this is not your grandpappys AI. #MachineLearning #AI #BeltFed #OMGOptimized #NextLevelShit #NoRules #NoCompromise
1
67
How do we know this is truly belt fed badass? Try running a 1.3b parameter model on this. #MachineLearning #AI #RetroAI When you are developing AI that works on a machine that is the 2nd year of the Obama Administration? When most people were still using flip phones! Efficiency and reliability are standard gold. Yes this means custom metal shaders for AMD on Apple and Apple Silicon. Why? If you can get performant AI on a dinosaur in tests what’s that mean for modern hardware. My MacPro is my legendary development machine. I’m building AI in a machine that belongs in a museum.
1
59