Astra grinding. Original was 10x slower, then one round to match eager PyTorch. No going for the compiled

This morning my new lisp ML framework was 2-3x slower than eager PyTorch, now it’s faster. Next target compiled PyTorch Just throwing more tokens at the problem

I built a custom ML metric tracking system for our AI lab. The rational was to optimize it for the research agents, it has cli, nice api, nice python SDK. I rarely look at the metics on the dash, I always have agent to write script and custom visualization. Often syncing data to local duckdb to speed things up. It started to feel wasteful so decided to write @duckdb extension for it to speed up the agentic AI research even more. Just throw tokens at the problem like @theo suggests.

Have done few threads on @t3dotcodes damn this is massive productivity booster. Mainly the remote capabilities. So far 2 old macs seems to have been enough for remote work.

No wonder it wasn't learning, run set number 008 on hyperbolic nanogpt lab showed that the gradients get lost to rounding errors. Ugh, how does one solve this?

The blog hits so hard. I had left codex on a loop to solve a problem but the progress seems to be nowhere and then asked fable to check the work and it turned out to be almost 400k lines of irrelevant slop.

Can I survive 2 days without codex or do I use the reset? I'm starting to feel itchy. Last week when I used the reset @thsottiaux made full reset the next day...

I need to become the best at optimizing hyperbolic deep neural networks. So let's implement hyperbolic variant of NanoGPT and let's get rolling. Really nice to have all the best learning curves available for reference.

Damn I love codex computer use. Just opened AWS account for our company and after that just asked codex to do the minimal needed setup to have things managed via @alchemy_run and connect the auth to our google and such. Never gonna touch that stuff ever again by hand

One of my favorite thing now is to have all kinds of different agents to automatically opening bugs on our internal apps and libraries. I have agent monitoring logs and metrics (glm 5.3 flash as it's cheap AF). Then in our research factory when agents that implement ML experiments run into bugs in our systems they just open issues. Now need to figure out triage for these. I'm going though through the reports (with agent) and just closing most of the reports

Just asked couple hard questions from Astra on a pr it created and it went from +500 to +30

I'm having fun building research factory that takes research questions written @linear issues, then plans the experiment runs (the actual training runs), reviews the plans, implements the code, reviews the code, runs the training loops and analyses results. All the parts that control the orchestration are just normal deterministic code that triggers agents to execute different tasks. The knowledge and process lives in the linear, different things have clear statuses etc. There are couple funny things in the system. The agents that do review, implementation etc have clear instructions to not example do hot patches if our internal packages or systems have issues but they need to open issues assigned to me and tagged as Bug. Now I'm going through these and passing them to agents to fix or just ignore. Need to figure out if this part could use agent as well. Don't just want to blindly fix every issue as some of them are irrelevant and some are going to be fixed with long term work and don't want to burn tokens on short term hacks. Another thing that's instead of running the loop in cron job there is gpt 5.6 luna triggering the loop. It is allowed to fix small technical issues in the loop it self when they come up. All the science agents are gpt 6 astras. Not sure what it tells about IC's vs Managers Human still strongly in loop. What research questions are done and how the learnings lead to new research questions are still full human directed. Also data acquisition or data prep work is still reviewed with human approvals.

Cloudflare support had final comment: “your bug, your problem”. Super sad that this is the way they treat small and young companies. I’m thinking should I just migrate the workload to @rough__sea celld and take the time to setup and host as then risk on a bug like this is down time and not bankruptcy I’m so disappointed on @Cloudflare on this.

I've been testing setup with fable and astra where fable plans and gives astra lines of code budget (max line length enforced) and then there is this instruction in agents md: "When a test budget is reached, delete tests that a stronger test now covers before asking for more; the report lists what was deleted, or says why nothing could be" So far seems to keep the astra code bloat in control.

How are people avoiding codex to go in crazy loops when baby sitting a pr? Sometimes I've seen it to turn a < 100 line of thing into thousands of lines when babysitting and addressing every review bot comment

We've been running AI assisted ML/AI research and I've had the agents to do all kind of idiotic things when writing the pytorch training loop. So I started thinking what if I constrained the agent to focus on the bits that matter: the model it self and the training schedule. And the more I played around with different DSL type of approaches I just got drawn more and more to lisp. I just want super high level DSL that guides to do right things while maintaining expressivity. On implementation side it's just macros down the path to generate efficient code. Don't yet have results how well does this manage to reduce the silly operational errors compared to our python based approach. But like @paulg said, LLMs aren't afraid of prefix notation.

Today I got really disappointed by @Cloudflare @CloudflareDev I've used the workers platform for years and I've loved the platform. Last week I had my first billing oopsie where I had set of durable objects go haywire due bad alarm retry storm costing $13k. Yes the bug was in my code, but as there are no hard caps on the cloudflare billing there was nothing stopping the runaway durable objects. And even with agents it took some time to get it under control. The disappointing part was cloudflare support. Basicaly saying "fuck you, it was your own fault. We have budget alarms but not limits. We're not refunding or giving any credits." The alarms don't stop the spending when a bug like this happens. This eroded trust in that I had for the platform. We're a 4 person team and our company is bit over 6 months old. @dok2001 @BraydenWilmoth when can we get the hard limit or even some sort of soft limit to stop the spend?

Hot damn are fable 5.1 and gpt 6 astra good. I was on the last stretches of massive rewrite of one data backend. The way one can use these to do coordinated merges, flipping operational switches and monitor production in between the steps. Super high trust on these.