techvatsa

Part 3 of 5 - Understanding how to measure answer quality In part 2 we saw how one can use standard model attributes provided by OpenLLMetry / OpenLIT libraries and inject their custom business attributes and get insights into the performance and cost of LLM calls. But when it comes to correctness, AI applications deviate from our standard practices.

The days around 21 October were very bad for the internet: • PHP2Shell: two critical PHP RCE bugs • Node.js: four high-severity fixes • LiteLLM 1.107.0 shipped with a credential stealer • eMudhra CA: 8 weak private keys, one was the number 4 How many squares on our card did your team get? #Security #SupplyChain #LiteLLM

Part 2 of 5 - Understanding traces for LLM requests Observing AI applications relies on the same telemetry signals as everything else - metrics, logs and traces. This post looks at what a useful AI trace is made of: which span attributes you get for free from standard libraries, and which ones you still have to work for.

No curl, wget or nc on a locked-down host? Try this next time - (exec 3<>/dev/tcp/HOST/443) && echo OPEN || echo CLOSED - On bash 5.2.21: dash (Ubuntu's sh) fails, - bash --posix still works, and HTTP/1.1 hangs without Connection: close. Read more here from our blog with @sphirani ! one2n.io/blog/test-tcp-conne…

1/5 SGLang v0.5.21 shipped last week with updates across prefill performance, disaggregated serving, prefix caching, AMD GPU support and decision models. A few changes stand out if you're building or operating LLM inference infrastructure. github.com/sgl-project/sglan…

Understanding memory on NVIDIA’s GH200 and GB200 superchips! > An AI model runs on a GPU. The GPU keeps the model in its own memory, called HBM. HBM is very fast, but it is small. > NVIDIA’s GH200 also lets the GPU use the CPU’s memory. CPU memory is much larger, but slower. Together they give up to 624 GB. Can an AI model use all 624 GB at the same speed? No. This thread explains why, in simple steps.

Problem-2 OOMKills at traffic peaks: memory_limiter first, with GOMEMLIMIT aligned to the memory budget. Problem-3 Health-check noise: filter routes using http.route. Problem-4 Missing pod metadata: use k8sattributes, but avoid enriching every span with high-cardinality fields you don't need.

Blog by Sanket (Our in house OTEL hero) tells us Six OTel Collector problems on Kubernetes you commonly face, and the processor setting fixes that might sit behind them. Problem-1 Backend 429s: use batch to reduce export request overhead. Put it after processors that filter or drop telemetry. Example: 30s timeout, batch size 1000.

If models releases are bothering you, here's something - 1/3 Sonnet 5.5 (28 Sep), GPT-6.1 Sol (29 Sep) and Gemini 4 Argon (30 Sep, gated access) launched around the same price: $2/M input tokens and $10/M output. Same list price doesn't mean the same cost per task. Interested? let's dive in

AI Observability Metrics! Time to first token and tokens per second are two different bottlenecks Time to first token and tokens per second hit different hardware limits. We're writing a small 5-part series on inference engineering from the silicon up. Part 1: the two phases of every LLM request - prefil and decode

A Postgres publication setting silently dropped CDC events for weeks! 1/4 Four tables in a Postgres to Debezium CDC pipeline stopped updating in prod. Flush lag 0.02 s, Debezium 62 ms behind source. Every health signal looked normal, and nothing alerted for two weeks.