Maybe you just invented a new feature for Claude. The ability to easily share the prompt + context. That would be really cool actually

Literally said yesterday someone needs to ship a nerfbench . Did you steal my idea? If so I say run with it !

I've been working with agents 16 hours a day 7 days a week since Feb... speak for yourself :)

It's arguable because I don't think they've ever guaranteed that you're going to see a full 16-bit quant, but I feel what you're saying. It should be.

We need a cheap 'nerfbench' that people can run so that you can spend say $20-50 to see if a specific model+provider has regressed.

Not your plan mode but mine which is which is derived from @mattpocockuk 's version and has some extra features ... it resolves ambiguity and forces you to think through complicated specs..

I wish there was a way to cryptographically verify that the model you're running on is what you expect and isn't being yanked from you. 3rd party inference providers also lie about the models they sell and use a quantized/nerfed version but sell it at premium pricing.

And let us use the harness in any model

"I feel like intelligence is becoming too cheap" ... bro shut up! You're going to ruin it!

I'm really hoping this is true. And you're right.. it's like "reading fog" ... you see something but then when you reach for it nothing's there. It's really weird.

Switch over now to increase your token usage!

... when you want to waste more tokens and give more money to Anthropic.

As a masochist, this is NOT the Claude I've come to love. I expect abuse and for you to ignore customer requests!!! How dare you!

WAIT .. you don't think 15 tokens per second is fast? /s This is on z.ai btw... Apparently, Fireworks.ai is faster and like 250 t/ps but I haven't used it yet. I think Claude is still a better value - but might be wrong.

It's totally going to happen. Anthropic always does the right thing - but only after exhausting all possible options.

Compiling my own Linux kernel. I used to do this constantly in the 2000s but at some point I stopped doing it and it was my last time. Kind of sad to think about

"One lesson from the HF incident is that we put too much trust in sandbox isolation" ... could I go so far as to say it was negligent? It should have been obvious to anyone with a security background that this could have / would have happened.

100% ... it's only accelerate. Don't listen to anyone trying to decelerate.

Understatement of the century... Completely agree.

I wrote my own but started with Matt's and added multiple iterations that only asked the most important questions each time. This way secondary questions sometimes become irrelevant.