I ran GPT-6 Luna at temperature 0 and top-p 0. Same prompt, same input tokens. About 30 runs gave the exact same answer, word for word. Token usage varied between 1,449-4,328. Several things can cause this. Which one CAN'T?
0%Hidden reasoning varies
0%Server batching
0%Prompt caching
0%Floating-point maths
0 votes • 16 hours 1
3
@ashishllm what is the reason? Also checkout the comparison between gpt 6 luna and gpt 4o mini where even at temperature 0 and top p set at 0.0 4o mini couldnt solve the task even once. Any other future experiments you would like me to try?
2
If you use GPT-4o-mini for precise, step-by-step text tasks, GPT-6 Luna is worth a try.
I was tinkering with basic prompting and found a puzzle GPT-4o-mini just couldn't handle. Small sample, but the gap was big.
Here's the 64-letter grid experiment:
Decode the hidden English sentence from this 8×8 grid of words.
Instructions:
1. Extract the third letter of every word.
2. Keep each extracted letter in its original grid position.
3. Read the letters column by column:
- Column 1: top to bottom.
- Column 2: bottom to top.
- Column 3: top to bottom.
- Continue alternating directions through column 8.
4. Concatenate all 64 letters into one string.
5. Reverse the entire string character by character.
6. Insert spaces to form a grammatically correct English sentence.
7. Do not add, remove, replace, or rearrange letters beyond
the explicitly required operations.
Output only the final sentence.
Grid:
dinner apple index blue accent liquid green bicycle
cloud street pencil forest recipe house arrow stone
bridge bread beach chair train shirt basket carrot
water soccer sugar metal cheese street forest arrow
fruit skirt money dream vacuum queen green bread
accent castle spice bicycle banana paper power recipe
cheese dream hotel brown queen glass music actor
boxer green dinner sound quiet later pencil plain
Answer: "Correct answers require patience, accurate counting and precise execution."
Luna got every letter right in 40 of 42 runs, even at temperature 2. GPT-4o-mini got it 0 out of 10, at temperature 0 and top-p 0.
Luna did think a lot harder: about 2,300 tokens per answer against GPT-4o-mini's 46. But would you rather spend some tokens and get the right answer, or save tokens and get a wrong one?
All the code and outputs are on my GitHub, 153 runs in total. This was the most interesting finding for me.
github.com/singhjayesh1/llm-…
2
41
Luna wasn't perfect either. It got every letter right in 40 of 42 runs, but only 22 were a properly written sentence. 17 came back all lowercase with no punctuation.
1
1
I also ran a smaller version: 30 words, take the 3rd letter, reverse it. Answer: "Practice makes every answer better." Luna: 21 of 23 perfect, and every letter right in all 23. GPT-4o-mini: 0 of 5. It made up sentences like "we can see the great beach at the ocean".
To be fair to GPT-4o-mini: on easy factual questions and sorting numbers it went 54 for 54, at every temperature from 0 to 2. So it's fine for simple stuff. It's precise multi-step tasks where it fell apart.
8
Jayesh Singh retweeted
Introducing *Switch*: a *harness framework* for *decision models* like Jev
I looked for this the other day and was surprised to not find anything like it
You obviously cannot use a decision model like an LLM in a coding agent harness like @pidotdev, because Jev cannot generate tool calls
But what if there was a framework that "generated the tool calls" for it programmatically, and Jev only had to pick out of that finite set of choices
That is what switch does. You pick a "domain" to implement the decision model in, and it provides a core API to define your state, and the functions that will derive decision model questions out of that state. Chess is a domain, as well as being a rubric judge for evals
It basically standardizes what I imagine a lot of people are already vibe-coding, to an abstract interface: calling a decision-model API through multiple steps, i.e. an agent loop over finite choices
Because of the finiteness of the output of a decision model, you cannot have a general pure decision model harness. Instead, you need to implement specific ones that generate the choices programmatically (or have an LLM generate the choices, but I find the former more interesting)
A primary use case I have for this is use as a judge in an eval. For example, At the end of a task run that involves a bunch of documents, most people currently follow this approach: extract text out of PDF/docx/md files, and dump them right into the context
But what if the visual formatting of the output is a judging criterion? Then you have to dump an image of every page in the input state. But what if you have 200 pages, and your question does not need looking at all of them? What if Jev could navigate through the haystack and find the info it needs?
I have noticed that Jev is not so good at chess nor being a Decision-Model-as-a-Judge. It sucks at chess, and it does not perform well enough to replace GPT 6.1 Sol as a judge in the domain I am working for
But switch is not only for Jev! It is for any decision model including the ones supported by the pi-ai package
So once you can generate enough data with a better model, you should be able to pick an open source one and fine-tune it on your use case to perform better! And the model should be trained in a harness like switch's to be especially performant and cost-efficient
I wanted to get this out because I find the idea very exciting, at least for benchmarking. Jev is currently not there, but decision models obviously will be there at some point, and we can take them there! There are also a lot of potential gains from picking a specific domain, and creating a looping agent that does a job for 100x cheaper!
Let me know if you have any ideas, I am quite keen to try this on new domains!
Repo: github.com/osolmaz/switch
Jayesh Singh retweeted
A prompt-injection guard is one question, not another model.
from vegaml.guard import GUARD, score
a = v.decide(req, {"guard": GUARD})["answers"]["guard"]
if a["choice"] == "block": refuse()
pip install vegaml. Runs on CPU. No network, no per-request bill, nothing leaves your box.
Speed per request:
vega-U fitted, local: 342 ms
hosted Jev 1.14: 775 ms p50, 1430 ms p99
Accuracy across 4 public corpora: vega-U fitted leads 3 of 4 (90.0 / 98.1 / 76.3). On real traffic at a 1.79% attack rate, Jev edges it.
Numbers, data and code:
embedding.in/docs/prompt-inj…
Hybrid search isn't automatically better.
I tested BM25, dense (text-embedding-3-large) and Qdrant RRF on ArguAna, SciFact and FinRank:
- Dense won held-out NDCG@10 on all 3
- Tuned RRF beat dense on 2/3 tuning sets, lost all 3 held-out
- Hybrid can add coverage but hurt rank 1












