We were slated to sign for a mid 6-figure contract this week with @SpaceXSI to roll out @bot across the org.
That fell apart over the last few days with the severe drop in quality with no control over models and the comments on it being unsustainable.
Had a killer pilot with them but it died at the finish line.
Unlike the rest of X I’m not going to sit here and complain but it gave me a bit more insight into enterprise buyers (of which we sell to as well)
There is this massive uptick in what I’ll call buyer commitment reluctance.
Signing an annual or multi-year in a world where things are moving so fast is becoming brutally difficult to do.
Do I think @elonmusk will dial it in extremely fast? Of course. He’s the GOAT.
But for 3 days, when the quality became so undependable it forced us to go look at how we might be able to solve this in another way that doesn’t require the commitment of significant spend with the combination of no control over what models get used and on what.
Makes me consider how we might be positioning ourselves in our own sales process.
Still a huge fan. Will still use it with some key leaders on the team but the risk is just too high.
For those following along, here is my update after meeting with the SpaceXSI team
nitter.cf/JoeyGilkey/status/2119…
Update #2 (after meeting with the @bot @SpaceXSI engineering team):
- Team is great. They care a ton. They owned it.
- They admitted that it was almost entirely from pulling Opus 5.5 out. Which is now back as of this AM.
- Learned a little bit about the context-compaction process with Grok 4.7-summarize which is not a public model. Plays a role in routing config and the summary we get back.
- Grok 5.0 should be insane next level (just as an aside point)
- I am somewhat surmising on why stuff went so poorly over the weekend but a lot of it has to do with not just pulling Opus 5.5 out but when they did that a known problem (and was on cursor as well) showed up which is when you switch the model mid task (which would’ve happened by removing our 5.5) it cannot remember the context and even of what it can gather its truncated (likely do to 4.7-summarize compacting). So in our case what likely happened was we had a lot of stuff mid flight that was likely pulling on opus 5.5 heavy, then it got pulled, routed to different model, forgot the context, and turned incredibly dumb. We laughed when I showed them a project that was mid flight in a doc. The first half of the doc was intelligently designed, formatted, etc. and the second half of the doc was a wall of seemingly incoherent, non formatted text. Again, I’m surmising but I think it was dumber model with all context removed
- I won’t say much here but Elons comment on it being “unsustainably high relative to revenue” was a much larger chess move. Won’t say much past that. Not my place. But he’s definitely playing a much larger chess match than just “Grok bot isn’t sustainable”.
- The problems we all experienced should be fixed and I do not think they want to go back to what they experienced over the weekend with the hit it took.
- They are very much focused on the enterprise use case for grok bot in the future
All in all, how’s this impact my decision making moving forward with the enterprise contract we have:
- Honestly still weighing the decision vs just using team plan. Working through some terms
- the reason to move forward is pretty simple. Unlimited users (so not paying per seat but for usage), access to forward deployed team and engineers for support and circumstances like I just experienced, very granular governing and guardrails for the org. That’s it
Impressed with the team overall and I think the main benefit to staying vs moving to open source (for a company like mine vs being a solo/small company) is trusting their ability to ship and keep building plus keep it maintained.
Shoutout to @ASankarraman for jumping in to help and to the @SpaceXSI, @bot, and @elonmusk teams
35
13,712
Update: meeting with their engineering team today.
Hoping to understand what happened and what plans are to mitigate.
We were slated to sign for a mid 6-figure contract this week with @SpaceXSI to roll out @bot across the org.
That fell apart over the last few days with the severe drop in quality with no control over models and the comments on it being unsustainable.
Had a killer pilot with them but it died at the finish line.
Unlike the rest of X I’m not going to sit here and complain but it gave me a bit more insight into enterprise buyers (of which we sell to as well)
There is this massive uptick in what I’ll call buyer commitment reluctance.
Signing an annual or multi-year in a world where things are moving so fast is becoming brutally difficult to do.
Do I think @elonmusk will dial it in extremely fast? Of course. He’s the GOAT.
But for 3 days, when the quality became so undependable it forced us to go look at how we might be able to solve this in another way that doesn’t require the commitment of significant spend with the combination of no control over what models get used and on what.
Makes me consider how we might be positioning ourselves in our own sales process.
Still a huge fan. Will still use it with some key leaders on the team but the risk is just too high.
Update #2 (post meeting with team)
Update #2 (after meeting with the @bot @SpaceXSI engineering team):
- Team is great. They care a ton. They owned it.
- They admitted that it was almost entirely from pulling Opus 5.5 out. Which is now back as of this AM.
- Learned a little bit about the context-compaction process with Grok 4.7-summarize which is not a public model. Plays a role in routing config and the summary we get back.
- Grok 5.0 should be insane next level (just as an aside point)
- I am somewhat surmising on why stuff went so poorly over the weekend but a lot of it has to do with not just pulling Opus 5.5 out but when they did that a known problem (and was on cursor as well) showed up which is when you switch the model mid task (which would’ve happened by removing our 5.5) it cannot remember the context and even of what it can gather its truncated (likely do to 4.7-summarize compacting). So in our case what likely happened was we had a lot of stuff mid flight that was likely pulling on opus 5.5 heavy, then it got pulled, routed to different model, forgot the context, and turned incredibly dumb. We laughed when I showed them a project that was mid flight in a doc. The first half of the doc was intelligently designed, formatted, etc. and the second half of the doc was a wall of seemingly incoherent, non formatted text. Again, I’m surmising but I think it was dumber model with all context removed
- I won’t say much here but Elons comment on it being “unsustainably high relative to revenue” was a much larger chess move. Won’t say much past that. Not my place. But he’s definitely playing a much larger chess match than just “Grok bot isn’t sustainable”.
- The problems we all experienced should be fixed and I do not think they want to go back to what they experienced over the weekend with the hit it took.
- They are very much focused on the enterprise use case for grok bot in the future
All in all, how’s this impact my decision making moving forward with the enterprise contract we have:
- Honestly still weighing the decision vs just using team plan. Working through some terms
- the reason to move forward is pretty simple. Unlimited users (so not paying per seat but for usage), access to forward deployed team and engineers for support and circumstances like I just experienced, very granular governing and guardrails for the org. That’s it
Impressed with the team overall and I think the main benefit to staying vs moving to open source (for a company like mine vs being a solo/small company) is trusting their ability to ship and keep building plus keep it maintained.
Shoutout to @ASankarraman for jumping in to help and to the @SpaceXSI, @bot, and @elonmusk teams
3
1,579
Update #2 (after meeting with the @bot @SpaceXSI engineering team):
- Team is great. They care a ton. They owned it.
- They admitted that it was almost entirely from pulling Opus 5.5 out. Which is now back as of this AM.
- Learned a little bit about the context-compaction process with Grok 4.7-summarize which is not a public model. Plays a role in routing config and the summary we get back.
- Grok 5.0 should be insane next level (just as an aside point)
- I am somewhat surmising on why stuff went so poorly over the weekend but a lot of it has to do with not just pulling Opus 5.5 out but when they did that a known problem (and was on cursor as well) showed up which is when you switch the model mid task (which would’ve happened by removing our 5.5) it cannot remember the context and even of what it can gather its truncated (likely do to 4.7-summarize compacting). So in our case what likely happened was we had a lot of stuff mid flight that was likely pulling on opus 5.5 heavy, then it got pulled, routed to different model, forgot the context, and turned incredibly dumb. We laughed when I showed them a project that was mid flight in a doc. The first half of the doc was intelligently designed, formatted, etc. and the second half of the doc was a wall of seemingly incoherent, non formatted text. Again, I’m surmising but I think it was dumber model with all context removed
- I won’t say much here but Elons comment on it being “unsustainably high relative to revenue” was a much larger chess move. Won’t say much past that. Not my place. But he’s definitely playing a much larger chess match than just “Grok bot isn’t sustainable”.
- The problems we all experienced should be fixed and I do not think they want to go back to what they experienced over the weekend with the hit it took.
- They are very much focused on the enterprise use case for grok bot in the future
All in all, how’s this impact my decision making moving forward with the enterprise contract we have:
- Honestly still weighing the decision vs just using team plan. Working through some terms
- the reason to move forward is pretty simple. Unlimited users (so not paying per seat but for usage), access to forward deployed team and engineers for support and circumstances like I just experienced, very granular governing and guardrails for the org. That’s it
Impressed with the team overall and I think the main benefit to staying vs moving to open source (for a company like mine vs being a solo/small company) is trusting their ability to ship and keep building plus keep it maintained.
Shoutout to @ASankarraman for jumping in to help and to the @SpaceXSI, @bot, and @elonmusk teams
We were slated to sign for a mid 6-figure contract this week with @SpaceXSI to roll out @bot across the org.
That fell apart over the last few days with the severe drop in quality with no control over models and the comments on it being unsustainable.
Had a killer pilot with them but it died at the finish line.
Unlike the rest of X I’m not going to sit here and complain but it gave me a bit more insight into enterprise buyers (of which we sell to as well)
There is this massive uptick in what I’ll call buyer commitment reluctance.
Signing an annual or multi-year in a world where things are moving so fast is becoming brutally difficult to do.
Do I think @elonmusk will dial it in extremely fast? Of course. He’s the GOAT.
But for 3 days, when the quality became so undependable it forced us to go look at how we might be able to solve this in another way that doesn’t require the commitment of significant spend with the combination of no control over what models get used and on what.
Makes me consider how we might be positioning ourselves in our own sales process.
Still a huge fan. Will still use it with some key leaders on the team but the risk is just too high.
Joey Gilkey retweeted
We are using Joey’s team to build out our outbound sales and they are 💪🏼
This quoted post is unavailable.
Joey Gilkey retweeted
When scaling, look at your team.
Make sure 20% has “been there done that.”
Otherwise you’re going to be paying down a lot of ignorance tax for people to learn lessons with your company as their guinea pig.
…only to become another company’s “been there done that.”
Joey Gilkey retweeted
Just listened to an incredible episode of @JoeyGilkey’s podcast with @ian_martins.
Some very important points discussed about content strategy when it comes to growing an agency.
Joey Gilkey retweeted
Unlock Your Profit Potential!🔑 🔓
Our CEO Marcel was a recent guest on the Sales Driven Agency Podcast w/ @joeygilkey - allegedly the #1 Agency Growth Podcast in the universe.
Crucial metrics, pricing, taxes, you name it, all at the link below 👌
youtube.com/watch?v=H5YtbJG7…
