baserun.ai retweeted
Effy Zhang joins us for Office Hours this week!
She's the CEO & Founder at @baserunai.
Before baserun, @effyyzhang led design at Cruise and Square.
✨ Ask her a question on Elpha by 05/03:bit.ly/4aXaatp
baserun.ai retweeted
Congratulations to @DeepSourceHQ team on the launch of their Autofix AI feature, powered by @baserunai! There are many agents who write code; DeepSource Autofix feature ensures that your code written by agents is production-ready.
Static Analysis + Autofix™ AI
If you're using GitHub Copilot, DeepSource runs continuous static analysis in the background to detect thousands of code quality and security issues (static analysis, SAST, IaC) and helps you fix them with Autofix AI.
Here's a video of me converting some synchronous Python code to async; Copilot correctly (and very quickly) figures out how to re-write the relevant part of the function, but doesn't really follow the best practices.
Notice how DeepSource detects the anti-pattern as soon as the file is saved, prompts me with a warning, and helps me automatically fix the problem.
This is just one of the thousands of issues DeepSource can detect and fix. We're working on the public release for Autofix™ AI soon. DM for access!
baserun.ai retweeted
Baserun now supports Jinja2 template @baserunai
baserun.ai retweeted
Problems I'd do if I'm to do a startup again (though I probably won't any time soon because startups are hard). If you’re solving any of them, I’d love to chat.
1. Data synthesis: AI has become really good both at generating and annotating data. The challenge now is to make sure that the generated data is safe and legal, e.g. not violating any IP.
2. Evaluation: evaluation has gotten so much harder with LLMs, both because many people treat models as blackboxes (we deploy models someone else developed for us) and because outputs can be open-ended. At the same time, investment in evaluation is nowhere close to investment in model or application development. I’d be interested in arena-style evaluation, embedding evaluation, human-in-the-loop evaluation, as well as small, specialized scorers (instead of using large models like GPT-4 as judges).
3. Energy: the bottleneck to scaling AI is no longer compute but electricity. I’m interested in all energy-related problems, including both new energy sources and energy trading.
4. Any application that allows you to collect unique data that nobody has. I’ve heard concerns about building applications that seem to be “wrappers” around popular APIs. If you can get to the market early and gather sufficient data to continually improve your product, data is your moat.
5. GPU-native everything: many data science toolings, including scikit-learn, pandas, and Spark, aren’t built to run natively on GPUs. There have been efforts to make these tools more efficiently leverage GPUs, but I think there’s still a lot of room for the software layer for GPUs (and not just NVIDIA GPUs).
6. Curated Internet: bots are already ruining dating apps, search (bots are incredibly good at SEOs), and social media. I’d like to be able to set a boundary for my Internet, e.g. to limit the search results to those written by people I trust, or sources verified to be human.
baserun.ai retweeted
Replying to @brian_lovin @baserunai
Dude, thank you. Started using this yesterday and it’s a game changer.
baserun.ai retweeted
chatted with @effyyzhang today about what she is building at @baserunai. she's positioned to become the queen bee of ai tools imo.
most co's are following the industry buzz to build consumer-facing AI tools, and it's so refreshing to see a startup building to support the buzz.
baserun.ai retweeted
Baserun can now evaluate & improve your prompts with AI — automatically! Try it now →
app.baserun.ai/sign-up
@baserunai
baserun.ai retweeted
Crafting the perfect prompt can be challenging. There are tons of prompt techniques out there, but mastering them still requires lots of trial and error. What if we could make it easier? We're teaching an AI to understand all these tricks and write prompts for you.
Here is a sneak peek of what's coming next. Stay tuned!
baserun.ai
@baserunai
baserun.ai retweeted
Claude-3 is now available in Baserun Playground!
Why Baserun Playground?
- A collaborative workspace for teams to share prompts
- Version history ensures you don't lose any changes
- Dynamic Inputs for bulk testing
- Compare prompt versions side by side
lnkd.in/gUVw23WB
@baserunai
baserun.ai retweeted
Saving costs on evaluation: Baserun automatically caches your evaluation results to avoid running redundant assessments.
baserun.ai retweeted
Prototype and evaluate prompts in the Baserun UI: Anybody (not just developers!) can prototype ideas in the Playground, test and assess prompts with our Evaluation features directly in the UI, and then deploy those changes to staging or production environments using the Prompt Directory feature.
baserun.ai retweeted
Maximum flexibility: Baserun supports automatic evaluators, human evaluators, feedback, and checks. Automatic evaluators include string match, fuzzy match, JSON validation, and Regex match. Model-graded evaluators include fact checking, closed QA, and security evaluations. Users can also define their own custom code evaluators or custom model-graded evaluators. More evaluators to come!
docs.baserun.ai/evaluation/a…
baserun.ai retweeted
Comparing changes side-by-side: With our comparison report, developers can compare two branches before merging a PR to prevent regressions.
baserun.ai retweeted
Support both unit tests and end-to-end tests. Like traces, users can assess the outputs for a particular LLM call or the entire pipeline.
baserun.ai retweeted
This week at Baserun:
Developers can now create evaluators using custom code or a custom LLM prompt to grade testing results pre-release or to monitor production post-release.
baserun.ai retweeted
New Offline Evaluation Reports: Many UI improvements have been made to how metrics and evaluation details are displayed.











