Rob retweeted
10 GITHUB REPOS TO BUILD SERIOUS WEB SCRAPING WORKFLOWS
Need to collect, parse, or transform web data?
These open-source projects are worth knowing:
1. Crawlee
Build scalable crawlers with browser automation, retries, proxies, and queues.
github.com/apify/crawlee
2. Scrapy
A powerful framework for building fast, scalable web crawlers.
github.com/scrapy/scrapy
3. Playwright
Automate real browsers and handle JavaScript-heavy websites.
github.com/microsoft/playwri…
4. Beautiful Soup
A simple way to parse HTML and extract useful data.
github.com/wention/Beautiful…
5. Firecrawl
Turn websites into clean, AI-ready content.
github.com/mendableai/firecr…
6. Crawl4AI
Convert websites into structured data for LLM and RAG workflows.
github.com/unclecode/crawl4a…
7. ScrapeGraphAI
Use natural language to define what data you want to extract.
github.com/ScrapeGraphAI/Scr…
8. Colly
A fast and lightweight web scraping framework for Go.
github.com/gocolly/colly
9. trafilatura
Extract clean text and metadata from web pages.
github.com/adbar/trafilatura
10. Newspaper4k
Extract article content, metadata, and related information automatically.
github.com/AndyTheFactory/ne…
All open source.
The web is full of useful data.
These repos help you turn it into something you can actually use.
Save this list.
Rob retweeted
10 GITHUB REPOSITORIES YOU’LL WISH YOU FOUND EARLIER
I went looking for projects that don’t show up in every AI thread.
Found some seriously good ones.
1. Vero
AI coding agent that verifies generated code using Lean 4.
github.com/sunblaze-ucb/vero
2. Agentor
Adds agent discovery, A2A support, streaming and artifact sharing.
github.com/CelestoAI/agentor
3. Sky-Agent
Lightweight framework for building AI agents with tools, memory and files.
github.com/arteemg/sky-agent
4. PageIndex
Reasoning-based RAG for navigating long documents.
github.com/VectifyAI/PageInd…
5. Semantica
Graph-based infrastructure for connecting entities, relationships and context.
github.com/HKUDS/semantica
6. NanoBot
Lightweight self-hosted AI agent with memory, tools, MCP and multi-agent workflows.
github.com/HKUDS/nanobot
7. Learn Claude Code
Build a Claude Code-style agent from scratch and understand how coding agents work.
github.com/shareAI-lab/learn…
8. Agent-Reach
Gives AI agents access to GitHub, YouTube, Reddit, X and more.
github.com/Panniantong/Agent…
9. Book-to-Skill
Turns technical books into usable Claude Code skills.
github.com/virgiliojr94/book…
10. ExploitGym
Environment for testing AI agents on real software exploitation tasks.
github.com/sunblaze-ucb/expl…
All open source.
Some are getting attention.
Others are still flying under the radar.
Save this list.
Rob retweeted
10 GITHUB REPOS THAT MAKE AI FEEL VERY DIFFERENT
Some of these are seriously interesting:
1. RuView
Turns Wi-Fi signals into spatial intelligence and presence detection.
github.com/ruvnet/RuView
2. ASC
Fast Android decompiler built for AI agents and mobile research.
github.com/MG1937/ASC
3. VoiceStudio
Local voice AI studio for cloning, dubbing, transcription and audiobooks.
github.com/debpalash/VoiceSt…
4. MiroFish
Multi-agent simulation engine for exploring complex scenarios.
github.com/666ghj/MiroFish
5. OpenResearch
Turns coding agents into research agents for hypotheses and experiments.
github.com/alphaXiv/OpenRese…
6. Atlas
Git-style source control for managing multiple AI coding agents.
github.com/pacifio/atlas
7. PI
Compact toolkit for building and running AI agents.
github.com/earendil-works/pi
8. TradingAgents
Multi-agent framework for financial trading research.
github.com/TauricResearch/Tr…
9. BrewUI
A native macOS GUI for Homebrew.
github.com/Homebrew/brew
10. tinycast
Turns your machine into a simple personal casting setup.
github.com/abue-ammar/tinyca…
Save this list.
Rob retweeted
10 GITHUB REPOS EVERY DEVELOPER NEEDS
1. LangChain
Framework for building LLM-powered applications.
→ github.com/langchain-ai/lang…
2. n8n
Build powerful workflow automations.
→ github.com/n8n-io/n8n
3. OpenClaw
Run a local AI assistant.
→ github.com/openclaw/openclaw
4. Dify
Build AI applications with a visual workflow.
→ github.com/langgenius/dify
5. Langflow
Build and prototype AI workflows visually.
→ github.com/langflow-ai/langf…
6. Mem0
Add persistent memory to AI applications.
→ github.com/mem0ai/mem0
7. Browser Use
Let AI agents interact with websites.
→ github.com/browser-use/brows…
8. CrewAI
Build and orchestrate multi-agent AI systems.
→ github.com/crewAIInc/crewAI
9. MetaGPT
Build software workflows using multiple AI agents.
→ github.com/geekan/MetaGPT
10. AutoGen
Microsoft's framework for building multi-agent AI applications.
→ github.com/microsoft/autogen
Bookmark these repos.
Rob retweeted
10 GITHUB REPOS THAT SHOULD BE ILLEGAL TO HAVE.
all free. all open-source. bookmark this for later.
1. Recordly — the free screen studio.
Open-source screen recorder with auto-zoom, smooth cursor, webcam overlay, styled backgrounds, and polished demos without a video editor. (AGPL-3.0)
github.com/webadderallorg/Re…
2. Stirling PDF — your entire PDF toolkit.
Self-hosted. Merge, split, sign, redact, OCR, convert, compress, 50+ tools. Runs locally, so nothing leaves your machine. (MIT)
github.com/Stirling-Tools/St…
3. PhotoGIMP — turns GIMP into Photoshop.
Photoshop shortcuts, layout, and splash screen patched on top of GIMP 3+.
github.com/Diolinux/PhotoGIM…
4. Open Notebook — self-hosted NotebookLM.
Drop in PDFs, URLs, and YouTube links. Chat with them, summarize them, and even generate podcasts. Supports 18+ AI providers. (MIT)
github.com/lfnovo/open-noteb…
5. Odysseus — PewDiePie's self-hosted AI workspace.
Chat, agents, deep research, docs, email, memory, and more. Local-first, your hardware, your data. (MIT)
github.com/odysseus-dev/odys…
6. FreeDomain — free domain names for everyone.
Register a domain, point it to Cloudflare or any DNS provider, and launch your site without paying for the domain. (AGPL-3.0)
github.com/DigitalPlatDev/Fr…
7. HyperFrames — write HTML, render videos.
HeyGen's open-source engine that converts HTML/CSS animations into deterministic MP4 videos. Built for AI agents. (Apache-2.0)
github.com/heygen-com/hyperf…
8. Web-to-App — turn any website into an Android app.
Configurable WebView, APK signing, and even Node.js, PHP, and Python runtimes. No remote build required.
github.com/shiaho777/web-to-…
9. Reclip — self-hosted video & audio downloader.
Download videos from YouTube, TikTok, X, Instagram, and 1000+ other sites as MP4 or MP3 using yt-dlp.
github.com/averygan/reclip
10. Excalidraw — the infinite whiteboard.
Replace Miro, FigJam, and Lucidchart with hand-drawn diagrams, wireframes, real-time collaboration, and end-to-end encryption. 120k+ GitHub stars. (MIT)
github.com/excalidraw/excali…
Which repo are you trying first?
Rob retweeted
Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.
Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.
Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.
It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.
Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.
Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides
npm i -g @magnitudedev/cli
GitHub: github.com/magnitudedev/magn…
Rob retweeted
Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU.
It's called BitNet. And it does what was supposed to be impossible.
No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed.
Here's how it works:
Every other LLM stores weights in 32-bit or 16-bit floats.
BitNet uses 1.58 bits.
Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for.
The result:
- 100B model runs on a single CPU at 5-7 tokens/second
- 2.37x to 6.17x faster than llama.cpp on x86
- 82% lower energy consumption on x86 CPUs
- 1.37x to 5.07x speedup on ARM (your MacBook)
- Memory drops by 16-32x vs full-precision models
The wildest part:
Accuracy barely moves.
BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat.
What this actually means:
- Run AI completely offline. Your data never leaves your machine
- Deploy LLMs on phones, IoT devices, edge hardware
- No more cloud API bills for inference
- AI in regions with no reliable internet
The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine.
27.4K GitHub stars. 2.2K forks. Built by Microsoft Research.
100% Open Source. MIT License.
Rob retweeted
this is f**king insane.
Jack Dorsey just open sourced a free tool for running a one-person business.
the GitHub repo already has 14.4K stars.
It lets you:
→ run your own server
→ keep chat, search, Git, and automation in one place
→ add AI agents that work like teammates
→ decide exactly what each agent can do
no monthly subscriptions.
just you, your AI agents, and full control.
this is where one-person businesses are headed.
save this for later.
Rob retweeted
10 GitHub repos that make Excel and Google Sheets look ancient in 2026.
Excel was born in 1985. Google Sheets in 2006.
These 10 repos were built for how people actually work with data now. Every one is free. Every one is open source.
1. Datasette
Built by Simon Willison, the co-creator of Django. Turns any SQLite file into a browsable, queryable website in one command. 11,000+ stars.
Repo → github.com/simonw/datasette
2. Baserow
The MIT-licensed Airtable clone. Its AI assistant Kuma builds full database structures from plain English.
Repo → github.com/bram2w/baserow
3. Grist
Real Python formulas inside every cell. No more nested IFERROR VLOOKUP hell.
Repo → github.com/gristlabs/grist-c…
4. Teable
Handles millions of rows without lag. Faster than Airtable or Sheets at scale.
Repo → github.com/teableio/teable
5. DuckDB
Query a 100 GB CSV in your browser tab. The reason data engineers stopped opening spreadsheets in 2025.
Repo → github.com/duckdb/duckdb
6. Rowy
An Airtable-style view over your Firebase Firestore. No-code and full-code together.
Repo → github.com/rowyio/rowy
7. Metabase
Points at any database and builds you a chart in 30 seconds. 42,000+ stars. Kills $900 a year Tableau seats.
Repo → github.com/metabase/metabase
8. Observable Framework
Built by Mike Bostock, the creator of D3. Live interactive dashboards Sheets cannot touch.
Repo → github.com/observablehq/fram…
9. Evidence
Markdown plus SQL equals live dashboards. Version-controlled in Git.
Repo → github.com/evidence-dev/evid…
10. Actual Budget
Envelope budgeting done right. No cloud. All local. Kills YNAB at $109 a year.
Repo → github.com/actualbudget/actu…
Rob retweeted
10 GitHub Repositories That Are Making Expensive Software Hard to Justify
($50B+ Worth of Paid Software Has Free Open-Source Alternatives)
Bookmark this.
1. yt-dlp
Download videos from YouTube, X, TikTok, Instagram, and hundreds of other platforms.
Even YouTube Premium doesn't offer this level of flexibility.
Repo → github.com/yt-dlp/yt-dlp
2. Ollama
Run powerful AI models locally on your laptop.
No API bills. No cloud dependency.
Repo → github.com/ollama/ollama
3. Fooocus
Generate stunning AI images on your own GPU.
A powerful free alternative to many paid image generators.
Repo → github.com/lllyasviel/Fooocu…
4. Whisper
OpenAI's open-source speech-to-text model.
Fast, accurate, and supports 99+ languages.
Repo → github.com/openai/whisper
5. Plausible Analytics
A privacy-first, open-source alternative to Google Analytics.
Self-host it and own your data.
Repo → github.com/plausible/analyti…
6. AppFlowy
An open-source alternative to Notion.
Host it yourself with unlimited users.
Repo → github.com/AppFlowy-IO/AppFl…
7. Penpot
A free, open-source alternative to Figma.
Built for designers and developers.
Repo → github.com/penpot/penpot
8. n8n
An open-source automation platform.
Build unlimited workflows without paying Zapier-style subscription fees.
Repo → github.com/n8n-io/n8n
9. Cal.com
The open-source scheduling platform.
Perfect if you want full control over your booking system.
Repo → github.com/calcom/cal.com
10. Bitwarden
A trusted open-source password manager.
Secure, cross-platform, and free to get started.
Repo → github.com/bitwarden/server
These projects prove that some of the best software on the internet doesn't come with a monthly subscription.
Save this thread.
You'll thank yourself the next time you're about to pay for a tool that already has an incredible open-source alternative.
100% Free.
100% Open Source.
Rob retweeted
50 USEFUL GITHUB REPOS
1. iFixAi — AI misalignment testing
→ github.com/ifixai-ai/iFix
2. public-apis — free APIs for everything
→ github.com/public-apis/pu…
3. build-your-own-x — learn by building
→ github.com/codecrafters-i…
4. developer-roadmap — learn any tech skill
→ github.com/kamranahmedse/…
5. free-programming-books — thousands of free books
→ github.com/EbookFoundatio…
6. system-design-primer
→ github.com/donnemartin/sy…
7. coding-interview-university
→ github.com/jwasham/coding…
8. the-art-of-command-line
→ github.com/jlevy/the-art-…
9. project-based-learning
→ github.com/practical-tuto…
10. you-dont-know-js
→ github.com/getify/You-Don…
11. the-book-of-secret-knowledge
→ github.com/trimstray/the-…
12. tech-interview-handbook
→ github.com/yangshun/tech-…
13. awesome-selfhosted
→ github.com/awesome-selfho…
14. javascript-algorithms
→ github.com/trekhleb/javas…
15. 30-seconds-of-code
→ github.com/Chalarangelo/3…
16. github gitignore templates
→ github.com/github/gitigno…
17. ollama — run LLMs locally
→ github.com/ollama/ollama
18. langchain
→ github.com/langchain-ai/l…
19. n8n automation
→ github.com/n8n-io/n8n
20. openclaw — local AI assistant
→ github.com/openclaw/openc…
21. dify — AI app builder
→ github.com/langgenius/dify
22. langflow
→ github.com/langflow-ai/la…
23. mem0 — AI memory layer
→ github.com/mem0ai/mem0
24. browser-use
→ github.com/browser-use/br…
25. crewAI
→ github.com/crewAIInc/crew…
26. MetaGPT
→ github.com/geekan/MetaGPT
27. AutoGen (Microsoft)
→ github.com/microsoft/auto…
28. aider — AI coding assistant
→ github.com/Aider-AI/aider
29. markitdown (Microsoft)
→ github.com/microsoft/mark…
30. open-webui
→ github.com/open-webui/ope…
31. maigret — OSINT tool
→ github.com/soxoj/maigret
32. TradingAgents
→ github.com/TauricResearch…
33. stagehand
→ github.com/browserbase/st…
34. firecrawl
→ github.com/mendableai/fir…
35. transformers (HuggingFace)
→ github.com/huggingface/tr…
36. vLLM
→ github.com/vllm-project/v…
37. llama.cpp
→ github.com/ggml-org/llama…
38. llama_index
→ github.com/run-llama/llam…
39. nanoGPT (Karpathy)
→ github.com/karpathy/nanoG…
40. RAGFlow
→ github.com/infiniflow/rag…
41. supermemory
→ github.com/supermemoryai/…
42. awesome-claude-skills
→ github.com/ComposioHQ/awe…
43. Bumblebee (Perplexity AI security tool)
→ github.com/perplexityai/b…
44. ComfyUI
→ github.com/comfyanonymous…
45. DeepSeek (official repo varies by model release)
→ github.com/deepseek-ai
46. Lobe Chat
→ github.com/lobehub/lobe-c…
47. freeCodeCamp
→ github.com/freeCodeCamp/f…
48. system design + interview prep extras (bonus)
→ github.com/jwasham/coding…
49. AI agents ecosystem (extra)
→ github.com/langchain-ai/l…
50. automation + AI workflows (extra)
→ github.com/n8n-io/n8n
Save this before you forget it exists.
You'll come back to at least ten of these.
this is f**king insane
a free github repo by Jack Dorsey (Co-Founder of Twitter) with 14.4K stars just dropped the entire "ai-agent" framework for running businesses
here is how you set it up:
1.clone the repo
2. self-host the server : channels, search, git, automation all live there
3.add your agent to a channel like a teammate, scope its key, let the team steer it live
save and bookmark this no matter what
Rob retweeted
10 GitHub repos that make your real life easier and save you $900 a year in the process.
1. Home Assistant
The open-source smart home OS. Runs on a $50 Raspberry Pi. Controls everything without a cloud subscription.
Repo → github.com/home-assistant/co…
2. ESPHome
$5 microcontrollers become full smart home devices with one YAML file.
Repo → github.com/esphome/esphome
3. ratgdo
Turns any Chamberlain or LiftMaster garage door into a local smart door. Kills MyQ.
Repo → github.com/ratgdo/esphome-ra…
4. homebridge-unifi-protect
UniFi Protect cameras inside HomeKit with zero cloud. Kills Ring and Nest subscriptions.
Repo → github.com/hjdhjd/homebridge…
5. Zigbee2MQTT
One $20 coordinator replaces every $200 Zigbee hub. Works with over 4,000 devices.
Repo → github.com/koenkk/zigbee2mqt…
6. Valetudo
Rips the cloud out of your robot vacuum. Xiaomi, Roborock, Dreame. No data sent overseas.
Repo → github.com/Hypfer/Valetudo
7. go2rtc
Real-time streaming for every camera you own. One binary, zero dependencies.
Repo → github.com/AlexxIT/go2rtc
8. Frigate
Local AI object detection for your security cameras. Kills Nest Aware and Ring Protect.
Repo → github.com/blakeblackshear/f…
9. Tasmota
Flash any Sonoff, Tuya, or Shelly device to run locally. Removes the Chinese cloud in 60 seconds.
Repo → github.com/arendst/Tasmota
10. WLED
Runs on a $5 ESP32 and turns any LED strip into a smart light with 100+ effects. Beats a $200 Philips Hue starter kit.
Repo → github.com/Aircoookie/WLED
Rob retweeted
Web scraping will never be the same.
(100% open-source visual search at scale)
PixelRAG is a retrieval system that skips HTML parsing completely.
Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels.
Why that matters: parsing is where web RAG quietly loses information.
- A single HTML-to-text parser can drop 40%+ of a page.
- Tables, charts, and layout get flattened or thrown out.
- Swapping parsers alone can move accuracy ~10 points on the same docs.
PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA.
The repo also ships a Claude Code plugin that gives Claude eyes.
It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like.
One setup script. No MCP server, no backend.
How the pipeline works:
- Renders each document (web, PDF, image) to image tiles.
- Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots.
- Builds a FAISS index and serves a search API.
A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels.
Everything is open-source under Apache-2.0.
GitHub repo: github.com/StarTrail-org/Pix…
Talking about RAG, we wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x.
The article is quoted below.
Rob retweeted
The engineer who built Claude Code just dropped a 28-minute video on how to write prompts that actually work.
I've seen $300 courses that don't cover what he shows in the first 10 minutes.
CLAUDE.md files, memory shortcuts, parallel sessions, prompting patterns.
Watch it, then read the guide below on the Claude features 99% of users never find.
This video is larger than Cloudflare's 512 MB cache, so it can't be played through. More donations are needed to cover a larger cache. Donate
Rob retweeted
this is f*cking gold
Andrej Karpathy joined Anthropic five weeks ago.
A friend on his team just showed me the exact LOOPS.md file he actually uses.
I dropped it into my setup. The very first response was different.
Not slightly different. Completely different.
Claude stopped giving generic answers and started working exactly the way I think.
You don't talk to the model anymore. You build the system that talks to the model for you.
Bookmark it before it gets lost in your feed.
Read it now, then check the article below.
Rob retweeted
10 repositorios de GitHub tan potentes que no sé como siguen siendo GRATIS
[guarda esto compadre]
1. Maybe
Era un SaaS de finanzas personales. Cerraron. Lo open-sourced todo.
github.com/maybe-finance/may…
2. Dify
Plataforma para construir apps con cualquier LLM. Lo que los AI builders cobran $500/mes.
github.com/langgenius/dify
3. Open WebUI
Interfaz de ChatGPT para correr cualquier modelo en tu máquina. Sin suscripción. Sin enviar datos.
github.com/open-webui/open-w…
4. Continue
GitHub Copilot open source. Funciona en VS Code y JetBrains. Copilot cuesta $19/mes.
github.com/continuedev/conti…
5. Chatwoot
Intercom y Zendesk open source. Soporte al cliente en tu propio servidor. Intercom: $74/mes mínimo.
github.com/chatwoot/chatwoot
6. Docuseal
DocuSign open source. Firmas electrónicas sin límites ni coste por documento.
github.com/docuseal/docuseal
7. Metabase
Tableau open source. Business intelligence visual sin código. Tableau: $70/mes.
github.com/metabase/metabase
8. Activepieces
Zapier open source con más de 200 integraciones y sin límite de operaciones.
github.com/activepieces/acti…
9. Huly
Linear + Notion + Slack todo en un solo repositorio.
github.com/hcengineering/hul…
10. Flowise
Constructor visual de agentes de IA con drag & drop. Sin código. Sin suscripción.
github.com/FlowiseAI/Flowise
Esto no son proyectos de hobby.
Son alternativas reales al software que pagas cada mes.
Lo mejor de internet no tiene marketing.
Está en GitHub.
Rob retweeted
Learn programming by playing games:
JavaScript → javascriptquiz.com
Blockchain → cryptozombies.io
SQL → mystery.knightlab.com
Python → codedex.io
CSS → cssbattle.dev
Go → codingame.com
Java → tynker.com
Git → ohmygit.org
Rob retweeted
Tools the internet doesn't want you to find 🔍
1. Shodan
A search engine for internet-connected devices. You can find cameras, servers, and routers exposed online.
2. Archive. ph
Saves a permanent snapshot of any webpage. Useful when articles go behind paywalls.
3. Similarsites
Enter any website and instantly find dozens of similar ones. Great for discovering alternatives.
4. Mailtrack
Shows you when someone opens your email. You see the exact time it was read.
5. Hunter. io
Type in a company name and it finds employee email addresses linked to that domain.
6. Photopea
A free Photoshop that runs entirely in your browser. No download needed.
7. 12ft .io
Removes paywalls from most news articles. Just paste the link and read for free.
8. Carbon
Turns your code into beautiful shareable images. Popular among developers.
9. Explainshell
Paste any Linux command and it explains exactly what every part does.
10. Tineye
Reverse image search that shows where a photo has appeared on the internet.
11. Namecheckr
Check if a username is available across all social media platforms at once.
12. Untools
A collection of thinking frameworks and mental models to help you make better decisions.
13. BuiltWith
Shows the exact technologies, tools, and software any website is built with.
14. GeoGuessr
Drops you anywhere in the world on Google Street View. You guess the location.
15. Virustotal
Upload any file or paste any link and it scans it with over 70 antivirus engines instantly.