@rharsh
Seattle
Joined November 2008
10 GITHUB REPOS TO BUILD SERIOUS WEB SCRAPING WORKFLOWS Need to collect, parse, or transform web data? These open-source projects are worth knowing: 1. Crawlee Build scalable crawlers with browser automation, retries, proxies, and queues. github.com/apify/crawlee 2. Scrapy A powerful framework for building fast, scalable web crawlers. github.com/scrapy/scrapy 3. Playwright Automate real browsers and handle JavaScript-heavy websites. github.com/microsoft/playwri… 4. Beautiful Soup A simple way to parse HTML and extract useful data. github.com/wention/Beautiful… 5. Firecrawl Turn websites into clean, AI-ready content. github.com/mendableai/firecr… 6. Crawl4AI Convert websites into structured data for LLM and RAG workflows. github.com/unclecode/crawl4a… 7. ScrapeGraphAI Use natural language to define what data you want to extract. github.com/ScrapeGraphAI/Scr… 8. Colly A fast and lightweight web scraping framework for Go. github.com/gocolly/colly 9. trafilatura Extract clean text and metadata from web pages. github.com/adbar/trafilatura 10. Newspaper4k Extract article content, metadata, and related information automatically. github.com/AndyTheFactory/ne… All open source. The web is full of useful data. These repos help you turn it into something you can actually use. Save this list.
15
20
74
4,323
10 GITHUB REPOSITORIES YOU’LL WISH YOU FOUND EARLIER I went looking for projects that don’t show up in every AI thread. Found some seriously good ones. 1. Vero AI coding agent that verifies generated code using Lean 4. github.com/sunblaze-ucb/vero 2. Agentor Adds agent discovery, A2A support, streaming and artifact sharing. github.com/CelestoAI/agentor 3. Sky-Agent Lightweight framework for building AI agents with tools, memory and files. github.com/arteemg/sky-agent 4. PageIndex Reasoning-based RAG for navigating long documents. github.com/VectifyAI/PageInd… 5. Semantica Graph-based infrastructure for connecting entities, relationships and context. github.com/HKUDS/semantica 6. NanoBot Lightweight self-hosted AI agent with memory, tools, MCP and multi-agent workflows. github.com/HKUDS/nanobot 7. Learn Claude Code Build a Claude Code-style agent from scratch and understand how coding agents work. github.com/shareAI-lab/learn… 8. Agent-Reach Gives AI agents access to GitHub, YouTube, Reddit, X and more. github.com/Panniantong/Agent… 9. Book-to-Skill Turns technical books into usable Claude Code skills. github.com/virgiliojr94/book… 10. ExploitGym Environment for testing AI agents on real software exploitation tasks. github.com/sunblaze-ucb/expl… All open source. Some are getting attention. Others are still flying under the radar. Save this list.
24
27
122
7,750
10 GITHUB REPOS THAT MAKE AI FEEL VERY DIFFERENT Some of these are seriously interesting: 1. RuView Turns Wi-Fi signals into spatial intelligence and presence detection. github.com/ruvnet/RuView 2. ASC Fast Android decompiler built for AI agents and mobile research. github.com/MG1937/ASC 3. VoiceStudio Local voice AI studio for cloning, dubbing, transcription and audiobooks. github.com/debpalash/VoiceSt… 4. MiroFish Multi-agent simulation engine for exploring complex scenarios. github.com/666ghj/MiroFish 5. OpenResearch Turns coding agents into research agents for hypotheses and experiments. github.com/alphaXiv/OpenRese… 6. Atlas Git-style source control for managing multiple AI coding agents. github.com/pacifio/atlas 7. PI Compact toolkit for building and running AI agents. github.com/earendil-works/pi 8. TradingAgents Multi-agent framework for financial trading research. github.com/TauricResearch/Tr… 9. BrewUI A native macOS GUI for Homebrew. github.com/Homebrew/brew 10. tinycast Turns your machine into a simple personal casting setup. github.com/abue-ammar/tinyca… Save this list.
17
31
172
8,954
10 GITHUB REPOS EVERY DEVELOPER NEEDS 1. LangChain Framework for building LLM-powered applications. → github.com/langchain-ai/lang… 2. n8n Build powerful workflow automations. → github.com/n8n-io/n8n 3. OpenClaw Run a local AI assistant. → github.com/openclaw/openclaw 4. Dify Build AI applications with a visual workflow. → github.com/langgenius/dify 5. Langflow Build and prototype AI workflows visually. → github.com/langflow-ai/langf… 6. Mem0 Add persistent memory to AI applications. → github.com/mem0ai/mem0 7. Browser Use Let AI agents interact with websites. → github.com/browser-use/brows… 8. CrewAI Build and orchestrate multi-agent AI systems. → github.com/crewAIInc/crewAI 9. MetaGPT Build software workflows using multiple AI agents. → github.com/geekan/MetaGPT 10. AutoGen Microsoft's framework for building multi-agent AI applications. → github.com/microsoft/autogen Bookmark these repos.
5
32
2
110
5,556
10 GITHUB REPOS THAT SHOULD BE ILLEGAL TO HAVE. all free. all open-source. bookmark this for later. 1. Recordly — the free screen studio. Open-source screen recorder with auto-zoom, smooth cursor, webcam overlay, styled backgrounds, and polished demos without a video editor. (AGPL-3.0) github.com/webadderallorg/Re… 2. Stirling PDF — your entire PDF toolkit. Self-hosted. Merge, split, sign, redact, OCR, convert, compress, 50+ tools. Runs locally, so nothing leaves your machine. (MIT) github.com/Stirling-Tools/St… 3. PhotoGIMP — turns GIMP into Photoshop. Photoshop shortcuts, layout, and splash screen patched on top of GIMP 3+. github.com/Diolinux/PhotoGIM… 4. Open Notebook — self-hosted NotebookLM. Drop in PDFs, URLs, and YouTube links. Chat with them, summarize them, and even generate podcasts. Supports 18+ AI providers. (MIT) github.com/lfnovo/open-noteb… 5. Odysseus — PewDiePie's self-hosted AI workspace. Chat, agents, deep research, docs, email, memory, and more. Local-first, your hardware, your data. (MIT) github.com/odysseus-dev/odys… 6. FreeDomain — free domain names for everyone. Register a domain, point it to Cloudflare or any DNS provider, and launch your site without paying for the domain. (AGPL-3.0) github.com/DigitalPlatDev/Fr… 7. HyperFrames — write HTML, render videos. HeyGen's open-source engine that converts HTML/CSS animations into deterministic MP4 videos. Built for AI agents. (Apache-2.0) github.com/heygen-com/hyperf… 8. Web-to-App — turn any website into an Android app. Configurable WebView, APK signing, and even Node.js, PHP, and Python runtimes. No remote build required. github.com/shiaho777/web-to-… 9. Reclip — self-hosted video & audio downloader. Download videos from YouTube, TikTok, X, Instagram, and 1000+ other sites as MP4 or MP3 using yt-dlp. github.com/averygan/reclip 10. Excalidraw — the infinite whiteboard. Replace Miro, FigJam, and Lucidchart with hand-drawn diagrams, wireframes, real-time collaboration, and end-to-end encryption. 120k+ GitHub stars. (MIT) github.com/excalidraw/excali… Which repo are you trying first?
24
310
5
2,011
134,153
Introducing Magnitude: your actually local agent 100% private and offline. No token costs, no API keys. Open source. Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI. Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer. It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit. Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome. Use it for everyday work: - Analyze sensitive data - Manage private notes - Review code and logs - Search and organize files - Build docs or slides npm i -g @magnitudedev/cli GitHub: github.com/magnitudedev/magn…
130
263
47
2,305
397,701
Microsoft open sourced an inference framework that runs a 100B parameter LLM on a single CPU. It's called BitNet. And it does what was supposed to be impossible. No GPU. No cloud. No $10K hardware setup. Just your laptop running a 100-billion parameter model at human reading speed. Here's how it works: Every other LLM stores weights in 32-bit or 16-bit floats. BitNet uses 1.58 bits. Weights are ternary just -1, 0, or +1. That's it. No floats. No expensive matrix math. Pure integer operations your CPU was already built for. The result: - 100B model runs on a single CPU at 5-7 tokens/second - 2.37x to 6.17x faster than llama.cpp on x86 - 82% lower energy consumption on x86 CPUs - 1.37x to 5.07x speedup on ARM (your MacBook) - Memory drops by 16-32x vs full-precision models The wildest part: Accuracy barely moves. BitNet b1.58 2B4T their flagship model was trained on 4 trillion tokens and benchmarks competitively against full-precision models of the same size. The quantization isn't destroying quality. It's just removing the bloat. What this actually means: - Run AI completely offline. Your data never leaves your machine - Deploy LLMs on phones, IoT devices, edge hardware - No more cloud API bills for inference - AI in regions with no reliable internet The model supports ARM and x86. Works on your MacBook, your Linux box, your Windows machine. 27.4K GitHub stars. 2.2K forks. Built by Microsoft Research. 100% Open Source. MIT License.
20
75
7
447
49,436
this is f**king insane. Jack Dorsey just open sourced a free tool for running a one-person business. the GitHub repo already has 14.4K stars. It lets you: → run your own server → keep chat, search, Git, and automation in one place → add AI agents that work like teammates → decide exactly what each agent can do no monthly subscriptions. just you, your AI agents, and full control. this is where one-person businesses are headed. save this for later.
106
405
20
3,753
711,535
10 GitHub repos that make Excel and Google Sheets look ancient in 2026. Excel was born in 1985. Google Sheets in 2006. These 10 repos were built for how people actually work with data now. Every one is free. Every one is open source. 1. Datasette Built by Simon Willison, the co-creator of Django. Turns any SQLite file into a browsable, queryable website in one command. 11,000+ stars. Repo → github.com/simonw/datasette 2. Baserow The MIT-licensed Airtable clone. Its AI assistant Kuma builds full database structures from plain English. Repo → github.com/bram2w/baserow 3. Grist Real Python formulas inside every cell. No more nested IFERROR VLOOKUP hell. Repo → github.com/gristlabs/grist-c… 4. Teable Handles millions of rows without lag. Faster than Airtable or Sheets at scale. Repo → github.com/teableio/teable 5. DuckDB Query a 100 GB CSV in your browser tab. The reason data engineers stopped opening spreadsheets in 2025. Repo → github.com/duckdb/duckdb 6. Rowy An Airtable-style view over your Firebase Firestore. No-code and full-code together. Repo → github.com/rowyio/rowy 7. Metabase Points at any database and builds you a chart in 30 seconds. 42,000+ stars. Kills $900 a year Tableau seats. Repo → github.com/metabase/metabase 8. Observable Framework Built by Mike Bostock, the creator of D3. Live interactive dashboards Sheets cannot touch. Repo → github.com/observablehq/fram… 9. Evidence Markdown plus SQL equals live dashboards. Version-controlled in Git. Repo → github.com/evidence-dev/evid… 10. Actual Budget Envelope budgeting done right. No cloud. All local. Kills YNAB at $109 a year. Repo → github.com/actualbudget/actu…
12
40
179
13,743
10 GitHub Repositories That Are Making Expensive Software Hard to Justify ($50B+ Worth of Paid Software Has Free Open-Source Alternatives) Bookmark this. 1. yt-dlp Download videos from YouTube, X, TikTok, Instagram, and hundreds of other platforms. Even YouTube Premium doesn't offer this level of flexibility. Repo → github.com/yt-dlp/yt-dlp 2. Ollama Run powerful AI models locally on your laptop. No API bills. No cloud dependency. Repo → github.com/ollama/ollama 3. Fooocus Generate stunning AI images on your own GPU. A powerful free alternative to many paid image generators. Repo → github.com/lllyasviel/Fooocu… 4. Whisper OpenAI's open-source speech-to-text model. Fast, accurate, and supports 99+ languages. Repo → github.com/openai/whisper 5. Plausible Analytics A privacy-first, open-source alternative to Google Analytics. Self-host it and own your data. Repo → github.com/plausible/analyti… 6. AppFlowy An open-source alternative to Notion. Host it yourself with unlimited users. Repo → github.com/AppFlowy-IO/AppFl… 7. Penpot A free, open-source alternative to Figma. Built for designers and developers. Repo → github.com/penpot/penpot 8. n8n An open-source automation platform. Build unlimited workflows without paying Zapier-style subscription fees. Repo → github.com/n8n-io/n8n 9. Cal.com The open-source scheduling platform. Perfect if you want full control over your booking system. Repo → github.com/calcom/cal.com 10. Bitwarden A trusted open-source password manager. Secure, cross-platform, and free to get started. Repo → github.com/bitwarden/server These projects prove that some of the best software on the internet doesn't come with a monthly subscription. Save this thread. You'll thank yourself the next time you're about to pay for a tool that already has an incredible open-source alternative. 100% Free. 100% Open Source.
3
31
2,711
50 USEFUL GITHUB REPOS 1. iFixAi — AI misalignment testing → github.com/ifixai-ai/iFix 2. public-apis — free APIs for everything → github.com/public-apis/pu… 3. build-your-own-x — learn by building → github.com/codecrafters-i… 4. developer-roadmap — learn any tech skill → github.com/kamranahmedse/… 5. free-programming-books — thousands of free books → github.com/EbookFoundatio… 6. system-design-primer → github.com/donnemartin/sy… 7. coding-interview-university → github.com/jwasham/coding… 8. the-art-of-command-line → github.com/jlevy/the-art-… 9. project-based-learning → github.com/practical-tuto… 10. you-dont-know-js → github.com/getify/You-Don… 11. the-book-of-secret-knowledge → github.com/trimstray/the-… 12. tech-interview-handbook → github.com/yangshun/tech-… 13. awesome-selfhosted → github.com/awesome-selfho… 14. javascript-algorithms → github.com/trekhleb/javas… 15. 30-seconds-of-code → github.com/Chalarangelo/3… 16. github gitignore templates → github.com/github/gitigno… 17. ollama — run LLMs locally → github.com/ollama/ollama 18. langchain → github.com/langchain-ai/l… 19. n8n automation → github.com/n8n-io/n8n 20. openclaw — local AI assistant → github.com/openclaw/openc… 21. dify — AI app builder → github.com/langgenius/dify 22. langflow → github.com/langflow-ai/la… 23. mem0 — AI memory layer → github.com/mem0ai/mem0 24. browser-use → github.com/browser-use/br… 25. crewAI → github.com/crewAIInc/crew… 26. MetaGPT → github.com/geekan/MetaGPT 27. AutoGen (Microsoft) → github.com/microsoft/auto… 28. aider — AI coding assistant → github.com/Aider-AI/aider 29. markitdown (Microsoft) → github.com/microsoft/mark… 30. open-webui → github.com/open-webui/ope… 31. maigret — OSINT tool → github.com/soxoj/maigret 32. TradingAgents → github.com/TauricResearch… 33. stagehand → github.com/browserbase/st… 34. firecrawl → github.com/mendableai/fir… 35. transformers (HuggingFace) → github.com/huggingface/tr… 36. vLLM → github.com/vllm-project/v… 37. llama.cpp → github.com/ggml-org/llama… 38. llama_index → github.com/run-llama/llam… 39. nanoGPT (Karpathy) → github.com/karpathy/nanoG… 40. RAGFlow → github.com/infiniflow/rag… 41. supermemory → github.com/supermemoryai/… 42. awesome-claude-skills → github.com/ComposioHQ/awe… 43. Bumblebee (Perplexity AI security tool) → github.com/perplexityai/b… 44. ComfyUI → github.com/comfyanonymous… 45. DeepSeek (official repo varies by model release) → github.com/deepseek-ai 46. Lobe Chat → github.com/lobehub/lobe-c… 47. freeCodeCamp → github.com/freeCodeCamp/f… 48. system design + interview prep extras (bonus) → github.com/jwasham/coding… 49. AI agents ecosystem (extra) → github.com/langchain-ai/l… 50. automation + AI workflows (extra) → github.com/n8n-io/n8n Save this before you forget it exists. You'll come back to at least ten of these.
19
72
1
312
14,859
Rob retweeted
this is f**king insane a free github repo by Jack Dorsey (Co-Founder of Twitter) with 14.4K stars just dropped the entire "ai-agent" framework for running businesses here is how you set it up: 1.clone the repo 2. self-host the server : channels, search, git, automation all live there 3.add your agent to a channel like a teammate, scope its key, let the team steer it live save and bookmark this no matter what
95
400
9
3,288
506,280
10 GitHub repos that make your real life easier and save you $900 a year in the process. 1. Home Assistant The open-source smart home OS. Runs on a $50 Raspberry Pi. Controls everything without a cloud subscription. Repo → github.com/home-assistant/co… 2. ESPHome $5 microcontrollers become full smart home devices with one YAML file. Repo → github.com/esphome/esphome 3. ratgdo Turns any Chamberlain or LiftMaster garage door into a local smart door. Kills MyQ. Repo → github.com/ratgdo/esphome-ra… 4. homebridge-unifi-protect UniFi Protect cameras inside HomeKit with zero cloud. Kills Ring and Nest subscriptions. Repo → github.com/hjdhjd/homebridge… 5. Zigbee2MQTT One $20 coordinator replaces every $200 Zigbee hub. Works with over 4,000 devices. Repo → github.com/koenkk/zigbee2mqt… 6. Valetudo Rips the cloud out of your robot vacuum. Xiaomi, Roborock, Dreame. No data sent overseas. Repo → github.com/Hypfer/Valetudo 7. go2rtc Real-time streaming for every camera you own. One binary, zero dependencies. Repo → github.com/AlexxIT/go2rtc 8. Frigate Local AI object detection for your security cameras. Kills Nest Aware and Ring Protect. Repo → github.com/blakeblackshear/f… 9. Tasmota Flash any Sonoff, Tuya, or Shelly device to run locally. Removes the Chinese cloud in 60 seconds. Repo → github.com/arendst/Tasmota 10. WLED Runs on a $5 ESP32 and turns any LED strip into a smart light with 100+ effects. Beats a $200 Philips Hue starter kit. Repo → github.com/Aircoookie/WLED
5
35
176
13,874
Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. GitHub repo: github.com/StarTrail-org/Pix… Talking about RAG, we wrote an article on a new approach that makes retrieval much more efficient by cutting corpus size by 40x, reducing tokens per query by 3x, and improving vector search relevance by 2.3x. The article is quoted below.
6
76
4
535
46,071
The engineer who built Claude Code just dropped a 28-minute video on how to write prompts that actually work. I've seen $300 courses that don't cover what he shows in the first 10 minutes. CLAUDE.md files, memory shortcuts, parallel sessions, prompting patterns. Watch it, then read the guide below on the Claude features 99% of users never find.
66
691
21
3,779
753,802
this is f*cking gold Andrej Karpathy joined Anthropic five weeks ago. A friend on his team just showed me the exact LOOPS.md file he actually uses. I dropped it into my setup. The very first response was different. Not slightly different. Completely different. Claude stopped giving generic answers and started working exactly the way I think. You don't talk to the model anymore. You build the system that talks to the model for you. Bookmark it before it gets lost in your feed. Read it now, then check the article below.
65
520
9
3,429
640,096
10 repositorios de GitHub tan potentes que no sé como siguen siendo GRATIS [guarda esto compadre] 1. Maybe Era un SaaS de finanzas personales. Cerraron. Lo open-sourced todo. github.com/maybe-finance/may… 2. Dify Plataforma para construir apps con cualquier LLM. Lo que los AI builders cobran $500/mes. github.com/langgenius/dify 3. Open WebUI Interfaz de ChatGPT para correr cualquier modelo en tu máquina. Sin suscripción. Sin enviar datos. github.com/open-webui/open-w… 4. Continue GitHub Copilot open source. Funciona en VS Code y JetBrains. Copilot cuesta $19/mes. github.com/continuedev/conti… 5. Chatwoot Intercom y Zendesk open source. Soporte al cliente en tu propio servidor. Intercom: $74/mes mínimo. github.com/chatwoot/chatwoot 6. Docuseal DocuSign open source. Firmas electrónicas sin límites ni coste por documento. github.com/docuseal/docuseal 7. Metabase Tableau open source. Business intelligence visual sin código. Tableau: $70/mes. github.com/metabase/metabase 8. Activepieces Zapier open source con más de 200 integraciones y sin límite de operaciones. github.com/activepieces/acti… 9. Huly Linear + Notion + Slack todo en un solo repositorio. github.com/hcengineering/hul… 10. Flowise Constructor visual de agentes de IA con drag & drop. Sin código. Sin suscripción. github.com/FlowiseAI/Flowise Esto no son proyectos de hobby. Son alternativas reales al software que pagas cada mes. Lo mejor de internet no tiene marketing. Está en GitHub.
9
171
2
952
53,958
Learn programming by playing games: JavaScript → javascriptquiz.​com Blockchain → cryptozombies.​io SQL → mystery.knightlab.​com Python → codedex.​io CSS → cssbattle.​dev Go → codingame.​com Java → tynker.​com Git → ohmygit.​org
8
33
1
138
8,805
Rob retweeted
Your AI coding agent now knows Clerk. One command. Every auth pattern. npx skills add clerk/skills

395
65
5,207
29,587,418
Tools the internet doesn't want you to find 🔍 1. Shodan A search engine for internet-connected devices. You can find cameras, servers, and routers exposed online. 2. Archive. ph Saves a permanent snapshot of any webpage. Useful when articles go behind paywalls. 3. Similarsites Enter any website and instantly find dozens of similar ones. Great for discovering alternatives. 4. Mailtrack Shows you when someone opens your email. You see the exact time it was read. 5. Hunter. io Type in a company name and it finds employee email addresses linked to that domain. 6. Photopea A free Photoshop that runs entirely in your browser. No download needed. 7. 12ft .io Removes paywalls from most news articles. Just paste the link and read for free. 8. Carbon Turns your code into beautiful shareable images. Popular among developers. 9. Explainshell Paste any Linux command and it explains exactly what every part does. 10. Tineye Reverse image search that shows where a photo has appeared on the internet. 11. Namecheckr Check if a username is available across all social media platforms at once. 12. Untools A collection of thinking frameworks and mental models to help you make better decisions. 13. BuiltWith Shows the exact technologies, tools, and software any website is built with. 14. GeoGuessr Drops you anywhere in the world on Google Street View. You guess the location. 15. Virustotal Upload any file or paste any link and it scans it with over 70 antivirus engines instantly.
49
1,551
24
6,969
344,628