Ai News Aggregator Template

🤖 AI News Aggregator

A personal AI news and repo aggregator, multi-tenant-ready: it collects GitHub/Reddit/X, curates quality with Claude Haiku 4.5, delivers 1x/day on Telegram in 2 buckets (📦 repos / 🗞️ news) with 👍/👎 votes — and even exposes your curated archive to Claude via MCP.

> Open-source template. Each person spins up their own copy: their bot, their database, their data. Nothing is shared between instances.

> Not just AI. It ships configured for AI/tech, but the engine is topic-agnostic. Point config/sources.yaml at any subreddits / X searches you want (World Cup, a niche market, a sports league…), add your own gold/noise examples to seeds.yaml, and rewrite the curator prompt (src/curation/prompt.py) to judge your topic's quality. Semantic search, dedup, steering and delivery work regardless of domain. (The 📦 repos bucket is GitHub-specific — drop it or repurpose the source if your topic isn't code.)

---

✨ Features

• 📦 / 🗞️ Morning digest 1x/day. The digest arrives once a day, at a fixed time (DIGEST_HOUR/DIGEST_TZ, via run_daily) — a little morning "newspaper," not an "every-24h" blast. It comes split into repos (GitHub) and news (Reddit + X), each ranked within itself — no mixing apples and oranges. Delivery only includes posts published in the last 30 days (DELIVERY_MAX_AGE_DAYS).
• 🏃 /run on demand. Runs a full cycle now — ingest → embed → curate → deliver — without waiting for the digest time. It has a lock so two cycles don't run at once.
• 🧠 Curation via Claude Haiku 4.5. An APPLIED-AI persona (tools, capabilities, techniques, useful news for someone who USES AI — enemy #1 is AI Slop). Each post gets a global quality verdict (approve/reject + category + summary + rationale) via Structured Outputs, with the rubric cached to keep it cheap. Approve categories: ai_tools/ai_capabilities/applied_techniques/autonomous_agents/ai_industry; rejects: ai_slop/low_signal/research_only/corporate_hype/basic_tutorial/off_topic. Concrete substance survives a hype tone (a number/release/event approves; rumor/leak/hot-take is ai_slop). The card summary comes out in English. Active /focus topics loosen the bar. A SpendGuard (accurate cost estimate, with the cache discount) pauses curation at the monthly cap. Swappable provider via CURATOR_PROVIDER (anthropic | kimi).
• 👍 / 👎 with PER-BUCKET affinity. Your votes train the ranking, and affinity is separated per bucket: what you like in repos doesn't interfere with what shows up in news. Affinity only ranks, it never hides.
• 🎯 /focus by speech. Say in natural language "I want more about agents" and the focus becomes bidirectional: it re-ranks delivery and injects the topic into ingestion, pulling in new content on that subject — broadening the search on X (Latest + Top), on Reddit (top of the day/week/month + hot), and on GitHub. And more: the curator now listens to the focus — active topics loosen the quality bar (approving on-topic content, including funding/VC) instead of just reordering what already exists.
• 🔎 Conversational recall & /search. Chat recall distinguishes the polarity of your question: if you ask about what you voted on ("did I like something about XPTO?"), the bot searches your 👍/👎; but a general question ("any news about XPTO?") searches the whole archive — anything good that passed curation, whether you voted on it or not, with ❤️ marking what you liked. /search does semantic search and, since the archive is embedded in English, it translates the query (translate_to_en) before searching — so you can ask in PT-BR.
• 🎯 Two-stage search (relevance rerank). In a single-domain archive (all AI), cosine distance alone can't separate relevant from irrelevant. So search has two stages (semantic_recall): broad vector recall → a reranker (Voyage rerank-2.5) that reads query+text together and gives the real relevance score. Off-topic is discarded; if nothing passes, the bot says "I found nothing about X" instead of filling the screen with off-topic. Applies to chat, /search and MCP.
• ♻️ No repeated news. The bot won't send you the same story twice — even from another source or on another day (and even if you liked it). Before delivering, it runs semantic dedup (_dedup_pending) against everything already delivered; distinct stories still come through. Repos and news.
• 🎚️ Focus quota + digest size. A /focus is a dial, not a switch: it occupies up to N of the bucket's slots ("up to 6 VC news"), the rest stays normal — so one topic never starves a platform. If you don't give a number, the bot asks "how many?". And you can resize the digest by speech ("up to 20 news a day").
• ⚖️ New × relevant rebalancing. Adjust by speech how much of the digest is freshness (newer) vs relevance (affinity + focus) — and, beyond the manual adjustment, the bot auto-balances by learning from your votes (it raises novelty if you like what's discovered by the freshness slot, lowers it if you reject). It's saved in your settings. Say "undo that" / "reset" and the bot zeroes out the adjustment for that bucket (or both) and the mix returns to default. /mix shows the current new×relevant balance of each bucket (marked default or adjusted).
• 🔌 MCP server. Plug your curated archive into Claude Code/Desktop and query it with search_archive, recall_votes, and see_focus.
• 🔗 Pasted link = 👍. Paste a URL in the chat: the bot reads the content via Jina Reader and saves it to your archive already as origin='manual' with a positive vote.

---

💬 Commands

All commands respond only to the allowlist in sources.yaml:

| Command | What it does | | --- | --- | | /start | Welcome + registers you in the database; lists the commands. | | /feed | Delivers now whatever is approved-and-undelivered (doesn't touch the mix). | | /run | Runs a full cycle now: ingest → embed → curate → deliver (with a lock so two don't run at once). | | /search | Semantic search on the curated archive (translates PT→EN first; ❤️ = you liked it). | | /focus | No argument: shows the active direction per bucket. /focus clear (or off/reset): clears it. /focus : steers (same path as free chat). | | /mix | Shows the current new×relevant balance of each bucket (marked default or adjusted). |

Beyond the commands, just talk normally to the bot: steer the feed ("for 3 days I want repos about RAG"), ask ("any news about agents?" → searches the whole archive; "what was that news I liked?" → searches your votes), query the state ("what's in focus?", "what's the mix?"), adjust the mix ("more novelty in the news") or reset it ("undo that" / "reset") — or paste a link to save it to the archive.

---

🏗️ Architecture

Everything runs inside the bot itself: two jobs on the JobQueue (delivery 1x/day at a fixed time via run_daily at DIGEST_HOUR/DIGEST_TZ, pipeline every 30min) — and /run forces a full cycle (ingest → embed → curate → deliver) any time. No separate cron needed.

                config/sources.yaml  +  active /focus (broadens ingestion)
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
  RedditSource                GitHubSource                  XSource
  (fixed subs +          (Search API + README,         (twitter-cli/cookies:
   top day/week/month+hot)  + focus queries)           Latest + Top of focus)
        └───────────────────────────┼───────────────────────────┘
                                    ▼
                      INGEST  →  upsert_post (dedup)
                                    │
                       SHARED POOL: posts  ◀── curated 1x (quality)
                                    │
                  ┌─────────────────┼─────────────────┐
                  ▼                                   ▼
        EMBED (Voyage voyage-4-lite)       CURATE (curator → Verdict, English summary)
        embedding IS NULL, batches of 100  verdict IS NULL, batches of 100
                  │                        ▲ LISTENS to /focus (interests loosen the bar)
                  └─────────────────┬─────────────────┘  SpendGuard pauses $$
                                    ▼
              DELIVERY IN 2 BUCKETS  (daily digest · /feed · /run)
              approved_undelivered (≤ 30 days)  →  ranks WITHIN the bucket:
                  📦 repos  = (github)
                  🗞️ news   = (reddit, twitter)
              slots split:  RELEVANCE (affinity+focus)  ×  FRESHNESS
                            │  (governed by BALANCE, auto-tuned from votes)
                                    ▼
              Telegram: cards with 👍/👎  →  on_vote writes to votes
                                    │
            ┌───────────────────────┼───────────────────────┐
            ▼                       ▼                       ▼
       AFFINITY                   FOCO                   BALANCE
   (ranks, per bucket)    (ingest + curate + rank)  (new×relevant, learns from votes)
            └───────────────────────┼───────────────────────┘
                                    ▼
                 RECALL / MCP  →  semantic_recall (broad vector recall → RERANK) · active_focus
                    (/search, chat, and the MCP server use the SAME methods)
              ▸ GENERAL question → whole archive; "liked X?" → votes
              ▸ RERANK relevance cut: off-topic is discarded → "I found nothing about X"

                       ▸ EVERYTHING scoped by user_id ◂

Components

| Component | Path | Role | | --- | --- | --- | | Telegram bot | src/bot/bot.py | Delivery and interface (python-telegram-bot 22.8, long-polling). Locked to an allowlist from sources.yaml, delivers the morning digest in 2 buckets (30-day cutoff), records inline votes, commands /start /feed /run /search /focus /mix, saves pasted links and routes free chat to steer/recall/balance/status (general recall falls back to the whole archive; "undo that"/"reset" zeroes the mix; "what's in focus?"/"what's the mix?" query the state). Runs the _job_deliver (daily, fixed hour via run_daily, with auto-balancing) and _job_pipeline (30min) jobs. | | Pipeline runner | src/pipeline.py | One ingest → embed → curate cycle, idempotent. Runs standalone (python -m src.pipeline), via the bot's job, or via /run. The active /focus topics enter ingestion (Reddit/X/GitHub) and curation (as interests). It does not deliver to Telegram. | | Reddit source | src/ingestion/reddit_source.py | Collects via the public RSS/Atom feed of the fixed subreddits; /focus (news) topics broaden the search (top of day/week/month + hot). Parses with feedparser + BeautifulSoup. | | GitHub source | src/ingestion/github_source.py | Trending repos by topic via the Search API (recent + stars>=min, ordered by stars) and reads the README best-effort; /focus (repos) topics enter as extra queries. GITHUB_TOKEN optional. | | X/Twitter source | src/ingestion/x_source.py | Collects via subprocess of the twitter CLI (free mode via cookies): user-posts and search, both --json. /focus (news) topics broaden the search (Latest + Top). | | Source interface | src/ingestion/base.py | IngestionSource ABC: every source implements async fetch() -> list[IngestedPost]. Dedup is the database's job. | | Curator (swappable) | src/curation/curator.py | make_curator(settings) picks the provider by CURATOR_PROVIDER (anthropic → AnthropicCurator with Haiku 4.5; kimi → Moonshot/Kimi). Global quality verdict (Structured Outputs Verdict, cached rubric, English summary), with the /focus interests loosening the bar. SpendGuard persists spend and raises BudgetExceeded. | | Steerer (chat→intent) | src/curation/steering.py | Classifies free chat into ChatIntent (steer/recall/balance/status/capacity/other) via Haiku. steer → directives for /focus (with an optional quota); recall → search (polarity any falls back to the whole archive, liked/disliked to votes); balance → new×relevant mix (balance_reset to default); status → QUERIES the state (focus/mix); capacity → resizes a bucket's per-day cap. | | Config / Settings | src/common/config.py | Loads .env, config/sources.yaml, and config/seeds.yaml. load_settings/load_sources/load_seeds. | | Database (pgvector) | src/common/db.py | Async access (asyncpg + pgvector, statement_cache_size=0 for the Supabase pooler). Everything scoped by user_id. | | Data models | src/common/models.py | IngestedPost + Pydantic schemas for the Structured Outputs (Verdict, FocusItem, ChatIntent). | | Embedder + reranker (Voyage) | src/common/embeddings.py | Voyage AI wrapper: embeddings (voyage-4-lite, 1024-dim, L2-normalized → cosine=dot) + reranker (rerank-2.5, RERANK_MODEL) for the search's 2nd stage. | | Two-stage search | src/common/recall.py | semantic_recall: broad vector recall → rerank (cut by RERANK_MIN_SCORE). Used by /search, chat and MCP. | | MCP server | src/mcp_server/server.py | FastMCP (stdio) that exposes the archive to Claude: search_archive, recall_votes, see_focus. | | SQL schema | db/schema.sql | Postgres 15+/pgvector DDL: users, posts (shared pool), deliveries, votes, focus. HNSW index, updated_at triggers. | | config/sources.yaml | config/sources.yaml | Sources per user (multi-tenant). The bot's allowlist is derived from here. |

---

🚀 Step-by-step setup

1. Clone, create a venv, and install dependencies

git clone  ai-news-aggregator
cd ai-news-aggregator
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt

2. Database (Supabase + pgvector)

1. Create a free project on Supabase.
2. Copy the POOLED connection string (the Connection Pooler one, not the direct one) — it becomes your DATABASE_URL.
3. Apply the schema:

psql "$DATABASE_URL" -f db/schema.sql

> Alternative: paste the contents of db/schema.sql into the SQL Editor of the Supabase dashboard. This creates the pgvector extension, the tables, the HNSW index, and the triggers.

> ⚠️ Starting over from scratch: db/reset.sql drops all the tables that schema.sql creates (focus, votes, deliveries, posts, users) — run it before reapplying the schema if you need to wipe it. It ERASES all data. Use it only on a throwaway/freshly-created database, never on one that already has your votes.

3. Anthropic and Voyage keys

• Anthropic → ANTHROPIC_API_KEY (Haiku 4.5 curator + steerer). At console.anthropic.com. The curator is swappable via CURATOR_PROVIDER (anthropic | kimi); if you're going to use kimi, fill in MOONSHOT_API_KEY/MOONSHOT_BASE_URL/KIMI_MODEL instead. The steerer stays on Anthropic.
• Voyage AI → VOYAGE_API_KEY (embeddings). At voyageai.com. Generous free tier.

4. Telegram bot

1. Talk to @BotFather, /newbot, and copy the token → TELEGRAM_BOT_TOKEN.
2. Find your numeric user_id: talk to @userinfobot (or send /start to your bot, which logs the id).

5. (Optional) X / Twitter cookies

The X source uses twitter-cli in free mode via cookies. Use a throwaway account (risk of banning your main one). Extract 2 cookies from a session logged into x.com (DevTools or the Cookie-Editor extension):

| Cookie | Varia

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: catu46
• Source: catu46/ai-news-aggregator-template
• License: MIT

Install and usage instructions live in the source repository linked above.