# Ai News Aggregator Template

> Self-hostable personal AI news feed — GitHub/Reddit/X -> Claude curation -> daily 2-bucket Telegram digest with thumbs up/down learning + an MCP server to query your archive from Claude.

- **Type:** MCP server
- **Install:** `agentstack add mcp-catu46-ai-news-aggregator-template`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [catu46](https://agentstack.voostack.com/s/catu46)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [catu46](https://github.com/catu46)
- **Source:** https://github.com/catu46/ai-news-aggregator-template

## Install

```sh
agentstack add mcp-catu46-ai-news-aggregator-template
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# 🤖 AI News Aggregator

A personal AI news and repo aggregator, **multi-tenant-ready**: it collects GitHub/Reddit/X, curates quality with **Claude Haiku 4.5**, delivers **1x/day on Telegram** in 2 buckets (📦 repos / 🗞️ news) with 👍/👎 votes — and even exposes your curated archive to Claude via **MCP**.

> **Open-source template.** Each person spins up their own copy: their bot, their database, their data. Nothing is shared between instances.

> **Not just AI.** It ships configured for AI/tech, but the **engine is topic-agnostic**. Point `config/sources.yaml` at any subreddits / X searches you want (World Cup, a niche market, a sports league…), add your own `gold`/`noise` examples to `seeds.yaml`, and rewrite the curator prompt (`src/curation/prompt.py`) to judge **your** topic's quality. Semantic search, dedup, steering and delivery work regardless of domain. *(The 📦 repos bucket is GitHub-specific — drop it or repurpose the source if your topic isn't code.)*

---

## ✨ Features

- **📦 / 🗞️ Morning digest 1x/day.** The digest arrives **once a day, at a fixed time** (`DIGEST_HOUR`/`DIGEST_TZ`, via `run_daily`) — a little morning "newspaper," not an "every-24h" blast. It comes split into **repos** (GitHub) and **news** (Reddit + X), each ranked within itself — no mixing apples and oranges. Delivery only includes posts published in the last **30 days** (`DELIVERY_MAX_AGE_DAYS`).
- **🏃 `/run` on demand.** Runs a **full cycle now** — ingest → embed → curate → deliver — without waiting for the digest time. It has a lock so two cycles don't run at once.
- **🧠 Curation via Claude Haiku 4.5.** An **APPLIED-AI** persona (tools, capabilities, techniques, useful news for someone who USES AI — enemy #1 is **AI Slop**). Each post gets a **global quality** verdict (approve/reject + category + summary + rationale) via Structured Outputs, with the rubric cached to keep it cheap. Approve categories: `ai_tools`/`ai_capabilities`/`applied_techniques`/`autonomous_agents`/`ai_industry`; rejects: `ai_slop`/`low_signal`/`research_only`/`corporate_hype`/`basic_tutorial`/`off_topic`. **Concrete substance survives a hype tone** (a number/release/event approves; rumor/leak/hot-take is `ai_slop`). The **card summary comes out in English**. Active `/focus` topics loosen the bar. A `SpendGuard` (accurate cost estimate, with the cache discount) pauses curation at the monthly cap. **Swappable provider** via `CURATOR_PROVIDER` (`anthropic` | `kimi`).
- **👍 / 👎 with PER-BUCKET affinity.** Your votes train the ranking, and affinity is **separated per bucket**: what you like in repos doesn't interfere with what shows up in news. Affinity only **ranks**, it never hides.
- **🎯 `/focus` by speech.** Say in natural language "I want more about agents" and the focus becomes **bidirectional**: it re-ranks delivery **and** injects the topic into ingestion, pulling in new content on that subject — broadening the search on **X** (Latest + Top), on **Reddit** (top of the day/week/month + hot), and on **GitHub**. And more: the **curator now listens to the focus** — active topics loosen the quality bar (approving on-topic content, including funding/VC) instead of just reordering what already exists.
- **🔎 Conversational recall & `/search`.** Chat recall distinguishes **the polarity of your question**: if you ask about what you **voted** on ("did I like something about XPTO?"), the bot searches your 👍/👎; but a **general** question ("any news about XPTO?") searches the **whole archive** — anything good that passed curation, whether you voted on it or not, with ❤️ marking what you liked. `/search` does semantic search and, since the archive is embedded in English, it **translates the query** (`translate_to_en`) before searching — so you can ask in PT-BR.
- **🎯 Two-stage search (relevance rerank).** In a single-domain archive (all AI), cosine distance alone can't separate relevant from irrelevant. So search has **two stages** (`semantic_recall`): broad vector recall → a **reranker** (Voyage `rerank-2.5`) that reads query+text together and gives the real relevance score. Off-topic is **discarded**; if nothing passes, the bot says **"I found nothing about X"** instead of filling the screen with off-topic. Applies to chat, `/search` and MCP.
- **♻️ No repeated news.** The bot won't send you the **same story twice** — even from another source or on another day (and even if you liked it). Before delivering, it runs **semantic dedup** (`_dedup_pending`) against everything already delivered; distinct stories still come through. Repos **and** news.
- **🎚️ Focus quota + digest size.** A `/focus` is a **dial, not a switch**: it occupies up to **N** of the bucket's slots ("up to 6 VC news"), the rest stays normal — so one topic never starves a platform. If you don't give a number, the bot **asks** "how many?". And you can **resize the digest** by speech ("up to 20 news a day").
- **⚖️ New × relevant rebalancing.** Adjust by speech how much of the digest is **freshness** (newer) vs **relevance** (affinity + focus) — and, beyond the manual adjustment, the bot **auto-balances** by learning from your votes (it raises novelty if you like what's discovered by the freshness slot, lowers it if you reject). It's saved in your settings. Say **"undo that" / "reset"** and the bot **zeroes out the adjustment** for that bucket (or both) and the mix returns to default. **`/mix`** shows the current new×relevant balance of each bucket (marked `default` or `adjusted`).
- **🔌 MCP server.** Plug your curated archive into Claude Code/Desktop and query it with `search_archive`, `recall_votes`, and `see_focus`.
- **🔗 Pasted link = 👍.** Paste a URL in the chat: the bot reads the content via Jina Reader and saves it to your archive already as `origin='manual'` with a positive vote.

---

## 💬 Commands

All commands respond only to the allowlist in `sources.yaml`:

| Command | What it does |
| --- | --- |
| `/start` | Welcome + registers you in the database; lists the commands. |
| `/feed` | Delivers **now** whatever is approved-and-undelivered (doesn't touch the mix). |
| `/run` | Runs a **full cycle** now: ingest → embed → curate → deliver (with a lock so two don't run at once). |
| `/search ` | Semantic search on the curated archive (translates PT→EN first; ❤️ = you liked it). |
| `/focus` | No argument: shows the active direction per bucket. `/focus clear` (or `off`/`reset`): clears it. `/focus `: steers (same path as free chat). |
| `/mix` | Shows the current **new×relevant balance** of each bucket (marked `default` or `adjusted`). |

Beyond the commands, **just talk normally** to the bot: steer the feed ("for 3 days I want repos about RAG"), ask ("any news about agents?" → searches the whole archive; "what was that news I liked?" → searches your votes), query the state ("what's in focus?", "what's the mix?"), adjust the mix ("more novelty in the news") or reset it ("undo that" / "reset") — or paste a link to save it to the archive.

---

## 🏗️ Architecture

Everything runs **inside the bot itself**: two jobs on the `JobQueue` (delivery **1x/day** at a fixed time via `run_daily` at `DIGEST_HOUR`/`DIGEST_TZ`, pipeline every **30min**) — and `/run` forces a full cycle (ingest → embed → curate → deliver) any time. **No separate cron needed.**

```
                config/sources.yaml  +  active /focus (broadens ingestion)
                                    │
        ┌───────────────────────────┼───────────────────────────┐
        ▼                           ▼                           ▼
  RedditSource                GitHubSource                  XSource
  (fixed subs +          (Search API + README,         (twitter-cli/cookies:
   top day/week/month+hot)  + focus queries)           Latest + Top of focus)
        └───────────────────────────┼───────────────────────────┘
                                    ▼
                      INGEST  →  upsert_post (dedup)
                                    │
                       SHARED POOL: posts  ◀── curated 1x (quality)
                                    │
                  ┌─────────────────┼─────────────────┐
                  ▼                                   ▼
        EMBED (Voyage voyage-4-lite)       CURATE (curator → Verdict, English summary)
        embedding IS NULL, batches of 100  verdict IS NULL, batches of 100
                  │                        ▲ LISTENS to /focus (interests loosen the bar)
                  └─────────────────┬─────────────────┘  SpendGuard pauses $$
                                    ▼
              DELIVERY IN 2 BUCKETS  (daily digest · /feed · /run)
              approved_undelivered (≤ 30 days)  →  ranks WITHIN the bucket:
                  📦 repos  = (github)
                  🗞️ news   = (reddit, twitter)
              slots split:  RELEVANCE (affinity+focus)  ×  FRESHNESS
                            │  (governed by BALANCE, auto-tuned from votes)
                                    ▼
              Telegram: cards with 👍/👎  →  on_vote writes to votes
                                    │
            ┌───────────────────────┼───────────────────────┐
            ▼                       ▼                       ▼
       AFFINITY                   FOCO                   BALANCE
   (ranks, per bucket)    (ingest + curate + rank)  (new×relevant, learns from votes)
            └───────────────────────┼───────────────────────┘
                                    ▼
                 RECALL / MCP  →  semantic_recall (broad vector recall → RERANK) · active_focus
                    (/search, chat, and the MCP server use the SAME methods)
              ▸ GENERAL question → whole archive; "liked X?" → votes
              ▸ RERANK relevance cut: off-topic is discarded → "I found nothing about X"

                       ▸ EVERYTHING scoped by user_id ◂
```

### Components

| Component | Path | Role |
| --- | --- | --- |
| **Telegram bot** | `src/bot/bot.py` | Delivery and interface (python-telegram-bot 22.8, long-polling). Locked to an allowlist from `sources.yaml`, delivers the morning digest in 2 buckets (30-day cutoff), records inline votes, commands `/start /feed /run /search /focus /mix`, saves pasted links and routes free chat to steer/recall/balance/status (general recall falls back to the whole archive; "undo that"/"reset" zeroes the mix; "what's in focus?"/"what's the mix?" query the state). Runs the `_job_deliver` (daily, fixed hour via `run_daily`, with auto-balancing) and `_job_pipeline` (30min) jobs. |
| **Pipeline runner** | `src/pipeline.py` | One `ingest → embed → curate` cycle, idempotent. Runs standalone (`python -m src.pipeline`), via the bot's job, or via `/run`. The active `/focus` topics enter ingestion (Reddit/X/GitHub) **and** curation (as *interests*). It does **not** deliver to Telegram. |
| **Reddit source** | `src/ingestion/reddit_source.py` | Collects via the **public RSS/Atom feed** of the fixed subreddits; `/focus` (news) topics broaden the search (top of day/week/month + hot). Parses with feedparser + BeautifulSoup. |
| **GitHub source** | `src/ingestion/github_source.py` | Trending repos by topic via the Search API (recent + `stars>=min`, ordered by stars) and reads the README best-effort; `/focus` (repos) topics enter as extra queries. `GITHUB_TOKEN` optional. |
| **X/Twitter source** | `src/ingestion/x_source.py` | Collects via subprocess of the `twitter` CLI (free mode via cookies): `user-posts` and `search`, both `--json`. `/focus` (news) topics broaden the search (Latest + Top). |
| **Source interface** | `src/ingestion/base.py` | `IngestionSource` ABC: every source implements `async fetch() -> list[IngestedPost]`. Dedup is the database's job. |
| **Curator (swappable)** | `src/curation/curator.py` | `make_curator(settings)` picks the provider by `CURATOR_PROVIDER` (`anthropic` → `AnthropicCurator` with Haiku 4.5; `kimi` → Moonshot/Kimi). **Global** quality verdict (Structured Outputs `Verdict`, cached rubric, English summary), with the `/focus` *interests* loosening the bar. `SpendGuard` persists spend and raises `BudgetExceeded`. |
| **Steerer (chat→intent)** | `src/curation/steering.py` | Classifies free chat into `ChatIntent` (steer/recall/balance/status/capacity/other) via Haiku. `steer` → directives for `/focus` (with an optional `quota`); `recall` → search (polarity `any` falls back to the whole archive, `liked`/`disliked` to votes); `balance` → new×relevant mix (`balance_reset` to default); `status` → QUERIES the state (focus/mix); `capacity` → resizes a bucket's per-day cap. |
| **Config / Settings** | `src/common/config.py` | Loads `.env`, `config/sources.yaml`, and `config/seeds.yaml`. `load_settings/load_sources/load_seeds`. |
| **Database (pgvector)** | `src/common/db.py` | Async access (asyncpg + pgvector, `statement_cache_size=0` for the Supabase pooler). Everything scoped by `user_id`. |
| **Data models** | `src/common/models.py` | `IngestedPost` + Pydantic schemas for the Structured Outputs (`Verdict`, `FocusItem`, `ChatIntent`). |
| **Embedder + reranker (Voyage)** | `src/common/embeddings.py` | Voyage AI wrapper: embeddings (`voyage-4-lite`, 1024-dim, L2-normalized → cosine=dot) + reranker (`rerank-2.5`, `RERANK_MODEL`) for the search's 2nd stage. |
| **Two-stage search** | `src/common/recall.py` | `semantic_recall`: broad vector recall → rerank (cut by `RERANK_MIN_SCORE`). Used by `/search`, chat and MCP. |
| **MCP server** | `src/mcp_server/server.py` | FastMCP (stdio) that exposes the archive to Claude: `search_archive`, `recall_votes`, `see_focus`. |
| **SQL schema** | `db/schema.sql` | Postgres 15+/pgvector DDL: `users`, `posts` (shared pool), `deliveries`, `votes`, `focus`. HNSW index, `updated_at` triggers. |
| **config/sources.yaml** | `config/sources.yaml` | Sources per user (multi-tenant). The bot's allowlist is derived from here. |

---

## 🚀 Step-by-step setup

### 1. Clone, create a venv, and install dependencies

```bash
git clone  ai-news-aggregator
cd ai-news-aggregator
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install -r requirements.txt
```

### 2. Database (Supabase + pgvector)

1. Create a free project on [Supabase](https://supabase.com/).
2. Copy the **POOLED connection string** (the *Connection Pooler* one, not the direct one) — it becomes your `DATABASE_URL`.
3. Apply the schema:

```bash
psql "$DATABASE_URL" -f db/schema.sql
```

> Alternative: paste the contents of `db/schema.sql` into the **SQL Editor** of the Supabase dashboard. This creates the pgvector extension, the tables, the HNSW index, and the triggers.

> ⚠️ **Starting over from scratch:** `db/reset.sql` drops all the tables that `schema.sql` creates (`focus`, `votes`, `deliveries`, `posts`, `users`) — run it **before** reapplying the schema if you need to wipe it. **It ERASES all data.** Use it only on a throwaway/freshly-created database, never on one that already has your votes.

### 3. Anthropic and Voyage keys

- **Anthropic** → `ANTHROPIC_API_KEY` (Haiku 4.5 curator + steerer). At [console.anthropic.com](https://console.anthropic.com/). The **curator is swappable** via `CURATOR_PROVIDER` (`anthropic` | `kimi`); if you're going to use `kimi`, fill in `MOONSHOT_API_KEY`/`MOONSHOT_BASE_URL`/`KIMI_MODEL` instead. The steerer stays on Anthropic.
- **Voyage AI** → `VOYAGE_API_KEY` (embeddings). At [voyageai.com](https://www.voyageai.com/). Generous free tier.

### 4. Telegram bot

1. Talk to [@BotFather](https://t.me/BotFather), `/newbot`, and copy the token → `TELEGRAM_BOT_TOKEN`.
2. Find **your** numeric `user_id`: talk to [@userinfobot](https://t.me/userinfobot) (or send `/start` to your bot, which logs the id).

### 5. (Optional) X / Twitter cookies

The X source uses `twitter-cli` in **free mode via cookies**. Use a **throwaway account** (risk of banning your main one). Extract 2 cookies from a session logged into `x.com` (DevTools or the Cookie-Editor extension):

| Cookie | Varia

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [catu46](https://github.com/catu46)
- **Source:** [catu46/ai-news-aggregator-template](https://github.com/catu46/ai-news-aggregator-template)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** yes
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-catu46-ai-news-aggregator-template
- Seller: https://agentstack.voostack.com/s/catu46
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.