Collegeum Pipeline System Architecture

Ingestion sources, deterministic gating, multi-provider LLM extraction, SQLite v5 deduplication, and multi-channel syndication

Collegeum Pipeline System Architecture Ingestion sources, deterministic gating, multi-provider LLM extraction, SQLite v5 deduplication, and multi-channel syndication Ingestion Sources (Global Academic Opportunities) Scheduled Batch Processing Pipeline (GitHub Actions Runner) Multi-Channel Broadcast Outlets & Public Dashboards Academic Boards · NBER, EJM, IFS, ZEW · Ingestion Sources (Global Academic Opportunities) · paged scrapers Academic Boards NBER, EJM, IFS, ZEW paged scrapers University ATS · Workday, Greenhouse, Lever · Ingestion Sources (Global Academic Opportunities) · Schema.org LD University ATS Workday, Greenhouse, Lever Schema.org LD Syndicated Feeds · RSS / Atom XML feeds · Ingestion Sources (Global Academic Opportunities) · Econ portals Syndicated Feeds RSS / Atom XML feeds Econ portals X Search Ingest · API v2 & Xquik proxy · Ingestion Sources (Global Academic Opportunities) · @econ_RA search X Search Ingest API v2 & Xquik proxy @econ_RA search Seen Filter · URL & content hashes · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · data/seen.ndjson Seen Filter URL & content hashes data/seen.ndjson Deterministic Gate · Lexicon & negative rules · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · 7 languages Deterministic Gate Lexicon & negative rules 7 languages Rule Extractor · Zero-cost offline fallback · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · offline ready Rule Extractor Zero-cost offline fallback offline ready KeyRotator · Multi-key cooldown pool · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · thread-safe KeyRotator Multi-key cooldown pool thread-safe Parallel LLM Pool · Gemini, Groq, NIM, Claude · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · 12 workers Parallel LLM Pool Gemini, Groq, NIM, Claude 12 workers Deduplicator · MinHash LSH + fuzzy title · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · 3-tier dedupe Deduplicator MinHash LSH + fuzzy title 3-tier dedupe SQLite Cache v5 · WAL mode + locked writes · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · data/predocs.db SQLite Cache v5 WAL mode + locked writes data/predocs.db Committed State · listings.ndjson + seen.ndjson · Scheduled Batch Processing Pipeline (GitHub Actions Runner) · git-versioned Committed State listings.ndjson + seen.ndjson git-versioned Telegram Channel · Rich cards & digests · Multi-Channel Broadcast Outlets & Public Dashboards · broadcast bot Telegram Channel Rich cards & digests broadcast bot X Broadcaster · OAuth 1.0a auto-threads · Multi-Channel Broadcast Outlets & Public Dashboards · @predoc posts X Broadcaster OAuth 1.0a auto-threads @predoc posts GitHub Pages · Web dashboard & RSS · Multi-Channel Broadcast Outlets & Public Dashboards · docs/data/*.json GitHub Pages Web dashboard & RSS docs/data/*.json Telegram Sync · 30-min cron feedback loop · Multi-Channel Broadcast Outlets & Public Dashboards · data/feedback.json Telegram Sync 30-min cron feedback loop data/feedback.json scraped job unseen items gated vacancy active key zero-cost rule extracted listing regex listing canonical insert export journal broadcast Legend External Cloud Database Security Backend Frontend

Ingestion & Deterministic Gating

  • • Academic scrapers, university ATS portals, and X search discover new listings daily
  • • Seen cache blocks repeat page fetches using URL and content hashes
  • • Deterministic gate enforces hiring intent across 7 languages and rejects non-predocs

Parallel Extraction & Deduplication

  • • Multi-provider LLM pool rotates API keys with thread-safe exponential cooldown
  • • Heuristic extractor provides 100% zero-cost fallback for unauthenticated environments
  • • Three-tier deduplication prevents duplicate broadcasts across overlapping sources

Multi-Channel Broadcast & Resilience

  • • Concurrent broadcasts to Telegram channels and X/Twitter accounts
  • • Zero-credential fault tolerance keeps dashboard exports active even with missing keys
  • • Telegram sync cron job runs every 30 minutes to record user feedback marks

Architecture diagram • Built with Archify • Create yours ↗ • Hover to trace • R route • Click to focus • +/− zoom • M radar • [/] views • P play story • T theme • E export