XFEED
X/Twitter Intelligence Monitoring Pipeline
XFEED is a financial intelligence pipeline that monitors curated X/Twitter Lists — following macro analysts, hedge fund managers, and market commentators — to surface market-relevant signals before they appear in mainstream news feeds. The system bypasses the costly official X API by interfacing directly with Twitter's internal GraphQL endpoints via the twscrape library, with smart CSRF token management and ct0 session auto-renewal for long-running stability.
Content flows through a two-stage processing pipeline: raw tweets are stored with full metadata in either Supabase PostgreSQL (cloud) or SQLite (local development, automatic fallback), then processed asynchronously via APScheduler background tasks. Groq and Google GenAI LLMs analyze content for market sentiment, extract ticker mentions, flag significant position announcements, and categorize signals by asset class and conviction level.
The FastAPI backend serves a terminal-aesthetic glassmorphism dashboard that displays the processed intelligence feed in real-time — styled to evoke Bloomberg Terminal aesthetics for the finance audience. Multi-account scraper management with structlog structured logging provides production-level observability. Downstream, processed data exports to Excel/JSON format for quantitative researchers to incorporate into backtesting pipelines and factor models.
Key Highlights
- GraphQL-based X/Twitter scraping bypassing traditional API rate limits
- Smart CSRF token management with ct0 auto-renewal for session stability
- LLM-powered content analysis via Groq and Google GenAI
- Hybrid storage: Supabase (cloud) with automatic SQLite fallback for local dev
- Terminal-aesthetic glassmorphism UI with real-time feed monitoring
- Excel/JSON data export for downstream quantitative analysis
- Multi-account scraper management with structlog observability