Article Ingestion & Scraper Architecture

1. Ingestion Overview

The news ingestion pipeline is responsible for collecting news articles from configured sources, validating and cleaning the text, and storing the clean articles in Supabase. The primary discovery mechanism is RSS feeds, falling back to homepage scraping only when RSS is broken or unavailable.

2. Pipeline Execution Steps

Both manual scraping (via POST /api/scrape) and automatic scheduled scraping run the exact same pipeline steps:

1. Fetch Active Sources: Load active sources from the Supabase `sources` table. We do not hardcode source URLs.

2. RSS Feed Retrieval: Fetch the XML from the source's RSS URL.

3. Parse RSS XML: Extract candidate article URLs, titles, published dates, and description tags.

4. URL Existence Check: Normalize the candidate URLs and query Supabase to check if they already exist. To avoid heavy queries, we check in small chunks (maximum 15 URLs per query).

5. Fetch Details: For new candidate URLs, fetch the detail page. If the page is protected by anti-bot measures, we use Oxylabs Web Scraper API as a retrieval fallback.

6. Content Gate & Validation: Reject the article if it fails quality checks.

// Example of the URL existence check in database queries
const { data: existing } = await supabase
  .from("articles")
  .select("original_url")
  .in("original_url", chunkOf15Urls);

3. Article Content Gate & Cleanup

An article detail page is parsed using Cheerio to extract raw body text, published dates, and feature images. To be saved, the article must pass the content gate:

- Required Fields: Must contain a valid title, article-specific URL, published date, and image URL.

- Length Requirements: Body text must contain at least 3 paragraphs or 900+ meaningful characters.

- Reject List: We filter out non-article pages (e.g., category lists, show pages, product pages, tags, search pages, podcast indexes, or newsletter signup pages) based on URL patterns and content analysis.

- Cleanup: Before saving the `raw_text`, we strip scripts, style tags, ads, social sharing widgets, sidebars, and repeated navigation headers to ensure the saved text reads cleanly as one article.