Oxylabs Scheduler & Vercel Cron Integration

1. The Hourly Automatic Pipeline

To keep news feeds fresh, Lamu News schedules automated ingestion and analysis at the top of every hour. The flow consists of two independent systems working together:

1. Ingestion Job Trigger: Oxylabs Scheduler is configured to scrape publisher homepages/RSS feeds at :00 of every hour.

2. Pipeline Sync Cron: A Vercel Cron Job triggers at :15 past every hour to process the results and run AI evaluations.

2. Vercel Cron Route (/api/cron/pipeline)

The endpoint `/api/cron/pipeline` orchestrates the hourly pipeline. It performs two steps sequentially:

Step 1: Process scheduled results from the Oxylabs Scheduler. It fetches the completed scraper HTML, parses candidate links, filters duplicates, scrapes detail pages, and inserts new articles into Supabase.

Step 2: Run AI Analysis. Immediately trigger bias and sentiment analysis on all newly inserted articles that do not have analysis records.

Note: If Step 1 fails, Step 2 still executes, ensuring pre-existing pending articles are processed.

// Protection check in app/api/cron/pipeline/route.ts
if (process.env.NODE_ENV === "production") {
  const authHeader = request.headers.get("Authorization");
  if (authHeader !== `Bearer ${process.env.CRON_SECRET}`) {
    return NextResponse.json({ error: "Unauthorized" }, { status: 401 });
  }
}

3. Large Integer Precision gotchas

A critical technical detail: Oxylabs schedule IDs and job IDs are 64-bit integers. JavaScript's `Number` type cannot safely represent integers larger than 2^53 - 1 (Number.MAX_SAFE_INTEGER).

If you parse Oxylabs JSON responses using `JSON.parse()`, the last digits of the IDs will be corrupted. Always read these IDs from the raw HTTP response text using string extraction or regular expressions before performing JSON parsing.

// Safe ID extraction from raw response text
const rawText = await response.text();
const idMatch = rawText.match(/"id":\s*(\d+)/);
const scheduleIdString = idMatch ? idMatch[1] : null;