obscura/docs/Intercept-and-modify-requests.md at f3f9b8ec6e3fefeb2aad412880568189b088248b

Files

SGavrl 5db0dc0e01 docs: document the interception API and refresh the skill

The request interception API (#306) and the global --stealth flag are
merged, so:

- Use-as-a-Rust-library.md: add an "Intercept requests" section covering
  on_request/on_response, enable_interception (Continue/Fulfill/Fail),
  and add_preload_script, with examples.
- Intercept-and-modify-requests.md: point embedders at the native API.
- AGENTS.md: --stealth is now global (all subcommands incl. scrape, via
  OBSCURA_STEALTH for workers); note the interception API on the obscura
  crate and that op_fetch_url rewrites must stay behind validate_fetch_url.
- skills/obscura/SKILL.md: fix the build note (stealth is not a default
  feature), correct "randomized" to a consistent fingerprint, add the
  global --stealth flag, --dump assets/cookies, and an interception note.

2026-06-24 23:27:19 +00:00

2.6 KiB

Raw Blame History

CDP Fetch.enable lets you inspect, block, or modify every request the page makes.

Block by resource type

Puppeteer:

await page.setRequestInterception(true);

page.on('request', req => {
  if (['image', 'media', 'font'].includes(req.resourceType())) {
    req.abort();
  } else {
    req.continue();
  }
});

Playwright:

await page.route('**/*', route => {
  if (['image', 'media', 'font'].includes(route.request().resourceType())) {
    route.abort();
  } else {
    route.continue();
  }
});

Block by URL pattern

// Puppeteer
page.on('request', req => {
  const url = req.url();
  if (url.includes('google-analytics.com') || url.includes('doubleclick.net')) {
    req.abort();
  } else {
    req.continue();
  }
});

// Playwright
await page.route(/google-analytics\.com|doubleclick\.net/, route => route.abort());

Modify headers

// Puppeteer
page.on('request', req => {
  req.continue({
    headers: { ...req.headers(), 'X-Custom': 'value' },
  });
});

// Playwright
await page.route('**/*', route => {
  route.continue({
    headers: { ...route.request().headers(), 'X-Custom': 'value' },
  });
});

Return a fake response

// Puppeteer
page.on('request', req => {
  if (req.url().endsWith('/api/feature-flags')) {
    req.respond({
      status: 200,
      contentType: 'application/json',
      body: JSON.stringify({ newDashboard: true }),
    });
  } else {
    req.continue();
  }
});

// Playwright
await page.route('**/api/feature-flags', route => {
  route.fulfill({
    status: 200,
    contentType: 'application/json',
    body: JSON.stringify({ newDashboard: true }),
  });
});

Strip analytics in production scrapes

const BLOCK = [
  'google-analytics.com',
  'googletagmanager.com',
  'doubleclick.net',
  'facebook.net',
  'segment.io',
  'mixpanel.com',
  'hotjar.com',
];

page.on('request', req => {
  if (BLOCK.some(host => req.url().includes(host))) {
    req.abort();
  } else {
    req.continue();
  }
});

Built-in: --stealth ships with a tracker blocklist that handles most of these without per-script setup. See Configure stealth and proxies.

From the Rust library

The patterns above drive interception over CDP from Puppeteer or Playwright. If you embed the engine with the obscura crate, the same capability is a native API on Page: on_request / on_response callbacks, an enable_interception() channel that can block, mock, or rewrite requests, and add_preload_script to run code before the page's own scripts. See Use as a Rust library.

2.6 KiB Raw Blame History