diff --git a/archivebox-ts/README.md b/archivebox-ts/README.md index 8a98523d..bfc5577f 100644 --- a/archivebox-ts/README.md +++ b/archivebox-ts/README.md @@ -40,9 +40,10 @@ archivebox-ts/ ### Prerequisites - Node.js 18+ and npm +- Chrome or Chromium browser (for screenshot, title, and headers extractors) - For specific extractors: - `wget` extractor: wget - - `screenshot` extractor: Python 3 + Playwright + - `screenshot`, `title`, `headers` extractors: puppeteer-core + Chrome with remote debugging ### Setup @@ -57,6 +58,15 @@ npm run build # Initialize ArchiveBox node dist/cli.js init + +# Start Chrome with remote debugging (required for screenshot, title, headers extractors) +# In a separate terminal: +chrome --remote-debugging-port=9222 --headless +# Or on Linux: +chromium --remote-debugging-port=9222 --headless + +# Set the CDP URL environment variable +export CHROME_CDP_URL="http://localhost:9222" ``` ## Usage @@ -71,16 +81,35 @@ node dist/cli.js init ### Add a URL +First, make sure Chrome is running with remote debugging and CHROME_CDP_URL is set: + +```bash +# Terminal 1: Start Chrome +chrome --remote-debugging-port=9222 --headless + +# Terminal 2: Get the WebSocket URL +curl http://localhost:9222/json/version | jq -r .webSocketDebuggerUrl + +# Set the environment variable (use the URL from above) +export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/..." +``` + Archive a URL with all available extractors: ```bash node dist/cli.js add https://example.com ``` -Archive with specific extractors: +Archive with specific extractors (favicon and wget don't need Chrome): ```bash -node dist/cli.js add https://example.com --extractors favicon,title,headers +node dist/cli.js add https://example.com --extractors favicon,wget +``` + +Archive with Chrome-based extractors: + +```bash +node dist/cli.js add https://example.com --extractors title,headers,screenshot ``` Add with custom title: @@ -285,44 +314,116 @@ print("output.txt") - **Language**: Bash - **Dependencies**: curl (auto-installed) - **Output**: `favicon.ico` or `favicon.png` +- **Requires Chrome**: No - **Config**: - `FAVICON_TIMEOUT` - Timeout in seconds (default: 10) ### title -- **Language**: Node.js -- **Dependencies**: Built-in Node.js modules +- **Language**: Node.js + Puppeteer +- **Dependencies**: puppeteer-core, Chrome browser via CDP - **Output**: `title.txt` +- **Requires Chrome**: Yes (via CHROME_CDP_URL) - **Config**: + - `CHROME_CDP_URL` - Chrome DevTools Protocol WebSocket URL (required) - `TITLE_TIMEOUT` - Timeout in milliseconds (default: 10000) - - `TITLE_USER_AGENT` - User agent string ### headers -- **Language**: Bash -- **Dependencies**: curl (auto-installed) +- **Language**: Node.js + Puppeteer +- **Dependencies**: puppeteer-core, Chrome browser via CDP - **Output**: `headers.json` +- **Requires Chrome**: Yes (via CHROME_CDP_URL) - **Config**: - - `HEADERS_TIMEOUT` - Timeout in seconds (default: 10) - - `HEADERS_USER_AGENT` - User agent string + - `CHROME_CDP_URL` - Chrome DevTools Protocol WebSocket URL (required) + - `HEADERS_TIMEOUT` - Timeout in milliseconds (default: 10000) ### wget - **Language**: Bash - **Dependencies**: wget (auto-installed) - **Output**: `warc/archive.warc.gz` and downloaded files +- **Requires Chrome**: No - **Config**: - `WGET_TIMEOUT` - Timeout in seconds (default: 60) - `WGET_USER_AGENT` - User agent string - `WGET_ARGS` - Additional wget arguments ### screenshot -- **Language**: Python -- **Dependencies**: playwright (auto-installed) +- **Language**: Node.js + Puppeteer +- **Dependencies**: puppeteer-core, Chrome browser via CDP - **Output**: `screenshot.png` +- **Requires Chrome**: Yes (via CHROME_CDP_URL) - **Config**: + - `CHROME_CDP_URL` - Chrome DevTools Protocol WebSocket URL (required) - `SCREENSHOT_TIMEOUT` - Timeout in milliseconds (default: 30000) - `SCREENSHOT_WIDTH` - Viewport width (default: 1920) - `SCREENSHOT_HEIGHT` - Viewport height (default: 1080) - `SCREENSHOT_WAIT` - Wait time before screenshot in ms (default: 1000) +## Setting up Chrome for Remote Debugging + +The `screenshot`, `title`, and `headers` extractors require a Chrome browser accessible via the Chrome DevTools Protocol (CDP). This allows multiple extractors to share a single browser instance. + +### Start Chrome with Remote Debugging + +```bash +# Linux/Mac +chromium --remote-debugging-port=9222 --headless --disable-gpu + +# Or with Chrome +chrome --remote-debugging-port=9222 --headless --disable-gpu + +# Windows +chrome.exe --remote-debugging-port=9222 --headless --disable-gpu +``` + +### Get the WebSocket URL + +```bash +# Query the Chrome instance for the WebSocket URL +curl http://localhost:9222/json/version + +# Example output: +# { +# "Browser": "Chrome/120.0.0.0", +# "Protocol-Version": "1.3", +# "User-Agent": "Mozilla/5.0...", +# "V8-Version": "12.0.267.8", +# "WebKit-Version": "537.36", +# "webSocketDebuggerUrl": "ws://localhost:9222/devtools/browser/..." +# } +``` + +### Set the Environment Variable + +```bash +# Extract just the WebSocket URL +export CHROME_CDP_URL=$(curl -s http://localhost:9222/json/version | jq -r .webSocketDebuggerUrl) + +# Or set it manually +export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/12345678-1234-1234-1234-123456789abc" + +# Verify it's set +echo $CHROME_CDP_URL +``` + +### Docker Setup + +For running in Docker, you can use a separate Chrome container: + +```bash +# Start Chrome in a container +docker run -d --name chrome \ + -p 9222:9222 \ + browserless/chrome:latest \ + --remote-debugging-port=9222 \ + --remote-debugging-address=0.0.0.0 + +# Get the CDP URL +export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/$(curl -s http://localhost:9222/json/version | jq -r .webSocketDebuggerUrl | cut -d'/' -f5-)" + +# Run archivebox-ts +node dist/cli.js add https://example.com +``` + ## Development ### Build diff --git a/archivebox-ts/extractors/headers b/archivebox-ts/extractors/headers index a82d4801..95a60f82 100755 --- a/archivebox-ts/extractors/headers +++ b/archivebox-ts/extractors/headers @@ -1,85 +1,129 @@ -#!/bin/bash -# -# Headers Extractor -# Extracts HTTP headers from a given URL -# -# Usage: headers -# Output: headers.json in current directory -# Config: All configuration via environment variables -# HEADERS_TIMEOUT - Timeout in seconds (default: 10) -# HEADERS_USER_AGENT - User agent string -# +#!/usr/bin/env node +// +// Headers Extractor +// Extracts HTTP headers from a given URL using Puppeteer +// +// Usage: headers +// Output: headers.json in current directory +// Config: All configuration via environment variables +// CHROME_CDP_URL - Chrome DevTools Protocol URL (e.g., ws://localhost:9222/devtools/browser/...) +// If not set, will launch a local browser instance +// HEADERS_TIMEOUT - Timeout in milliseconds (default: 10000) +// -set -e +const { spawn } = require('child_process'); +const fs = require('fs'); -URL="$1" +// Check if puppeteer is available +function checkPuppeteer() { + try { + require.resolve('puppeteer-core'); + return true; + } catch (e) { + console.error('Error: puppeteer-core is not installed.'); + console.error('Please install it with: npm install puppeteer-core'); + console.error('Or install chromium: npm install puppeteer'); + return false; + } +} -if [ -z "$URL" ]; then - echo "Error: URL argument required" >&2 - exit 1 -fi +async function main() { + const url = process.argv[2]; -# Auto-install dependencies -if ! command -v curl &> /dev/null; then - echo "Installing curl..." >&2 - if command -v apt-get &> /dev/null; then - sudo apt-get update && sudo apt-get install -y curl - elif command -v yum &> /dev/null; then - sudo yum install -y curl - elif command -v brew &> /dev/null; then - brew install curl - else - echo "Error: Cannot install curl. Please install manually." >&2 - exit 1 - fi -fi + if (!url) { + console.error('Error: URL argument required'); + process.exit(1); + } -# Configuration from environment -TIMEOUT="${HEADERS_TIMEOUT:-10}" -USER_AGENT="${HEADERS_USER_AGENT:-Mozilla/5.0 (compatible; ArchiveBox-TS/0.1)}" + // Configuration from environment + const cdpUrl = process.env.CHROME_CDP_URL; + const timeout = parseInt(process.env.HEADERS_TIMEOUT || '10000', 10); -echo "Extracting headers from: $URL" >&2 + console.error(`Extracting headers from: ${url}`); + if (cdpUrl) { + console.error(`Connecting to browser via CDP: ${cdpUrl}`); + } -# Get headers using curl -HEADERS=$(curl -I -L -s --max-time "$TIMEOUT" --user-agent "$USER_AGENT" "$URL" 2>&1 || echo "") + // Check puppeteer is installed + if (!checkPuppeteer()) { + process.exit(1); + } -if [ -z "$HEADERS" ]; then - echo "Error: Failed to fetch headers" >&2 - exit 1 -fi + const puppeteer = require('puppeteer-core'); -# Convert headers to JSON format (simple key-value pairs) -echo "{" > headers.json + let browser = null; + let shouldCloseBrowser = false; -# Parse headers line by line -FIRST=1 -while IFS=: read -r key value; do - # Skip empty lines and HTTP status line - if [ -z "$key" ] || [[ "$key" =~ ^HTTP ]]; then - continue - fi + try { + // Connect to CDP browser or launch local one + if (cdpUrl) { + browser = await puppeteer.connect({ + browserWSEndpoint: cdpUrl + }); + } else { + console.error('Error: CHROME_CDP_URL environment variable not set.'); + console.error('Please set CHROME_CDP_URL to connect to a Chrome browser via CDP.'); + console.error('Example: export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/..."'); + console.error(''); + console.error('To start Chrome with remote debugging:'); + console.error(' chrome --remote-debugging-port=9222 --headless'); + console.error(' chromium --remote-debugging-port=9222 --headless'); + process.exit(1); + } - # Clean up key and value - key=$(echo "$key" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') - value=$(echo "$value" | tr -d '\r\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//') + const page = await browser.newPage(); - if [ -n "$key" ] && [ -n "$value" ]; then - # Escape quotes in value - value=$(echo "$value" | sed 's/"/\\"/g') + let capturedHeaders = null; - # Add comma if not first entry - if [ "$FIRST" -eq 0 ]; then - echo "," >> headers.json - fi + // Listen for response to capture headers + page.on('response', async (response) => { + if (response.url() === url) { + capturedHeaders = response.headers(); + } + }); - echo -n " \"$key\": \"$value\"" >> headers.json - FIRST=0 - fi -done <<< "$HEADERS" + // Navigate to URL + await page.goto(url, { + timeout, + waitUntil: 'domcontentloaded' + }); -echo "" >> headers.json -echo "}" >> headers.json + await page.close(); -echo "✓ Extracted headers to headers.json" >&2 -echo "headers.json" -exit 0 + if (capturedHeaders) { + // Write to file + fs.writeFileSync('headers.json', JSON.stringify(capturedHeaders, null, 2), 'utf8'); + console.error('✓ Extracted headers to headers.json'); + console.log('headers.json'); + + if (shouldCloseBrowser) { + await browser.close(); + } + + process.exit(0); + } else { + console.error('Warning: Could not capture headers'); + + if (shouldCloseBrowser) { + await browser.close(); + } + + process.exit(1); + } + + } catch (err) { + console.error(`Error: ${err.message}`); + + if (browser && shouldCloseBrowser) { + try { + await browser.close(); + } catch (closeErr) { + // Ignore close errors + } + } + + process.exit(1); + } +} + +main(); diff --git a/archivebox-ts/extractors/screenshot b/archivebox-ts/extractors/screenshot index 6afb39b9..2cf278ed 100755 --- a/archivebox-ts/extractors/screenshot +++ b/archivebox-ts/extractors/screenshot @@ -1,77 +1,125 @@ -#!/usr/bin/env python3 -# -# Screenshot Extractor -# Captures a screenshot of a given URL using Playwright -# -# Usage: screenshot -# Output: screenshot.png in current directory -# Config: All configuration via environment variables -# SCREENSHOT_TIMEOUT - Timeout in milliseconds (default: 30000) -# SCREENSHOT_WIDTH - Viewport width (default: 1920) -# SCREENSHOT_HEIGHT - Viewport height (default: 1080) -# SCREENSHOT_WAIT - Time to wait before screenshot in ms (default: 1000) -# +#!/usr/bin/env node +// +// Screenshot Extractor +// Captures a screenshot of a given URL using Puppeteer +// +// Usage: screenshot +// Output: screenshot.png in current directory +// Config: All configuration via environment variables +// CHROME_CDP_URL - Chrome DevTools Protocol URL (e.g., ws://localhost:9222/devtools/browser/...) +// If not set, will launch a local browser instance +// SCREENSHOT_TIMEOUT - Timeout in milliseconds (default: 30000) +// SCREENSHOT_WIDTH - Viewport width (default: 1920) +// SCREENSHOT_HEIGHT - Viewport height (default: 1080) +// SCREENSHOT_WAIT - Time to wait before screenshot in ms (default: 1000) +// -import sys -import os -import subprocess +const { spawn } = require('child_process'); +const fs = require('fs'); -def ensure_playwright(): - """Auto-install playwright if not available""" - try: - from playwright.sync_api import sync_playwright - return True - except ImportError: - print("Installing playwright...", file=sys.stderr) - try: - subprocess.check_call([sys.executable, "-m", "pip", "install", "playwright"]) - subprocess.check_call([sys.executable, "-m", "playwright", "install", "chromium"]) - from playwright.sync_api import sync_playwright - return True - except Exception as e: - print(f"Error installing playwright: {e}", file=sys.stderr) - return False +// Check if puppeteer is available +function checkPuppeteer() { + try { + require.resolve('puppeteer-core'); + return true; + } catch (e) { + console.error('Error: puppeteer-core is not installed.'); + console.error('Please install it with: npm install puppeteer-core'); + console.error('Or install chromium: npm install puppeteer'); + return false; + } +} -def main(): - if len(sys.argv) < 2: - print("Error: URL argument required", file=sys.stderr) - sys.exit(1) +async function main() { + const url = process.argv[2]; - url = sys.argv[1] + if (!url) { + console.error('Error: URL argument required'); + process.exit(1); + } - # Configuration from environment - timeout = int(os.environ.get('SCREENSHOT_TIMEOUT', '30000')) - width = int(os.environ.get('SCREENSHOT_WIDTH', '1920')) - height = int(os.environ.get('SCREENSHOT_HEIGHT', '1080')) - wait = int(os.environ.get('SCREENSHOT_WAIT', '1000')) + // Configuration from environment + const cdpUrl = process.env.CHROME_CDP_URL; + const timeout = parseInt(process.env.SCREENSHOT_TIMEOUT || '30000', 10); + const width = parseInt(process.env.SCREENSHOT_WIDTH || '1920', 10); + const height = parseInt(process.env.SCREENSHOT_HEIGHT || '1080', 10); + const wait = parseInt(process.env.SCREENSHOT_WAIT || '1000', 10); - print(f"Capturing screenshot of: {url}", file=sys.stderr) + console.error(`Capturing screenshot of: ${url}`); + if (cdpUrl) { + console.error(`Connecting to browser via CDP: ${cdpUrl}`); + } - # Ensure playwright is installed - if not ensure_playwright(): - sys.exit(1) + // Check puppeteer is installed + if (!checkPuppeteer()) { + process.exit(1); + } - from playwright.sync_api import sync_playwright + const puppeteer = require('puppeteer-core'); - try: - with sync_playwright() as p: - browser = p.chromium.launch() - page = browser.new_page(viewport={'width': width, 'height': height}) - page.goto(url, timeout=timeout, wait_until='networkidle') + let browser = null; + let shouldCloseBrowser = false; - # Wait a bit for any dynamic content - page.wait_for_timeout(wait) + try { + // Connect to CDP browser or launch local one + if (cdpUrl) { + browser = await puppeteer.connect({ + browserWSEndpoint: cdpUrl, + defaultViewport: { width, height } + }); + } else { + console.error('Error: CHROME_CDP_URL environment variable not set.'); + console.error('Please set CHROME_CDP_URL to connect to a Chrome browser via CDP.'); + console.error('Example: export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/..."'); + console.error(''); + console.error('To start Chrome with remote debugging:'); + console.error(' chrome --remote-debugging-port=9222 --headless'); + console.error(' chromium --remote-debugging-port=9222 --headless'); + process.exit(1); + } - page.screenshot(path='screenshot.png', full_page=True) - browser.close() + const page = await browser.newPage(); + await page.setViewport({ width, height }); - print("✓ Captured screenshot: screenshot.png", file=sys.stderr) - print("screenshot.png") - sys.exit(0) + // Navigate to URL + await page.goto(url, { + timeout, + waitUntil: 'networkidle2' + }); - except Exception as e: - print(f"Error: {e}", file=sys.stderr) - sys.exit(1) + // Wait a bit for any dynamic content + await page.waitForTimeout(wait); -if __name__ == '__main__': - main() + // Take screenshot + await page.screenshot({ + path: 'screenshot.png', + fullPage: true + }); + + await page.close(); + + console.error('✓ Captured screenshot: screenshot.png'); + console.log('screenshot.png'); + + if (shouldCloseBrowser) { + await browser.close(); + } + + process.exit(0); + + } catch (err) { + console.error(`Error: ${err.message}`); + + if (browser && shouldCloseBrowser) { + try { + await browser.close(); + } catch (closeErr) { + // Ignore close errors + } + } + + process.exit(1); + } +} + +main(); diff --git a/archivebox-ts/extractors/title b/archivebox-ts/extractors/title index 4321d923..0ab78e20 100755 --- a/archivebox-ts/extractors/title +++ b/archivebox-ts/extractors/title @@ -1,89 +1,123 @@ #!/usr/bin/env node // // Title Extractor -// Extracts the page title from a given URL +// Extracts the page title from a given URL using Puppeteer // // Usage: title // Output: title.txt in current directory // Config: All configuration via environment variables +// CHROME_CDP_URL - Chrome DevTools Protocol URL (e.g., ws://localhost:9222/devtools/browser/...) +// If not set, will launch a local browser instance // TITLE_TIMEOUT - Timeout in milliseconds (default: 10000) -// TITLE_USER_AGENT - User agent string // -const https = require('https'); -const http = require('http'); +const { spawn } = require('child_process'); const fs = require('fs'); -const { URL } = require('url'); -const url = process.argv[2]; - -if (!url) { - console.error('Error: URL argument required'); - process.exit(1); +// Check if puppeteer is available +function checkPuppeteer() { + try { + require.resolve('puppeteer-core'); + return true; + } catch (e) { + console.error('Error: puppeteer-core is not installed.'); + console.error('Please install it with: npm install puppeteer-core'); + console.error('Or install chromium: npm install puppeteer'); + return false; + } } -// Configuration from environment -const TIMEOUT = parseInt(process.env.TITLE_TIMEOUT || '10000', 10); -const USER_AGENT = process.env.TITLE_USER_AGENT || 'Mozilla/5.0 (compatible; ArchiveBox-TS/0.1)'; +async function main() { + const url = process.argv[2]; -console.error(`Extracting title from: ${url}`); + if (!url) { + console.error('Error: URL argument required'); + process.exit(1); + } -// Parse URL -let parsedUrl; -try { - parsedUrl = new URL(url); -} catch (err) { - console.error(`Error: Invalid URL: ${err.message}`); - process.exit(1); -} + // Configuration from environment + const cdpUrl = process.env.CHROME_CDP_URL; + const timeout = parseInt(process.env.TITLE_TIMEOUT || '10000', 10); -// Choose http or https module -const client = parsedUrl.protocol === 'https:' ? https : http; + console.error(`Extracting title from: ${url}`); + if (cdpUrl) { + console.error(`Connecting to browser via CDP: ${cdpUrl}`); + } -// Make request -const options = { - headers: { - 'User-Agent': USER_AGENT, - }, - timeout: TIMEOUT, -}; + // Check puppeteer is installed + if (!checkPuppeteer()) { + process.exit(1); + } -client.get(url, options, (res) => { - let html = ''; + const puppeteer = require('puppeteer-core'); - res.on('data', (chunk) => { - html += chunk; + let browser = null; + let shouldCloseBrowser = false; - // Early exit if we found the title (optimization) - if (html.includes('')) { - res.destroy(); - } - }); - - res.on('end', () => { - // Extract title using regex - const titleMatch = html.match(/]*>(.*?)<\/title>/is); - - if (titleMatch && titleMatch[1]) { - const title = titleMatch[1] - .replace(/<[^>]*>/g, '') // Remove any HTML tags - .replace(/\s+/g, ' ') // Normalize whitespace - .trim(); - - // Write to file - fs.writeFileSync('title.txt', title, 'utf8'); - console.error(`✓ Extracted title: ${title}`); - console.log('title.txt'); - process.exit(0); + try { + // Connect to CDP browser or launch local one + if (cdpUrl) { + browser = await puppeteer.connect({ + browserWSEndpoint: cdpUrl + }); } else { - console.error('Warning: Could not find title tag'); + console.error('Error: CHROME_CDP_URL environment variable not set.'); + console.error('Please set CHROME_CDP_URL to connect to a Chrome browser via CDP.'); + console.error('Example: export CHROME_CDP_URL="ws://localhost:9222/devtools/browser/..."'); + console.error(''); + console.error('To start Chrome with remote debugging:'); + console.error(' chrome --remote-debugging-port=9222 --headless'); + console.error(' chromium --remote-debugging-port=9222 --headless'); process.exit(1); } - }); -}).on('error', (err) => { - console.error(`Error: ${err.message}`); - process.exit(1); -}).on('timeout', () => { - console.error('Error: Request timeout'); - process.exit(1); -}); + + const page = await browser.newPage(); + + // Navigate to URL + await page.goto(url, { + timeout, + waitUntil: 'domcontentloaded' + }); + + // Get the title + const title = await page.title(); + + await page.close(); + + if (title && title.trim()) { + // Write to file + fs.writeFileSync('title.txt', title.trim(), 'utf8'); + console.error(`✓ Extracted title: ${title.trim()}`); + console.log('title.txt'); + + if (shouldCloseBrowser) { + await browser.close(); + } + + process.exit(0); + } else { + console.error('Warning: Could not find title'); + + if (shouldCloseBrowser) { + await browser.close(); + } + + process.exit(1); + } + + } catch (err) { + console.error(`Error: ${err.message}`); + + if (browser && shouldCloseBrowser) { + try { + await browser.close(); + } catch (closeErr) { + // Ignore close errors + } + } + + process.exit(1); + } +} + +main(); diff --git a/archivebox-ts/package-lock.json b/archivebox-ts/package-lock.json index 67698d0b..60dfc839 100644 --- a/archivebox-ts/package-lock.json +++ b/archivebox-ts/package-lock.json @@ -11,7 +11,8 @@ "dependencies": { "better-sqlite3": "^11.0.0", "commander": "^12.0.0", - "nanoid": "^3.3.7" + "nanoid": "^3.3.7", + "puppeteer-core": "^24.28.0" }, "bin": { "archivebox-ts": "dist/cli.js" @@ -22,6 +23,58 @@ "typescript": "^5.3.3" } }, + "node_modules/@puppeteer/browsers": { + "version": "2.10.13", + "resolved": "https://registry.npmjs.org/@puppeteer/browsers/-/browsers-2.10.13.tgz", + "integrity": "sha512-a9Ruw3j3qlnB5a/zHRTkruppynxqaeE4H9WNj5eYGRWqw0ZauZ23f4W2ARf3hghF5doozyD+CRtt7XSYuYRI/Q==", + "license": "Apache-2.0", + "dependencies": { + "debug": "^4.4.3", + "extract-zip": "^2.0.1", + "progress": "^2.0.3", + "proxy-agent": "^6.5.0", + "semver": "^7.7.3", + "tar-fs": "^3.1.1", + "yargs": "^17.7.2" + }, + "bin": { + "browsers": "lib/cjs/main-cli.js" + }, + "engines": { + "node": ">=18" + } + }, + "node_modules/@puppeteer/browsers/node_modules/tar-fs": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.1.tgz", + "integrity": "sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg==", + "license": "MIT", + "dependencies": { + "pump": "^3.0.0", + "tar-stream": "^3.1.5" + }, + "optionalDependencies": { + "bare-fs": "^4.0.1", + "bare-path": "^3.0.0" + } + }, + "node_modules/@puppeteer/browsers/node_modules/tar-stream": { + "version": "3.1.7", + "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.7.tgz", + "integrity": "sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ==", + "license": "MIT", + "dependencies": { + "b4a": "^1.6.4", + "fast-fifo": "^1.2.0", + "streamx": "^2.15.0" + } + }, + "node_modules/@tootallnate/quickjs-emscripten": { + "version": "0.23.0", + "resolved": "https://registry.npmjs.org/@tootallnate/quickjs-emscripten/-/quickjs-emscripten-0.23.0.tgz", + "integrity": "sha512-C5Mc6rdnsaJDjO3UpGW/CQTHtCKaYlScZTly4JIu97Jxo/odCiH0ITnDXSJPTOrEKk/ycSZ0AOgTmkDtkOsvIA==", + "license": "MIT" + }, "node_modules/@types/better-sqlite3": { "version": "7.6.13", "resolved": "https://registry.npmjs.org/@types/better-sqlite3/-/better-sqlite3-7.6.13.tgz", @@ -36,12 +89,172 @@ "version": "20.19.24", "resolved": "https://registry.npmjs.org/@types/node/-/node-20.19.24.tgz", "integrity": "sha512-FE5u0ezmi6y9OZEzlJfg37mqqf6ZDSF2V/NLjUyGrR9uTZ7Sb9F7bLNZ03S4XVUNRWGA7Ck4c1kK+YnuWjl+DA==", - "dev": true, + "devOptional": true, "license": "MIT", "dependencies": { "undici-types": "~6.21.0" } }, + "node_modules/@types/yauzl": { + "version": "2.10.3", + "resolved": "https://registry.npmjs.org/@types/yauzl/-/yauzl-2.10.3.tgz", + "integrity": "sha512-oJoftv0LSuaDZE3Le4DbKX+KS9G36NzOeSap90UIK0yMA/NhKJhqlSGtNDORNRaIbQfzjXDrQa0ytJ6mNRGz/Q==", + "license": "MIT", + "optional": true, + "dependencies": { + "@types/node": "*" + } + }, + "node_modules/agent-base": { + "version": "7.1.4", + "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", + "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", + "license": "MIT", + "engines": { + "node": ">= 14" + } + }, + "node_modules/ansi-regex": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-5.0.1.tgz", + "integrity": "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/ansi-styles": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-4.3.0.tgz", + "integrity": "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg==", + "license": "MIT", + "dependencies": { + "color-convert": "^2.0.1" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/chalk/ansi-styles?sponsor=1" + } + }, + "node_modules/ast-types": { + "version": "0.13.4", + "resolved": "https://registry.npmjs.org/ast-types/-/ast-types-0.13.4.tgz", + "integrity": "sha512-x1FCFnFifvYDDzTaLII71vG5uvDwgtmDTEVWAxrgeiR8VjMONcCXJx7E+USjDtHlwFmt9MysbqgF9b9Vjr6w+w==", + "license": "MIT", + "dependencies": { + "tslib": "^2.0.1" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/b4a": { + "version": "1.7.3", + "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.7.3.tgz", + "integrity": "sha512-5Q2mfq2WfGuFp3uS//0s6baOJLMoVduPYVeNmDYxu5OUA1/cBfvr2RIS7vi62LdNj/urk1hfmj867I3qt6uZ7Q==", + "license": "Apache-2.0", + "peerDependencies": { + "react-native-b4a": "*" + }, + "peerDependenciesMeta": { + "react-native-b4a": { + "optional": true + } + } + }, + "node_modules/bare-events": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.8.1.tgz", + "integrity": "sha512-oxSAxTS1hRfnyit2CL5QpAOS5ixfBjj6ex3yTNvXyY/kE719jQ/IjuESJBK2w5v4wwQRAHGseVJXx9QBYOtFGQ==", + "license": "Apache-2.0", + "peerDependencies": { + "bare-abort-controller": "*" + }, + "peerDependenciesMeta": { + "bare-abort-controller": { + "optional": true + } + } + }, + "node_modules/bare-fs": { + "version": "4.5.0", + "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.5.0.tgz", + "integrity": "sha512-GljgCjeupKZJNetTqxKaQArLK10vpmK28or0+RwWjEl5Rk+/xG3wkpmkv+WrcBm3q1BwHKlnhXzR8O37kcvkXQ==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "bare-events": "^2.5.4", + "bare-path": "^3.0.0", + "bare-stream": "^2.6.4", + "bare-url": "^2.2.2", + "fast-fifo": "^1.3.2" + }, + "engines": { + "bare": ">=1.16.0" + }, + "peerDependencies": { + "bare-buffer": "*" + }, + "peerDependenciesMeta": { + "bare-buffer": { + "optional": true + } + } + }, + "node_modules/bare-os": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.6.2.tgz", + "integrity": "sha512-T+V1+1srU2qYNBmJCXZkUY5vQ0B4FSlL3QDROnKQYOqeiQR8UbjNHlPa+TIbM4cuidiN9GaTaOZgSEgsvPbh5A==", + "license": "Apache-2.0", + "optional": true, + "engines": { + "bare": ">=1.14.0" + } + }, + "node_modules/bare-path": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz", + "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "bare-os": "^3.0.1" + } + }, + "node_modules/bare-stream": { + "version": "2.7.0", + "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.7.0.tgz", + "integrity": "sha512-oyXQNicV1y8nc2aKffH+BUHFRXmx6VrPzlnaEvMhram0nPBrKcEdcyBg5r08D0i8VxngHFAiVyn1QKXpSG0B8A==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "streamx": "^2.21.0" + }, + "peerDependencies": { + "bare-buffer": "*", + "bare-events": "*" + }, + "peerDependenciesMeta": { + "bare-buffer": { + "optional": true + }, + "bare-events": { + "optional": true + } + } + }, + "node_modules/bare-url": { + "version": "2.3.2", + "resolved": "https://registry.npmjs.org/bare-url/-/bare-url-2.3.2.tgz", + "integrity": "sha512-ZMq4gd9ngV5aTMa5p9+UfY0b3skwhHELaDkhEHetMdX0LRkW9kzaym4oo/Eh+Ghm0CCDuMTsRIGM/ytUc1ZYmw==", + "license": "Apache-2.0", + "optional": true, + "dependencies": { + "bare-path": "^3.0.0" + } + }, "node_modules/base64-js": { "version": "1.5.1", "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", @@ -62,6 +275,15 @@ ], "license": "MIT" }, + "node_modules/basic-ftp": { + "version": "5.0.5", + "resolved": "https://registry.npmjs.org/basic-ftp/-/basic-ftp-5.0.5.tgz", + "integrity": "sha512-4Bcg1P8xhUuqcii/S0Z9wiHIrQVPMermM1any+MX5GeGD7faD3/msQUDGLol9wOcz4/jbg/WJnGqoJF6LiBdtg==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + } + }, "node_modules/better-sqlite3": { "version": "11.10.0", "resolved": "https://registry.npmjs.org/better-sqlite3/-/better-sqlite3-11.10.0.tgz", @@ -117,12 +339,66 @@ "ieee754": "^1.1.13" } }, + "node_modules/buffer-crc32": { + "version": "0.2.13", + "resolved": "https://registry.npmjs.org/buffer-crc32/-/buffer-crc32-0.2.13.tgz", + "integrity": "sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ==", + "license": "MIT", + "engines": { + "node": "*" + } + }, "node_modules/chownr": { "version": "1.1.4", "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", "license": "ISC" }, + "node_modules/chromium-bidi": { + "version": "10.5.1", + "resolved": "https://registry.npmjs.org/chromium-bidi/-/chromium-bidi-10.5.1.tgz", + "integrity": "sha512-rlj6OyhKhVTnk4aENcUme3Jl9h+cq4oXu4AzBcvr8RMmT6BR4a3zSNT9dbIfXr9/BS6ibzRyDhowuw4n2GgzsQ==", + "license": "Apache-2.0", + "dependencies": { + "mitt": "^3.0.1", + "zod": "^3.24.1" + }, + "peerDependencies": { + "devtools-protocol": "*" + } + }, + "node_modules/cliui": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/cliui/-/cliui-8.0.1.tgz", + "integrity": "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ==", + "license": "ISC", + "dependencies": { + "string-width": "^4.2.0", + "strip-ansi": "^6.0.1", + "wrap-ansi": "^7.0.0" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/color-convert": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", + "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", + "license": "MIT", + "dependencies": { + "color-name": "~1.1.4" + }, + "engines": { + "node": ">=7.0.0" + } + }, + "node_modules/color-name": { + "version": "1.1.4", + "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", + "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", + "license": "MIT" + }, "node_modules/commander": { "version": "12.1.0", "resolved": "https://registry.npmjs.org/commander/-/commander-12.1.0.tgz", @@ -132,6 +408,32 @@ "node": ">=18" } }, + "node_modules/data-uri-to-buffer": { + "version": "6.0.2", + "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-6.0.2.tgz", + "integrity": "sha512-7hvf7/GW8e86rW0ptuwS3OcBGDjIi6SZva7hCyWC0yYry2cOPmLIjXAUHI6DK2HsnwJd9ifmt57i8eV2n4YNpw==", + "license": "MIT", + "engines": { + "node": ">= 14" + } + }, + "node_modules/debug": { + "version": "4.4.3", + "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz", + "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==", + "license": "MIT", + "dependencies": { + "ms": "^2.1.3" + }, + "engines": { + "node": ">=6.0" + }, + "peerDependenciesMeta": { + "supports-color": { + "optional": true + } + } + }, "node_modules/decompress-response": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", @@ -156,6 +458,20 @@ "node": ">=4.0.0" } }, + "node_modules/degenerator": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/degenerator/-/degenerator-5.0.1.tgz", + "integrity": "sha512-TllpMR/t0M5sqCXfj85i4XaAzxmS5tVA16dqvdkMwGmzI+dXLXnw3J+3Vdv7VKw+ThlTMboK6i9rnZ6Nntj5CQ==", + "license": "MIT", + "dependencies": { + "ast-types": "^0.13.4", + "escodegen": "^2.1.0", + "esprima": "^4.0.1" + }, + "engines": { + "node": ">= 14" + } + }, "node_modules/detect-libc": { "version": "2.1.2", "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.1.2.tgz", @@ -165,6 +481,18 @@ "node": ">=8" } }, + "node_modules/devtools-protocol": { + "version": "0.0.1521046", + "resolved": "https://registry.npmjs.org/devtools-protocol/-/devtools-protocol-0.0.1521046.tgz", + "integrity": "sha512-vhE6eymDQSKWUXwwA37NtTTVEzjtGVfDr3pRbsWEQ5onH/Snp2c+2xZHWJJawG/0hCCJLRGt4xVtEVUVILol4w==", + "license": "BSD-3-Clause" + }, + "node_modules/emoji-regex": { + "version": "8.0.0", + "resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz", + "integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==", + "license": "MIT" + }, "node_modules/end-of-stream": { "version": "1.4.5", "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", @@ -174,6 +502,76 @@ "once": "^1.4.0" } }, + "node_modules/escalade": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/escalade/-/escalade-3.2.0.tgz", + "integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==", + "license": "MIT", + "engines": { + "node": ">=6" + } + }, + "node_modules/escodegen": { + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/escodegen/-/escodegen-2.1.0.tgz", + "integrity": "sha512-2NlIDTwUWJN0mRPQOdtQBzbUHvdGY2P1VXSyU83Q3xKxM7WHX2Ql8dKq782Q9TgQUNOLEzEYu9bzLNj1q88I5w==", + "license": "BSD-2-Clause", + "dependencies": { + "esprima": "^4.0.1", + "estraverse": "^5.2.0", + "esutils": "^2.0.2" + }, + "bin": { + "escodegen": "bin/escodegen.js", + "esgenerate": "bin/esgenerate.js" + }, + "engines": { + "node": ">=6.0" + }, + "optionalDependencies": { + "source-map": "~0.6.1" + } + }, + "node_modules/esprima": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/esprima/-/esprima-4.0.1.tgz", + "integrity": "sha512-eGuFFw7Upda+g4p+QHvnW0RyTX/SVeJBDM/gCtMARO0cLuT2HcEKnTPvhjV6aGeqrCB/sbNop0Kszm0jsaWU4A==", + "license": "BSD-2-Clause", + "bin": { + "esparse": "bin/esparse.js", + "esvalidate": "bin/esvalidate.js" + }, + "engines": { + "node": ">=4" + } + }, + "node_modules/estraverse": { + "version": "5.3.0", + "resolved": "https://registry.npmjs.org/estraverse/-/estraverse-5.3.0.tgz", + "integrity": "sha512-MMdARuVEQziNTeJD8DgMqmhwR11BRQ/cBP+pLtYdSTnf3MIO8fFeiINEbX36ZdNlfU/7A9f3gUw49B3oQsvwBA==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=4.0" + } + }, + "node_modules/esutils": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/esutils/-/esutils-2.0.3.tgz", + "integrity": "sha512-kVscqXk4OCp68SZ0dkgEKVi6/8ij300KBWTJq32P/dYeWTSwK41WyTxalN1eRmA5Z9UU/LX9D7FWSmV9SAYx6g==", + "license": "BSD-2-Clause", + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/events-universal": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/events-universal/-/events-universal-1.0.1.tgz", + "integrity": "sha512-LUd5euvbMLpwOF8m6ivPCbhQeSiYVNb8Vs0fQ8QjXo0JTkEHpz8pxdQf0gStltaPpw0Cca8b39KxvK9cfKRiAw==", + "license": "Apache-2.0", + "dependencies": { + "bare-events": "^2.7.0" + } + }, "node_modules/expand-template": { "version": "2.0.3", "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", @@ -183,6 +581,41 @@ "node": ">=6" } }, + "node_modules/extract-zip": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/extract-zip/-/extract-zip-2.0.1.tgz", + "integrity": "sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg==", + "license": "BSD-2-Clause", + "dependencies": { + "debug": "^4.1.1", + "get-stream": "^5.1.0", + "yauzl": "^2.10.0" + }, + "bin": { + "extract-zip": "cli.js" + }, + "engines": { + "node": ">= 10.17.0" + }, + "optionalDependencies": { + "@types/yauzl": "^2.9.1" + } + }, + "node_modules/fast-fifo": { + "version": "1.3.2", + "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz", + "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", + "license": "MIT" + }, + "node_modules/fd-slicer": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/fd-slicer/-/fd-slicer-1.1.0.tgz", + "integrity": "sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g==", + "license": "MIT", + "dependencies": { + "pend": "~1.2.0" + } + }, "node_modules/file-uri-to-path": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/file-uri-to-path/-/file-uri-to-path-1.0.0.tgz", @@ -195,12 +628,76 @@ "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", "license": "MIT" }, + "node_modules/get-caller-file": { + "version": "2.0.5", + "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", + "integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==", + "license": "ISC", + "engines": { + "node": "6.* || 8.* || >= 10.*" + } + }, + "node_modules/get-stream": { + "version": "5.2.0", + "resolved": "https://registry.npmjs.org/get-stream/-/get-stream-5.2.0.tgz", + "integrity": "sha512-nBF+F1rAZVCu/p7rjzgA+Yb4lfYXrpl7a6VmJrU8wF9I1CKvP/QwPNZHnOlwbTkY6dvtFIzFMSyQXbLoTQPRpA==", + "license": "MIT", + "dependencies": { + "pump": "^3.0.0" + }, + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/get-uri": { + "version": "6.0.5", + "resolved": "https://registry.npmjs.org/get-uri/-/get-uri-6.0.5.tgz", + "integrity": "sha512-b1O07XYq8eRuVzBNgJLstU6FYc1tS6wnMtF1I1D9lE8LxZSOGZ7LhxN54yPP6mGw5f2CkXY2BQUL9Fx41qvcIg==", + "license": "MIT", + "dependencies": { + "basic-ftp": "^5.0.2", + "data-uri-to-buffer": "^6.0.2", + "debug": "^4.3.4" + }, + "engines": { + "node": ">= 14" + } + }, "node_modules/github-from-package": { "version": "0.0.0", "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", "license": "MIT" }, + "node_modules/http-proxy-agent": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", + "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.0", + "debug": "^4.3.4" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/https-proxy-agent": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", + "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.2", + "debug": "4" + }, + "engines": { + "node": ">= 14" + } + }, "node_modules/ieee754": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", @@ -233,6 +730,33 @@ "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", "license": "ISC" }, + "node_modules/ip-address": { + "version": "10.0.1", + "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-10.0.1.tgz", + "integrity": "sha512-NWv9YLW4PoW2B7xtzaS3NCot75m6nK7Icdv0o3lfMceJVRfSoQwqD4wEH5rLwoKJwUiZ/rfpiVBhnaF0FK4HoA==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, + "node_modules/is-fullwidth-code-point": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-3.0.0.tgz", + "integrity": "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg==", + "license": "MIT", + "engines": { + "node": ">=8" + } + }, + "node_modules/lru-cache": { + "version": "7.18.3", + "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-7.18.3.tgz", + "integrity": "sha512-jumlc0BIUrS3qJGgIkWZsyfAM7NCWiBcCDhnd+3NNM5KbBmLTgHVfWBcg6W+rLUsIpzpERPsvwUP7CckAQSOoA==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, "node_modules/mimic-response": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", @@ -254,12 +778,24 @@ "url": "https://github.com/sponsors/ljharb" } }, + "node_modules/mitt": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/mitt/-/mitt-3.0.1.tgz", + "integrity": "sha512-vKivATfr97l2/QBCYAkXYDbrIWPM2IIKEl7YPhjCvKlG3kE2gm+uBo6nEXK3M5/Ffh/FLpKExzOQ3JJoJGFKBw==", + "license": "MIT" + }, "node_modules/mkdirp-classic": { "version": "0.5.3", "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", "license": "MIT" }, + "node_modules/ms": { + "version": "2.1.3", + "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", + "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", + "license": "MIT" + }, "node_modules/nanoid": { "version": "3.3.11", "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz", @@ -284,6 +820,15 @@ "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", "license": "MIT" }, + "node_modules/netmask": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/netmask/-/netmask-2.0.2.tgz", + "integrity": "sha512-dBpDMdxv9Irdq66304OLfEmQ9tbNRFnFTuZiLo+bD+r332bBmMJ8GBLXklIXXgxd3+v9+KUnZaUR5PJMa75Gsg==", + "license": "MIT", + "engines": { + "node": ">= 0.4.0" + } + }, "node_modules/node-abi": { "version": "3.80.0", "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.80.0.tgz", @@ -305,6 +850,44 @@ "wrappy": "1" } }, + "node_modules/pac-proxy-agent": { + "version": "7.2.0", + "resolved": "https://registry.npmjs.org/pac-proxy-agent/-/pac-proxy-agent-7.2.0.tgz", + "integrity": "sha512-TEB8ESquiLMc0lV8vcd5Ql/JAKAoyzHFXaStwjkzpOpC5Yv+pIzLfHvjTSdf3vpa2bMiUQrg9i6276yn8666aA==", + "license": "MIT", + "dependencies": { + "@tootallnate/quickjs-emscripten": "^0.23.0", + "agent-base": "^7.1.2", + "debug": "^4.3.4", + "get-uri": "^6.0.1", + "http-proxy-agent": "^7.0.0", + "https-proxy-agent": "^7.0.6", + "pac-resolver": "^7.0.1", + "socks-proxy-agent": "^8.0.5" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/pac-resolver": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/pac-resolver/-/pac-resolver-7.0.1.tgz", + "integrity": "sha512-5NPgf87AT2STgwa2ntRMr45jTKrYBGkVU36yT0ig/n/GMAa3oPqhZfIQ2kMEimReg0+t9kZViDVZ83qfVUlckg==", + "license": "MIT", + "dependencies": { + "degenerator": "^5.0.0", + "netmask": "^2.0.2" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/pend": { + "version": "1.2.0", + "resolved": "https://registry.npmjs.org/pend/-/pend-1.2.0.tgz", + "integrity": "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==", + "license": "MIT" + }, "node_modules/prebuild-install": { "version": "7.1.3", "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", @@ -331,6 +914,40 @@ "node": ">=10" } }, + "node_modules/progress": { + "version": "2.0.3", + "resolved": "https://registry.npmjs.org/progress/-/progress-2.0.3.tgz", + "integrity": "sha512-7PiHtLll5LdnKIMw100I+8xJXR5gW2QwWYkT6iJva0bXitZKa/XMrSbdmg3r2Xnaidz9Qumd0VPaMrZlF9V9sA==", + "license": "MIT", + "engines": { + "node": ">=0.4.0" + } + }, + "node_modules/proxy-agent": { + "version": "6.5.0", + "resolved": "https://registry.npmjs.org/proxy-agent/-/proxy-agent-6.5.0.tgz", + "integrity": "sha512-TmatMXdr2KlRiA2CyDu8GqR8EjahTG3aY3nXjdzFyoZbmB8hrBsTyMezhULIXKnC0jpfjlmiZ3+EaCzoInSu/A==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.2", + "debug": "^4.3.4", + "http-proxy-agent": "^7.0.1", + "https-proxy-agent": "^7.0.6", + "lru-cache": "^7.14.1", + "pac-proxy-agent": "^7.1.0", + "proxy-from-env": "^1.1.0", + "socks-proxy-agent": "^8.0.5" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/proxy-from-env": { + "version": "1.1.0", + "resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz", + "integrity": "sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==", + "license": "MIT" + }, "node_modules/pump": { "version": "3.0.3", "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.3.tgz", @@ -341,6 +958,24 @@ "once": "^1.3.1" } }, + "node_modules/puppeteer-core": { + "version": "24.28.0", + "resolved": "https://registry.npmjs.org/puppeteer-core/-/puppeteer-core-24.28.0.tgz", + "integrity": "sha512-QpAqaYgeZHF5/xAZ4jAOzsU+l0Ed4EJoWkRdfw8rNqmSN7itcdYeCJaSPQ0s5Pyn/eGNC4xNevxbgY+5bzNllw==", + "license": "Apache-2.0", + "dependencies": { + "@puppeteer/browsers": "2.10.13", + "chromium-bidi": "10.5.1", + "debug": "^4.4.3", + "devtools-protocol": "0.0.1521046", + "typed-query-selector": "^2.12.0", + "webdriver-bidi-protocol": "0.3.8", + "ws": "^8.18.3" + }, + "engines": { + "node": ">=18" + } + }, "node_modules/rc": { "version": "1.2.8", "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", @@ -370,6 +1005,15 @@ "node": ">= 6" } }, + "node_modules/require-directory": { + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/require-directory/-/require-directory-2.1.1.tgz", + "integrity": "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q==", + "license": "MIT", + "engines": { + "node": ">=0.10.0" + } + }, "node_modules/safe-buffer": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", @@ -447,6 +1091,65 @@ "simple-concat": "^1.0.0" } }, + "node_modules/smart-buffer": { + "version": "4.2.0", + "resolved": "https://registry.npmjs.org/smart-buffer/-/smart-buffer-4.2.0.tgz", + "integrity": "sha512-94hK0Hh8rPqQl2xXc3HsaBoOXKV20MToPkcXvwbISWLEs+64sBq5kFgn2kJDHb1Pry9yrP0dxrCI9RRci7RXKg==", + "license": "MIT", + "engines": { + "node": ">= 6.0.0", + "npm": ">= 3.0.0" + } + }, + "node_modules/socks": { + "version": "2.8.7", + "resolved": "https://registry.npmjs.org/socks/-/socks-2.8.7.tgz", + "integrity": "sha512-HLpt+uLy/pxB+bum/9DzAgiKS8CX1EvbWxI4zlmgGCExImLdiad2iCwXT5Z4c9c3Eq8rP2318mPW2c+QbtjK8A==", + "license": "MIT", + "dependencies": { + "ip-address": "^10.0.1", + "smart-buffer": "^4.2.0" + }, + "engines": { + "node": ">= 10.0.0", + "npm": ">= 3.0.0" + } + }, + "node_modules/socks-proxy-agent": { + "version": "8.0.5", + "resolved": "https://registry.npmjs.org/socks-proxy-agent/-/socks-proxy-agent-8.0.5.tgz", + "integrity": "sha512-HehCEsotFqbPW9sJ8WVYB6UbmIMv7kUUORIF2Nncq4VQvBfNBLibW9YZR5dlYCSUhwcD628pRllm7n+E+YTzJw==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.2", + "debug": "^4.3.4", + "socks": "^2.8.3" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/source-map": { + "version": "0.6.1", + "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", + "integrity": "sha512-UjgapumWlbMhkBgzT7Ykc5YXUT46F0iKu8SGXq0bcwP5dz/h0Plj6enJqjz1Zbq2l5WaqYnrVbwWOWMyF3F47g==", + "license": "BSD-3-Clause", + "optional": true, + "engines": { + "node": ">=0.10.0" + } + }, + "node_modules/streamx": { + "version": "2.23.0", + "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.23.0.tgz", + "integrity": "sha512-kn+e44esVfn2Fa/O0CPFcex27fjIL6MkVae0Mm6q+E6f0hWv578YCERbv+4m02cjxvDsPKLnmxral/rR6lBMAg==", + "license": "MIT", + "dependencies": { + "events-universal": "^1.0.0", + "fast-fifo": "^1.3.2", + "text-decoder": "^1.1.0" + } + }, "node_modules/string_decoder": { "version": "1.3.0", "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", @@ -456,6 +1159,32 @@ "safe-buffer": "~5.2.0" } }, + "node_modules/string-width": { + "version": "4.2.3", + "resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz", + "integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==", + "license": "MIT", + "dependencies": { + "emoji-regex": "^8.0.0", + "is-fullwidth-code-point": "^3.0.0", + "strip-ansi": "^6.0.1" + }, + "engines": { + "node": ">=8" + } + }, + "node_modules/strip-ansi": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz", + "integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==", + "license": "MIT", + "dependencies": { + "ansi-regex": "^5.0.1" + }, + "engines": { + "node": ">=8" + } + }, "node_modules/strip-json-comments": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", @@ -493,6 +1222,21 @@ "node": ">=6" } }, + "node_modules/text-decoder": { + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.3.tgz", + "integrity": "sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==", + "license": "Apache-2.0", + "dependencies": { + "b4a": "^1.6.4" + } + }, + "node_modules/tslib": { + "version": "2.8.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", + "integrity": "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w==", + "license": "0BSD" + }, "node_modules/tunnel-agent": { "version": "0.6.0", "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", @@ -505,6 +1249,12 @@ "node": "*" } }, + "node_modules/typed-query-selector": { + "version": "2.12.0", + "resolved": "https://registry.npmjs.org/typed-query-selector/-/typed-query-selector-2.12.0.tgz", + "integrity": "sha512-SbklCd1F0EiZOyPiW192rrHZzZ5sBijB6xM+cpmrwDqObvdtunOHHIk9fCGsoK5JVIYXoyEp4iEdE3upFH3PAg==", + "license": "MIT" + }, "node_modules/typescript": { "version": "5.9.3", "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz", @@ -523,7 +1273,7 @@ "version": "6.21.0", "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz", "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==", - "dev": true, + "devOptional": true, "license": "MIT" }, "node_modules/util-deprecate": { @@ -532,11 +1282,110 @@ "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", "license": "MIT" }, + "node_modules/webdriver-bidi-protocol": { + "version": "0.3.8", + "resolved": "https://registry.npmjs.org/webdriver-bidi-protocol/-/webdriver-bidi-protocol-0.3.8.tgz", + "integrity": "sha512-21Yi2GhGntMc671vNBCjiAeEVknXjVRoyu+k+9xOMShu+ZQfpGQwnBqbNz/Sv4GXZ6JmutlPAi2nIJcrymAWuQ==", + "license": "Apache-2.0" + }, + "node_modules/wrap-ansi": { + "version": "7.0.0", + "resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz", + "integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==", + "license": "MIT", + "dependencies": { + "ansi-styles": "^4.0.0", + "string-width": "^4.1.0", + "strip-ansi": "^6.0.0" + }, + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/chalk/wrap-ansi?sponsor=1" + } + }, "node_modules/wrappy": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", "license": "ISC" + }, + "node_modules/ws": { + "version": "8.18.3", + "resolved": "https://registry.npmjs.org/ws/-/ws-8.18.3.tgz", + "integrity": "sha512-PEIGCY5tSlUt50cqyMXfCzX+oOPqN0vuGqWzbcJ2xvnkzkq46oOpz7dQaTDBdfICb4N14+GARUDw2XV2N4tvzg==", + "license": "MIT", + "engines": { + "node": ">=10.0.0" + }, + "peerDependencies": { + "bufferutil": "^4.0.1", + "utf-8-validate": ">=5.0.2" + }, + "peerDependenciesMeta": { + "bufferutil": { + "optional": true + }, + "utf-8-validate": { + "optional": true + } + } + }, + "node_modules/y18n": { + "version": "5.0.8", + "resolved": "https://registry.npmjs.org/y18n/-/y18n-5.0.8.tgz", + "integrity": "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA==", + "license": "ISC", + "engines": { + "node": ">=10" + } + }, + "node_modules/yargs": { + "version": "17.7.2", + "resolved": "https://registry.npmjs.org/yargs/-/yargs-17.7.2.tgz", + "integrity": "sha512-7dSzzRQ++CKnNI/krKnYRV7JKKPUXMEh61soaHKg9mrWEhzFWhFnxPxGl+69cD1Ou63C13NUPCnmIcrvqCuM6w==", + "license": "MIT", + "dependencies": { + "cliui": "^8.0.1", + "escalade": "^3.1.1", + "get-caller-file": "^2.0.5", + "require-directory": "^2.1.1", + "string-width": "^4.2.3", + "y18n": "^5.0.5", + "yargs-parser": "^21.1.1" + }, + "engines": { + "node": ">=12" + } + }, + "node_modules/yargs-parser": { + "version": "21.1.1", + "resolved": "https://registry.npmjs.org/yargs-parser/-/yargs-parser-21.1.1.tgz", + "integrity": "sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==", + "license": "ISC", + "engines": { + "node": ">=12" + } + }, + "node_modules/yauzl": { + "version": "2.10.0", + "resolved": "https://registry.npmjs.org/yauzl/-/yauzl-2.10.0.tgz", + "integrity": "sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g==", + "license": "MIT", + "dependencies": { + "buffer-crc32": "~0.2.3", + "fd-slicer": "~1.1.0" + } + }, + "node_modules/zod": { + "version": "3.25.76", + "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", + "integrity": "sha512-gzUt/qt81nXsFGKIFcC3YnfEAx5NkunCfnDlvuBSSFS02bcXu4Lmea0AFIUwbLWxWPx3d9p8S5QoaujKcNQxcQ==", + "license": "MIT", + "funding": { + "url": "https://github.com/sponsors/colinhacks" + } } } } diff --git a/archivebox-ts/package.json b/archivebox-ts/package.json index 6258b391..1b4ca1bd 100644 --- a/archivebox-ts/package.json +++ b/archivebox-ts/package.json @@ -22,7 +22,8 @@ "dependencies": { "better-sqlite3": "^11.0.0", "commander": "^12.0.0", - "nanoid": "^3.3.7" + "nanoid": "^3.3.7", + "puppeteer-core": "^24.28.0" }, "devDependencies": { "@types/better-sqlite3": "^7.6.9",