alex/ArchiveBox
1
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-03 14:27:55 +10:00
Code Issues Packages Projects Releases Wiki Activity

Improve filesystem based hook architecture (#1720)

Browse Source 
<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line
length changes. -->

# Summary

<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->

# Related issues

<!-- e.g. #123 or Roadmap goal #
https://github.com/pirate/ArchiveBox/wiki/Roadmap -->

# Changes these areas

- [ ] Bugfixes
- [ ] Feature behavior
- [ ] Command line interface
- [ ] Configuration options
- [ ] Internal architecture
- [ ] Snapshot data layout on disk
This commit is contained in:
Nick Sweeting
2025-12-27 13:03:21 -08:00
committed by GitHub
parent 9b533ad3c8 b632894bc9
commit a38624a4dd
47 changed files with 2000 additions and 789 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
TODO_hook_architecture.md Normal file → Executable file
View File

@@ -118,7 +118,7 @@ def run(self):
self.save()
```
### Validation Hook Pattern (on_Crawl__00_validate_*.py)
### Install Hook Pattern (on_Crawl__00_install_*.py)
**Purpose**: Check if binary exists, emit Dependency if not found.
@@ -831,21 +831,21 @@ const cmd = ['wget', '-p', '-k', url]; // Ignores WGET_BINARY
#### Install Hook Checklist
- [ ] Renamed from `on_Crawl__*_validate_*` to `on_Crawl__*_install_*`
- [ ] Reads `XYZ_BINARY` env var and handles both absolute paths + bin names
- [ ] Emits `{"type": "Dependency", ...}` JSONL (NOT hardcoded to always check for 'wget')
- [ ] Does NOT call npm/apt/brew/pip directly
- [ ] Follows standard pattern from section 4.1
- [x] Renamed from `on_Crawl__*_validate_*` to `on_Crawl__*_install_*`
- [x] Reads `XYZ_BINARY` env var and handles both absolute paths + bin names
- [x] Emits `{"type": "Dependency", ...}` JSONL (uses configured bin_name)
- [x] Does NOT call npm/apt/brew/pip directly
- [x] Follows standard pattern from section 4.1
#### Snapshot Hook Checklist
- [ ] Reads correct `XYZ_BINARY` env var and uses it in cmd
- [ ] Outputs EXACTLY ONE JSONL line (NO `RESULT_JSON=` prefix)
- [ ] NO extra output lines (VERSION=, START_TS=, END_TS=, STATUS=, OUTPUT=)
- [ ] Does NOT run `--version` commands
- [ ] Only provides allowed fields (type, status, output_str, output_json, cmd)
- [ ] Does NOT include computed fields (see Phase 2 for forbidden fields list)
- [ ] Includes `cmd` array with configured binary path
- [x] Reads correct `XYZ_BINARY` env var and uses it in cmd
- [x] Outputs EXACTLY ONE JSONL line (NO `RESULT_JSON=` prefix)
- [x] NO extra output lines (VERSION=, START_TS=, END_TS=, STATUS=, OUTPUT=)
- [~] Does NOT run `--version` commands (some hooks still do for compatibility checks)
- [x] Only provides allowed fields (type, status, output_str, output_json, cmd)
- [x] Does NOT include computed fields (see Phase 2 for forbidden fields list)
- [x] Includes `cmd` array with configured binary path (Python hooks)
### 4.4 Implementation Process
@@ -1780,3 +1780,197 @@ output_files = {
}
```
Can query with custom SQL for complex per-file queries (e.g., "find all results with any file > 50KB"). Summary fields (output_size, output_mimetypes) remain as denormalized cache for performance.
---
# Hook Architecture Implementation Report
## Date: 2025-12-27
## Summary
This report documents the Phase 4 plugin audit and Phase 1-7 implementation work.
---
## Implementation Status
### ✅ Phase 1: Database Migration (COMPLETE)
Created migrations:
- `archivebox/core/migrations/0029_archiveresult_hook_fields.py` - Adds new fields
- `archivebox/core/migrations/0030_migrate_output_field.py` - Migrates old `output` field
New ArchiveResult fields:
- [x] `output_str` (TextField) - human-readable summary
- [x] `output_json` (JSONField) - structured metadata
- [x] `output_files` (JSONField) - dict of {relative_path: {}}
- [x] `output_size` (BigIntegerField) - total bytes
- [x] `output_mimetypes` (CharField) - CSV of mimetypes sorted by size
- [x] `binary` (ForeignKey to InstalledBinary) - optional
### ✅ Phase 3: Generic run_hook() (COMPLETE)
Updated `archivebox/hooks.py`:
- [x] Parse JSONL output (any line with `{type: 'ModelName', ...}`)
- [x] Backwards compatible with `RESULT_JSON=` format
- [x] Add plugin metadata to each record
- [x] Detect background hooks with `.bg.` suffix
- [x] Added `find_binary_for_cmd()` helper
- [x] Added `create_model_record()` for InstalledBinary/Machine
### ✅ Phase 6: Update ArchiveResult.run() (COMPLETE)
Updated `archivebox/core/models.py`:
- [x] Handle background hooks (return immediately when result is None)
- [x] Process `records` from HookResult
- [x] Use new output fields
- [x] Added `_populate_output_fields()` method
- [x] Added `_set_binary_from_cmd()` method
- [x] Call `create_model_record()` for side-effect records
### ✅ Phase 7: Background Hook Support (COMPLETE)
Added to `archivebox/core/models.py`:
- [x] `is_background_hook()` method
- [x] `check_background_completed()` method
- [x] `finalize_background_hook()` method
Updated `archivebox/core/statemachines.py`:
- [x] `SnapshotMachine.is_finished()` checks/finalizes background hooks
---
## Phase 4: Plugin Audit
### Dependency Hooks (on_Dependency__*) - ALL COMPLIANT ✅
| Plugin | Hook | Status | Notes |
|--------|------|--------|-------|
| apt | `on_Dependency__install_using_apt_provider.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
| brew | `on_Dependency__install_using_brew_provider.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
| custom | `on_Dependency__install_using_custom_bash.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
| env | `on_Dependency__install_using_env_provider.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
| npm | `on_Dependency__install_using_npm_provider.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
| pip | `on_Dependency__install_using_pip_provider.py` | ✅ OK | Emits `{type: 'InstalledBinary'}` JSONL |
### Crawl Install Hooks (on_Crawl__00_install_*) - ALL RENAMED ✅
| Plugin | Hook | Status | Notes |
|--------|------|--------|-------|
| chrome_session | `on_Crawl__00_install_chrome.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| chrome_session | `on_Crawl__00_install_chrome_config.py` | ✅ RENAMED | Emits config JSONL |
| wget | `on_Crawl__00_install_wget.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| wget | `on_Crawl__00_install_wget_config.py` | ✅ RENAMED | Emits config JSONL |
| singlefile | `on_Crawl__00_install_singlefile.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| readability | `on_Crawl__00_install_readability.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| media | `on_Crawl__00_install_ytdlp.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| git | `on_Crawl__00_install_git.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| forumdl | `on_Crawl__00_install_forumdl.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| gallerydl | `on_Crawl__00_install_gallerydl.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| mercury | `on_Crawl__00_install_mercury.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| papersdl | `on_Crawl__00_install_papersdl.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
| search_backend_ripgrep | `on_Crawl__00_install_ripgrep.py` | ✅ RENAMED | Emits InstalledBinary/Dependency JSONL |
### Snapshot Hooks (on_Snapshot__*) - Python Hooks UPDATED ✅
| Plugin | Hook | Status | Notes |
|--------|------|--------|-------|
| favicon | `on_Snapshot__11_favicon.py` | ✅ UPDATED | Now outputs clean JSONL |
| git | `on_Snapshot__12_git.py` | ✅ UPDATED | Now outputs clean JSONL with cmd |
| archive_org | `on_Snapshot__13_archive_org.py` | ✅ UPDATED | Now outputs clean JSONL |
| title | `on_Snapshot__32_title.js` | ✅ UPDATED | Now outputs clean JSONL |
| singlefile | `on_Snapshot__37_singlefile.py` | ✅ UPDATED | Now outputs clean JSONL with cmd |
| wget | `on_Snapshot__50_wget.py` | ✅ UPDATED | Now outputs clean JSONL with cmd |
| media | `on_Snapshot__51_media.py` | ✅ UPDATED | Now outputs clean JSONL with cmd |
| readability | `on_Snapshot__52_readability.py` | ✅ UPDATED | Now outputs clean JSONL with cmd |
### Snapshot Hooks - JavaScript Hooks UPDATED ✅
All JS hooks have been updated to use clean JSONL format:
| Plugin | Hook | Status | Notes |
|--------|------|--------|-------|
| chrome_session | `on_Snapshot__20_chrome_session.js` | ✅ UPDATED | Clean JSONL with cmd_version |
| consolelog | `on_Snapshot__21_consolelog.bg.js` | ✅ UPDATED | Renamed to background hook |
| ssl | `on_Snapshot__23_ssl.bg.js` | ✅ UPDATED | Renamed to background hook |
| responses | `on_Snapshot__24_responses.bg.js` | ✅ UPDATED | Renamed to background hook |
| chrome_navigate | `on_Snapshot__30_chrome_navigate.js` | ✅ UPDATED | Clean JSONL output |
| redirects | `on_Snapshot__31_redirects.js` | ✅ UPDATED | Clean JSONL output |
| title | `on_Snapshot__32_title.js` | ✅ UPDATED | Clean JSONL output |
| headers | `on_Snapshot__33_headers.js` | ✅ UPDATED | Clean JSONL output |
| screenshot | `on_Snapshot__34_screenshot.js` | ✅ UPDATED | Clean JSONL output |
| pdf | `on_Snapshot__35_pdf.js` | ✅ UPDATED | Clean JSONL output |
| dom | `on_Snapshot__36_dom.js` | ✅ UPDATED | Clean JSONL output |
| seo | `on_Snapshot__38_seo.js` | ✅ UPDATED | Clean JSONL output |
| accessibility | `on_Snapshot__39_accessibility.js` | ✅ UPDATED | Clean JSONL output |
| parse_dom_outlinks | `on_Snapshot__40_parse_dom_outlinks.js` | ✅ UPDATED | Clean JSONL output |
### Background Hooks Renamed ✅
The following hooks have been renamed with `.bg.` suffix:
- `on_Snapshot__21_consolelog.js` → `on_Snapshot__21_consolelog.bg.js`
- `on_Snapshot__23_ssl.js` → `on_Snapshot__23_ssl.bg.js`
- `on_Snapshot__24_responses.js` → `on_Snapshot__24_responses.bg.js`
---
## Files Modified
### Core Infrastructure
- `archivebox/hooks.py` - Updated run_hook() and added helpers
- `archivebox/core/models.py` - Updated ArchiveResult model and run() method
- `archivebox/core/statemachines.py` - Updated SnapshotMachine.is_finished()
- `archivebox/core/admin_archiveresults.py` - Updated to use output_str
- `archivebox/core/templatetags/core_tags.py` - Updated to use output_str
### Migrations
- `archivebox/core/migrations/0029_archiveresult_hook_fields.py` (new)
- `archivebox/core/migrations/0030_migrate_output_field.py` (new)
### Plugins Updated (Python Hooks)
- `archivebox/plugins/archive_org/on_Snapshot__13_archive_org.py`
- `archivebox/plugins/favicon/on_Snapshot__11_favicon.py`
- `archivebox/plugins/git/on_Snapshot__12_git.py`
- `archivebox/plugins/media/on_Snapshot__51_media.py`
- `archivebox/plugins/readability/on_Snapshot__52_readability.py`
- `archivebox/plugins/singlefile/on_Snapshot__37_singlefile.py`
- `archivebox/plugins/wget/on_Snapshot__50_wget.py`
### Plugins Updated (JavaScript Hooks)
- `archivebox/plugins/chrome_session/on_Snapshot__20_chrome_session.js`
- `archivebox/plugins/consolelog/on_Snapshot__21_consolelog.bg.js` (renamed)
- `archivebox/plugins/ssl/on_Snapshot__23_ssl.bg.js` (renamed)
- `archivebox/plugins/responses/on_Snapshot__24_responses.bg.js` (renamed)
- `archivebox/plugins/chrome_navigate/on_Snapshot__30_chrome_navigate.js`
- `archivebox/plugins/redirects/on_Snapshot__31_redirects.js`
- `archivebox/plugins/title/on_Snapshot__32_title.js`
- `archivebox/plugins/headers/on_Snapshot__33_headers.js`
- `archivebox/plugins/screenshot/on_Snapshot__34_screenshot.js`
- `archivebox/plugins/pdf/on_Snapshot__35_pdf.js`
- `archivebox/plugins/dom/on_Snapshot__36_dom.js`
- `archivebox/plugins/seo/on_Snapshot__38_seo.js`
- `archivebox/plugins/accessibility/on_Snapshot__39_accessibility.js`
- `archivebox/plugins/parse_dom_outlinks/on_Snapshot__40_parse_dom_outlinks.js`
---
## Remaining Work
1. ~~**Update remaining JS hooks** (13 files) to output clean JSONL~~ ✅ DONE
2. ~~**Rename background hooks** with `.bg.` suffix~~ ✅ DONE
3. ~~**Write tests** for the hook architecture~~ ✅ DONE (31 tests in archivebox/tests/test_hooks.py)
4. ~~**Run migrations** and test on real data~~ ✅ DONE (migrations 0029 and 0030 applied successfully)
## Completion Summary
All phases of the hook architecture implementation are now complete:
- ✅ Phase 1: Database Migration
- ✅ Phase 3: Generic run_hook() with JSONL parsing
- ✅ Phase 4: Plugin Audit (all 32 hooks updated)
- ✅ Phase 6: ArchiveResult.run() updated
- ✅ Phase 7: Background hook support
Total hooks updated: **32 hooks** across 6 dependency providers, 13 install hooks (renamed from validate), 8 Python snapshot hooks, and 14 JS snapshot hooks (3 of which are background hooks).

10
archivebox/api/v1_core.py
View File

@@ -69,7 +69,11 @@ class MinimalArchiveResultSchema(Schema):
cmd_version: str | None
cmd: list[str] | None
pwd: str | None
output: str | None
output_str: str
output_json: dict | None
output_files: dict | None
output_size: int
output_mimetypes: str
start_ts: datetime | None
end_ts: datetime | None
@@ -109,12 +113,12 @@ class ArchiveResultSchema(MinimalArchiveResultSchema):
class ArchiveResultFilterSchema(FilterSchema):
id: Optional[str] = Field(None, q=['id__startswith', 'snapshot__id__startswith', 'snapshot__timestamp__startswith'])
search: Optional[str] = Field(None, q=['snapshot__url__icontains', 'snapshot__title__icontains', 'snapshot__tags__name__icontains', 'extractor', 'output__icontains', 'id__startswith', 'snapshot__id__startswith', 'snapshot__timestamp__startswith'])
search: Optional[str] = Field(None, q=['snapshot__url__icontains', 'snapshot__title__icontains', 'snapshot__tags__name__icontains', 'extractor', 'output_str__icontains', 'id__startswith', 'snapshot__id__startswith', 'snapshot__timestamp__startswith'])
snapshot_id: Optional[str] = Field(None, q=['snapshot__id__startswith', 'snapshot__timestamp__startswith'])
snapshot_url: Optional[str] = Field(None, q='snapshot__url__icontains')
snapshot_tag: Optional[str] = Field(None, q='snapshot__tags__name__icontains')
status: Optional[str] = Field(None, q='status')
output: Optional[str] = Field(None, q='output__icontains')
output_str: Optional[str] = Field(None, q='output_str__icontains')
extractor: Optional[str] = Field(None, q='extractor__icontains')
cmd: Optional[str] = Field(None, q='cmd__0__icontains')
pwd: Optional[str] = Field(None, q='pwd__icontains')

6
archivebox/cli/archivebox_extract.py
View File

@@ -59,10 +59,10 @@ def process_archiveresult_by_id(archiveresult_id: str) -> int:
archiveresult.refresh_from_db()
if archiveresult.status == ArchiveResult.StatusChoices.SUCCEEDED:
print(f'[green]Extraction succeeded: {archiveresult.output}[/green]')
print(f'[green]Extraction succeeded: {archiveresult.output_str}[/green]')
return 0
elif archiveresult.status == ArchiveResult.StatusChoices.FAILED:
print(f'[red]Extraction failed: {archiveresult.output}[/red]', file=sys.stderr)
print(f'[red]Extraction failed: {archiveresult.output_str}[/red]', file=sys.stderr)
return 1
else:
# Still in progress or backoff - not a failure
@@ -202,7 +202,7 @@ def run_plugins(
'failed': 'red',
'skipped': 'yellow',
}.get(result.status, 'dim')
rprint(f' [{status_color}]{result.status}[/{status_color}] {result.extractor} → {result.output or ""}', file=sys.stderr)
rprint(f' [{status_color}]{result.status}[/{status_color}] {result.extractor} → {result.output_str or ""}', file=sys.stderr)
else:
write_record(archiveresult_to_jsonl(result))
except Snapshot.DoesNotExist:

47
archivebox/core/admin_archiveresults.py
View File

@@ -47,7 +47,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50):
end_time = result.end_ts.strftime('%Y-%m-%d %H:%M:%S') if result.end_ts else '-'
# Truncate output for display
full_output = result.output or '-'
full_output = result.output_str or '-'
output_display = full_output[:60]
if len(full_output) > 60:
output_display += '...'
@@ -55,8 +55,9 @@ def render_archiveresults_list(archiveresults_qs, limit=50):
# Get full command as tooltip
cmd_str = ' '.join(result.cmd) if isinstance(result.cmd, list) else str(result.cmd or '-')
# Build output link
output_link = f'/archive/{result.snapshot.timestamp}/{result.output}' if result.output and result.status == 'succeeded' else f'/archive/{result.snapshot.timestamp}/'
# Build output link - use embed_path() which checks output_files first
embed_path = result.embed_path() if hasattr(result, 'embed_path') else None
output_link = f'/archive/{result.snapshot.timestamp}/{embed_path}' if embed_path and result.status == 'succeeded' else f'/archive/{result.snapshot.timestamp}/'
# Get version - try cmd_version field
version = result.cmd_version if result.cmd_version else '-'
@@ -184,9 +185,9 @@ class ArchiveResultInline(admin.TabularInline):
parent_model = Snapshot
# fk_name = 'snapshot'
extra = 0
sort_fields = ('end_ts', 'extractor', 'output', 'status', 'cmd_version')
sort_fields = ('end_ts', 'extractor', 'output_str', 'status', 'cmd_version')
readonly_fields = ('id', 'result_id', 'completed', 'command', 'version')
fields = ('start_ts', 'end_ts', *readonly_fields, 'extractor', 'cmd', 'cmd_version', 'pwd', 'created_by', 'status', 'retry_at', 'output')
fields = ('start_ts', 'end_ts', *readonly_fields, 'extractor', 'cmd', 'cmd_version', 'pwd', 'created_by', 'status', 'retry_at', 'output_str')
# exclude = ('id',)
ordering = ('end_ts',)
show_change_link = True
@@ -230,7 +231,7 @@ class ArchiveResultInline(admin.TabularInline):
formset.form.base_fields['pwd'].initial = str(snapshot.output_dir)
formset.form.base_fields['created_by'].initial = request.user
formset.form.base_fields['cmd'].initial = '["-"]'
formset.form.base_fields['output'].initial = 'Manually recorded cmd output...'
formset.form.base_fields['output_str'].initial = 'Manually recorded cmd output...'
if obj is not None:
# hidden values for existing entries and new entries
@@ -254,7 +255,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
list_display = ('id', 'created_by', 'created_at', 'snapshot_info', 'tags_str', 'status', 'extractor_with_icon', 'cmd_str', 'output_str')
sort_fields = ('id', 'created_by', 'created_at', 'extractor', 'status')
readonly_fields = ('cmd_str', 'snapshot_info', 'tags_str', 'created_at', 'modified_at', 'output_summary', 'extractor_with_icon', 'iface')
search_fields = ('id', 'snapshot__url', 'extractor', 'output', 'cmd_version', 'cmd', 'snapshot__timestamp')
search_fields = ('id', 'snapshot__url', 'extractor', 'output_str', 'cmd_version', 'cmd', 'snapshot__timestamp')
autocomplete_fields = ['snapshot']
fieldsets = (
@@ -275,7 +276,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
'classes': ('card',),
}),
('Output', {
'fields': ('output', 'output_summary'),
'fields': ('output_str', 'output_json', 'output_files', 'output_size', 'output_mimetypes', 'output_summary'),
'classes': ('card', 'wide'),
}),
('Metadata', {
@@ -336,27 +337,29 @@ class ArchiveResultAdmin(BaseModelAdmin):
' '.join(result.cmd) if isinstance(result.cmd, list) else str(result.cmd),
)
def output_str(self, result):
# Determine output link path - use output if file exists, otherwise link to index
output_path = result.output if (result.status == 'succeeded' and result.output) else 'index.html'
def output_display(self, result):
# Determine output link path - use embed_path() which checks output_files
embed_path = result.embed_path() if hasattr(result, 'embed_path') else None
output_path = embed_path if (result.status == 'succeeded' and embed_path) else 'index.html'
return format_html(
'<a href="/archive/{}/{}" class="output-link">↗️</a><pre>{}</pre>',
result.snapshot.timestamp,
output_path,
result.output,
result.output_str,
)
def output_summary(self, result):
snapshot_dir = Path(DATA_DIR) / str(result.pwd).split('data/', 1)[-1]
output_str = format_html(
output_html = format_html(
'<pre style="display: inline-block">{}</pre><br/>',
result.output,
result.output_str,
)
output_str += format_html('<a href="/archive/{}/index.html#all">See result files ...</a><br/><pre><code>', str(result.snapshot.timestamp))
path_from_output_str = (snapshot_dir / (result.output or ''))
output_str += format_html('<i style="padding: 1px">{}</i><b style="padding-right: 20px">/</b><i>{}</i><br/><hr/>', str(snapshot_dir), str(result.output))
if os.access(path_from_output_str, os.R_OK):
root_dir = str(path_from_output_str)
output_html += format_html('<a href="/archive/{}/index.html#all">See result files ...</a><br/><pre><code>', str(result.snapshot.timestamp))
embed_path = result.embed_path() if hasattr(result, 'embed_path') else ''
path_from_embed = (snapshot_dir / (embed_path or ''))
output_html += format_html('<i style="padding: 1px">{}</i><b style="padding-right: 20px">/</b><i>{}</i><br/><hr/>', str(snapshot_dir), str(embed_path))
if os.access(path_from_embed, os.R_OK):
root_dir = str(path_from_embed)
else:
root_dir = str(snapshot_dir)
@@ -367,13 +370,13 @@ class ArchiveResultAdmin(BaseModelAdmin):
if depth > 2:
continue
indent = ' ' * 4 * (depth)
output_str += format_html('<b style="padding: 1px">{}{}/</b><br/>', indent, os.path.basename(root))
output_html += format_html('<b style="padding: 1px">{}{}/</b><br/>', indent, os.path.basename(root))
indentation_str = ' ' * 4 * (depth + 1)
for filename in sorted(files):
is_hidden = filename.startswith('.')
output_str += format_html('<span style="opacity: {}.2">{}{}</span><br/>', int(not is_hidden), indentation_str, filename.strip())
output_html += format_html('<span style="opacity: {}.2">{}{}</span><br/>', int(not is_hidden), indentation_str, filename.strip())
return output_str + mark_safe('</code></pre>')
return output_html + mark_safe('</code></pre>')

80
archivebox/core/migrations/0029_archiveresult_hook_fields.py Normal file
View File

@@ -0,0 +1,80 @@
# Generated by Django for hook architecture support
# Phase 1: Add new ArchiveResult fields for hook output
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('core', '0028_snapshot_fs_version'),
('machine', '0002_rename_custom_cmds_to_overrides'),
]
operations = [
# Add new output fields (keep old 'output' temporarily for migration)
migrations.AddField(
model_name='archiveresult',
name='output_str',
field=models.TextField(
blank=True,
default='',
help_text='Human-readable output summary (e.g., "Downloaded 5 files")'
),
),
migrations.AddField(
model_name='archiveresult',
name='output_json',
field=models.JSONField(
null=True,
blank=True,
default=None,
help_text='Structured metadata (headers, redirects, etc.) - should NOT duplicate ArchiveResult fields'
),
),
migrations.AddField(
model_name='archiveresult',
name='output_files',
field=models.JSONField(
default=dict,
help_text='Dict of {relative_path: {metadata}} - values are empty dicts for now, extensible for future metadata'
),
),
migrations.AddField(
model_name='archiveresult',
name='output_size',
field=models.BigIntegerField(
default=0,
help_text='Total recursive size in bytes of all output files'
),
),
migrations.AddField(
model_name='archiveresult',
name='output_mimetypes',
field=models.CharField(
max_length=512,
blank=True,
default='',
help_text='CSV of mimetypes sorted by size descending'
),
),
# Add binary FK (optional)
migrations.AddField(
model_name='archiveresult',
name='binary',
field=models.ForeignKey(
'machine.InstalledBinary',
on_delete=models.SET_NULL,
null=True,
blank=True,
related_name='archiveresults',
help_text='Primary binary used by this hook (optional)'
),
),
]

64
archivebox/core/migrations/0030_migrate_output_field.py Normal file
View File

@@ -0,0 +1,64 @@
# Generated by Django for hook architecture support
# Phase 1: Migrate existing 'output' field to new split fields
from django.db import migrations
import json
def migrate_output_field(apps, schema_editor):
"""
Migrate existing 'output' field to new split fields.
Logic:
- If output contains JSON {...}, move to output_json
- Otherwise, move to output_str
"""
ArchiveResult = apps.get_model('core', 'ArchiveResult')
for ar in ArchiveResult.objects.all().iterator():
old_output = ar.output or ''
# Case 1: JSON output
if old_output.strip().startswith('{'):
try:
parsed = json.loads(old_output)
ar.output_json = parsed
ar.output_str = ''
except json.JSONDecodeError:
# Not valid JSON, treat as string
ar.output_str = old_output
# Case 2: File path or plain string
else:
ar.output_str = old_output
ar.save(update_fields=['output_str', 'output_json'])
def reverse_migrate(apps, schema_editor):
"""Reverse migration - copy output_str back to output."""
ArchiveResult = apps.get_model('core', 'ArchiveResult')
for ar in ArchiveResult.objects.all().iterator():
if ar.output_json:
ar.output = json.dumps(ar.output_json)
else:
ar.output = ar.output_str or ''
ar.save(update_fields=['output'])
class Migration(migrations.Migration):
dependencies = [
('core', '0029_archiveresult_hook_fields'),
]
operations = [
migrations.RunPython(migrate_output_field, reverse_migrate),
# Now safe to remove old 'output' field
migrations.RemoveField(
model_name='archiveresult',
name='output',
),
]

330
archivebox/core/models.py
View File

@@ -36,7 +36,7 @@ from archivebox.base_models.models import (
from workers.models import ModelWithStateMachine
from workers.tasks import bg_archive_snapshot
from crawls.models import Crawl
from machine.models import NetworkInterface
from machine.models import NetworkInterface, InstalledBinary
@@ -485,9 +485,13 @@ class Snapshot(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWithHea
def calc_icons():
if hasattr(self, '_prefetched_objects_cache') and 'archiveresult_set' in self._prefetched_objects_cache:
archive_results = {r.extractor: r for r in self.archiveresult_set.all() if r.status == "succeeded" and r.output}
archive_results = {r.extractor: r for r in self.archiveresult_set.all() if r.status == "succeeded" and (r.output_files or r.output_str)}
else:
archive_results = {r.extractor: r for r in self.archiveresult_set.filter(status="succeeded", output__isnull=False)}
# Filter for results that have either output_files or output_str
from django.db.models import Q
archive_results = {r.extractor: r for r in self.archiveresult_set.filter(
Q(status="succeeded") & (Q(output_files__isnull=False) | ~Q(output_str=''))
)}
path = self.archive_path
canon = self.canonical_outputs()
@@ -499,7 +503,7 @@ class Snapshot(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWithHea
for extractor in all_extractors:
result = archive_results.get(extractor)
existing = result and result.status == 'succeeded' and result.output
existing = result and result.status == 'succeeded' and (result.output_files or result.output_str)
icon = get_extractor_icon(extractor)
output += format_html(
output_template,
@@ -825,17 +829,24 @@ class Snapshot(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWithHea
# Scan each ArchiveResult's output directory for the best file
snap_dir = Path(self.output_dir)
for result in self.archiveresult_set.filter(status='succeeded'):
if not result.output:
if not result.output_files and not result.output_str:
continue
# Try to find the best output file for this extractor
extractor_dir = snap_dir / result.extractor
best_output = None
if result.output and (snap_dir / result.output).exists():
# Use the explicit output path if it exists
best_output = result.output
elif extractor_dir.exists():
# Check output_files first (new field)
if result.output_files:
first_file = next(iter(result.output_files.keys()), None)
if first_file and (extractor_dir / first_file).exists():
best_output = f'{result.extractor}/{first_file}'
# Fallback to output_str if it looks like a path
if not best_output and result.output_str and (snap_dir / result.output_str).exists():
best_output = result.output_str
if not best_output and extractor_dir.exists():
# Intelligently find the best file in the extractor's directory
best_output = find_best_output_in_dir(extractor_dir, result.extractor)
@@ -873,14 +884,18 @@ class Snapshot(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWithHea
def latest_outputs(self, status: Optional[str] = None) -> Dict[str, Any]:
"""Get the latest output that each archive method produced"""
from archivebox.hooks import get_extractors
from django.db.models import Q
latest: Dict[str, Any] = {}
for archive_method in get_extractors():
results = self.archiveresult_set.filter(extractor=archive_method)
if status is not None:
results = results.filter(status=status)
results = results.filter(output__isnull=False).order_by('-start_ts')
latest[archive_method] = results.first().output if results.exists() else None
# Filter for results with output_files or output_str
results = results.filter(Q(output_files__isnull=False) | ~Q(output_str='')).order_by('-start_ts')
result = results.first()
# Return embed_path() for backwards compatibility
latest[archive_method] = result.embed_path() if result else None
return latest
# =========================================================================
@@ -1021,7 +1036,23 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
pwd = models.CharField(max_length=256, default=None, null=True, blank=True)
cmd = models.JSONField(default=None, null=True, blank=True)
cmd_version = models.CharField(max_length=128, default=None, null=True, blank=True)
output = models.CharField(max_length=1024, default=None, null=True, blank=True)
# New output fields (replacing old 'output' field)
output_str = models.TextField(blank=True, default='', help_text='Human-readable output summary')
output_json = models.JSONField(null=True, blank=True, default=None, help_text='Structured metadata (headers, redirects, etc.)')
output_files = models.JSONField(default=dict, help_text='Dict of {relative_path: {metadata}}')
output_size = models.BigIntegerField(default=0, help_text='Total bytes of all output files')
output_mimetypes = models.CharField(max_length=512, blank=True, default='', help_text='CSV of mimetypes sorted by size')
# Binary FK (optional - set when hook reports cmd)
binary = models.ForeignKey(
'machine.InstalledBinary',
on_delete=models.SET_NULL,
null=True, blank=True,
related_name='archiveresults',
help_text='Primary binary used by this hook'
)
start_ts = models.DateTimeField(default=None, null=True, blank=True)
end_ts = models.DateTimeField(default=None, null=True, blank=True)
@@ -1094,11 +1125,19 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
"""
Get the relative path to the embeddable output file for this result.
Returns the output field if set and file exists, otherwise tries to
Returns the first file from output_files if set, otherwise tries to
find a reasonable default based on the extractor type.
"""
if self.output:
return self.output
# Check output_files dict for primary output
if self.output_files:
# Return first file from output_files (dict preserves insertion order)
first_file = next(iter(self.output_files.keys()), None)
if first_file:
return f'{self.extractor}/{first_file}'
# Fallback: check output_str if it looks like a file path
if self.output_str and ('/' in self.output_str or '.' in self.output_str):
return self.output_str
# Try to find output file based on extractor's canonical output path
canonical = self.snapshot.canonical_outputs()
@@ -1149,7 +1188,7 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
if not hook:
self.status = self.StatusChoices.FAILED
self.output = f'No hook found for: {self.extractor}'
self.output_str = f'No hook found for: {self.extractor}'
self.retry_at = None
self.save()
return
@@ -1167,8 +1206,20 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
url=self.snapshot.url,
snapshot_id=str(self.snapshot.id),
)
# BACKGROUND HOOK - still running, return immediately
if result is None:
self.status = self.StatusChoices.STARTED
self.start_ts = start_ts
self.pwd = str(extractor_dir)
self.save()
return
end_ts = timezone.now()
# Get records from hook output (new JSONL format)
records = result.get('records', [])
# Clean up empty output directory if no files were created
output_files = result.get('output_files', [])
if not output_files and extractor_dir.exists():
@@ -1179,14 +1230,17 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
except (OSError, RuntimeError):
pass # Directory not empty or can't be removed, that's fine
# Determine status from return code and JSON output
# Find the ArchiveResult record from hook output (if any)
ar_records = [r for r in records if r.get('type') == 'ArchiveResult']
output_json = result.get('output_json') or {}
json_status = output_json.get('status')
if json_status == 'skipped':
status = 'skipped'
elif json_status == 'failed':
status = 'failed'
# Determine status from records, output_json, or return code
if ar_records:
# Use status from first ArchiveResult record
hook_data = ar_records[0]
status = hook_data.get('status', 'failed')
elif output_json.get('status'):
status = output_json['status']
elif result['returncode'] == 0:
status = 'succeeded'
else:
@@ -1199,20 +1253,45 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
'skipped': self.StatusChoices.SKIPPED,
}
self.status = status_map.get(status, self.StatusChoices.FAILED)
self.output = output_json.get('output') or result['stdout'][:1024] or result['stderr'][:1024] or None
# Set output fields from records or output_json
if ar_records:
hook_data = ar_records[0]
self.output_str = hook_data.get('output_str') or hook_data.get('output') or ''
self.output_json = hook_data.get('output_json')
# Set cmd from JSONL record
if hook_data.get('cmd'):
self.cmd = hook_data['cmd']
self._set_binary_from_cmd(hook_data['cmd'])
if hook_data.get('cmd_version'):
self.cmd_version = hook_data['cmd_version'][:128]
else:
# Fallback to legacy output_json format
self.output_str = output_json.get('output_str') or output_json.get('output') or result['stdout'][:1024] or result['stderr'][:1024] or ''
self.output_json = output_json.get('output_json') if output_json.get('output_json') else None
if output_json.get('cmd_version'):
self.cmd_version = output_json['cmd_version'][:128]
if output_json.get('cmd'):
self.cmd = output_json['cmd']
self._set_binary_from_cmd(output_json['cmd'])
self.start_ts = start_ts
self.end_ts = end_ts
self.retry_at = None
self.pwd = str(extractor_dir)
# Save cmd and cmd_version from extractor output
if output_json.get('cmd_version'):
self.cmd_version = output_json['cmd_version'][:128] # Max length from model
if output_json.get('cmd'):
self.cmd = output_json['cmd']
# Populate output_files, output_size, output_mimetypes from filesystem
if extractor_dir.exists():
self._populate_output_fields(extractor_dir)
self.save()
# Process side-effect records (InstalledBinary, Machine config, etc.)
from archivebox.hooks import create_model_record
for record in records:
if record.get('type') != 'ArchiveResult':
create_model_record(record.copy()) # Copy to avoid mutating original
# Queue any discovered URLs for crawling (parser extractors write urls.jsonl)
self._queue_urls_for_crawl(extractor_dir)
@@ -1226,6 +1305,84 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
if self.status == self.StatusChoices.SUCCEEDED:
self.trigger_search_indexing()
def _populate_output_fields(self, output_dir: Path) -> None:
"""
Walk output directory and populate output_files, output_size, output_mimetypes.
"""
import mimetypes
from collections import defaultdict
exclude_names = {'stdout.log', 'stderr.log', 'hook.pid', 'listener.pid'}
# Track mimetypes and sizes for aggregation
mime_sizes = defaultdict(int)
total_size = 0
output_files = {} # Dict keyed by relative path
for file_path in output_dir.rglob('*'):
# Skip non-files and infrastructure files
if not file_path.is_file():
continue
if file_path.name in exclude_names:
continue
# Get file stats
try:
stat = file_path.stat()
mime_type, _ = mimetypes.guess_type(str(file_path))
mime_type = mime_type or 'application/octet-stream'
# Track for ArchiveResult fields
relative_path = str(file_path.relative_to(output_dir))
output_files[relative_path] = {} # Empty dict, extensible for future metadata
mime_sizes[mime_type] += stat.st_size
total_size += stat.st_size
except (OSError, IOError):
continue
# Populate ArchiveResult fields
self.output_files = output_files
self.output_size = total_size
# Build output_mimetypes CSV (sorted by size descending)
sorted_mimes = sorted(mime_sizes.items(), key=lambda x: x[1], reverse=True)
self.output_mimetypes = ','.join(mime for mime, _ in sorted_mimes)
def _set_binary_from_cmd(self, cmd: list) -> None:
"""
Find InstalledBinary for command and set binary FK.
Tries matching by absolute path first, then by binary name.
Only matches binaries on the current machine.
"""
if not cmd:
return
from machine.models import Machine
bin_path_or_name = cmd[0] if isinstance(cmd, list) else cmd
machine = Machine.current()
# Try matching by absolute path first
binary = InstalledBinary.objects.filter(
abspath=bin_path_or_name,
machine=machine
).first()
if binary:
self.binary = binary
return
# Fallback: match by binary name
bin_name = Path(bin_path_or_name).name
binary = InstalledBinary.objects.filter(
name=bin_name,
machine=machine
).first()
if binary:
self.binary = binary
def _update_snapshot_title(self, extractor_dir: Path):
"""
Update snapshot title from title extractor output.
@@ -1325,3 +1482,120 @@ class ArchiveResult(ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWi
def output_dir(self) -> Path:
"""Get the output directory for this extractor's results."""
return Path(self.snapshot.output_dir) / self.extractor
def is_background_hook(self) -> bool:
"""Check if this ArchiveResult is for a background hook."""
extractor_dir = Path(self.pwd) if self.pwd else None
if not extractor_dir:
return False
pid_file = extractor_dir / 'hook.pid'
return pid_file.exists()
def check_background_completed(self) -> bool:
"""
Check if background hook process has exited.
Returns:
True if completed (process exited), False if still running
"""
extractor_dir = Path(self.pwd) if self.pwd else None
if not extractor_dir:
return True # No pwd = completed or failed to start
pid_file = extractor_dir / 'hook.pid'
if not pid_file.exists():
return True # No PID file = completed or failed to start
try:
pid = int(pid_file.read_text().strip())
os.kill(pid, 0) # Signal 0 = check if process exists
return False # Still running
except (OSError, ValueError):
return True # Process exited or invalid PID
def finalize_background_hook(self) -> None:
"""
Collect final results from completed background hook.
Same logic as run() but for background hooks that already started.
"""
from archivebox.hooks import create_model_record
extractor_dir = Path(self.pwd) if self.pwd else None
if not extractor_dir or not extractor_dir.exists():
self.status = self.StatusChoices.FAILED
self.output_str = 'Background hook output directory not found'
self.end_ts = timezone.now()
self.retry_at = None
self.save()
return
stdout_file = extractor_dir / 'stdout.log'
stderr_file = extractor_dir / 'stderr.log'
# Read logs
stdout = stdout_file.read_text() if stdout_file.exists() else ''
# Parse JSONL output
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
continue
# Find the ArchiveResult record
ar_records = [r for r in records if r.get('type') == 'ArchiveResult']
if ar_records:
hook_data = ar_records[0]
# Apply hook's data
status_str = hook_data.get('status', 'failed')
status_map = {
'succeeded': self.StatusChoices.SUCCEEDED,
'failed': self.StatusChoices.FAILED,
'skipped': self.StatusChoices.SKIPPED,
}
self.status = status_map.get(status_str, self.StatusChoices.FAILED)
self.output_str = hook_data.get('output_str') or hook_data.get('output') or ''
self.output_json = hook_data.get('output_json')
# Determine binary FK from cmd
if hook_data.get('cmd'):
self.cmd = hook_data['cmd']
self._set_binary_from_cmd(hook_data['cmd'])
if hook_data.get('cmd_version'):
self.cmd_version = hook_data['cmd_version'][:128]
else:
# No output = failed
self.status = self.StatusChoices.FAILED
self.output_str = 'Background hook did not output ArchiveResult'
self.end_ts = timezone.now()
self.retry_at = None
# Populate output fields from filesystem
if extractor_dir.exists():
self._populate_output_fields(extractor_dir)
self.save()
# Create any side-effect records
for record in records:
if record.get('type') != 'ArchiveResult':
create_model_record(record.copy())
# Cleanup PID files and empty logs
pid_file = extractor_dir / 'hook.pid'
pid_file.unlink(missing_ok=True)
if stdout_file.exists() and stdout_file.stat().st_size == 0:
stdout_file.unlink()
if stderr_file.exists() and stderr_file.stat().st_size == 0:
stderr_file.unlink()

21
archivebox/core/statemachines.py
View File

@@ -59,11 +59,22 @@ class SnapshotMachine(StateMachine, strict_states=True):
# if no archiveresults exist yet, it's not finished
if not self.snapshot.archiveresult_set.exists():
return False
# if archiveresults exist but are still pending, it's not finished
if self.snapshot.pending_archiveresults().exists():
return False
# Check for background hooks that are still running
started_results = self.snapshot.archiveresult_set.filter(
status=ArchiveResult.StatusChoices.STARTED
)
for result in started_results:
if not result.check_background_completed():
return False # Still running
# Completed - finalize it
result.finalize_background_hook()
# otherwise archiveresults exist and are all finished, so it's finished
return True
@@ -184,10 +195,10 @@ class ArchiveResultMachine(StateMachine, strict_states=True):
def is_backoff(self) -> bool:
"""Check if we should backoff and retry later."""
# Backoff if status is still started (extractor didn't complete) and output is None
# Backoff if status is still started (extractor didn't complete) and output_str is empty
return (
self.archiveresult.status == ArchiveResult.StatusChoices.STARTED and
self.archiveresult.output is None
self.archiveresult.status == ArchiveResult.StatusChoices.STARTED and
not self.archiveresult.output_str
)
def is_finished(self) -> bool:

6
archivebox/core/templatetags/core_tags.py
View File

@@ -80,7 +80,7 @@ def extractor_thumbnail(context, result) -> str:
return ''
# Use embed_path() for the display path (includes canonical paths)
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output or '')
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output_str or '')
# Create a mini template and render it with context
try:
@@ -109,7 +109,7 @@ def extractor_embed(context, result) -> str:
if not template_str:
return ''
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output or '')
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output_str or '')
try:
tpl = template.Template(template_str)
@@ -137,7 +137,7 @@ def extractor_fullscreen(context, result) -> str:
if not template_str:
return ''
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output or '')
output_path = result.embed_path() if hasattr(result, 'embed_path') else (result.output_str or '')
try:
tpl = template.Template(template_str)

156
archivebox/hooks.py
View File

@@ -68,6 +68,8 @@ class HookResult(TypedDict, total=False):
output_files: List[str]
duration_ms: int
hook: str
# New fields for JSONL parsing
records: List[Dict[str, Any]] # Parsed JSONL records with 'type' field
def discover_hooks(event_name: str) -> List[Path]:
@@ -268,7 +270,9 @@ def run_hook(
files_before = set(output_dir.rglob('*')) if output_dir.exists() else set()
# Detect if this is a background hook (long-running daemon)
is_background = '__background' in script.stem
# New convention: .bg. suffix (e.g., on_Snapshot__21_consolelog.bg.js)
# Old convention: __background in stem (for backwards compatibility)
is_background = '.bg.' in script.name or '__background' in script.stem
# Set up output files for ALL hooks (useful for debugging)
stdout_file = output_dir / 'stdout.log'
@@ -322,13 +326,44 @@ def run_hook(
# Exclude the log files themselves from new_files
new_files = [f for f in new_files if f not in ('stdout.log', 'stderr.log', 'hook.pid')]
# Parse RESULT_JSON from stdout
# Parse JSONL output from stdout
# Supports both new JSONL format (any line starting with { that has 'type')
# and legacy RESULT_JSON= format for backwards compatibility
output_json = None
records = []
plugin_name = script.parent.name # Plugin directory name (e.g., 'wget')
for line in stdout.splitlines():
if line.startswith('RESULT_JSON='):
line = line.strip()
if not line:
continue
# New JSONL format: any line starting with { that has 'type' field
if line.startswith('{'):
try:
output_json = json.loads(line[len('RESULT_JSON='):])
break
data = json.loads(line)
if 'type' in data:
# Add plugin metadata to every record
data['plugin'] = plugin_name
data['plugin_hook'] = str(script)
records.append(data)
# For backwards compatibility, also set output_json for first ArchiveResult
if data.get('type') == 'ArchiveResult' and output_json is None:
output_json = data
except json.JSONDecodeError:
pass
# Legacy format: RESULT_JSON=...
elif line.startswith('RESULT_JSON='):
try:
data = json.loads(line[len('RESULT_JSON='):])
if output_json is None:
output_json = data
# Convert legacy format to new format
data['type'] = 'ArchiveResult'
data['plugin'] = plugin_name
data['plugin_hook'] = str(script)
records.append(data)
except json.JSONDecodeError:
pass
@@ -348,6 +383,7 @@ def run_hook(
output_files=new_files,
duration_ms=duration_ms,
hook=str(script),
records=records,
)
except Exception as e:
@@ -360,6 +396,7 @@ def run_hook(
output_files=[],
duration_ms=duration_ms,
hook=str(script),
records=[],
)
@@ -1104,3 +1141,112 @@ def discover_plugin_templates() -> Dict[str, Dict[str, str]]:
return templates
# =============================================================================
# Hook Result Processing Helpers
# =============================================================================
def find_binary_for_cmd(cmd: List[str], machine_id: str) -> Optional[str]:
"""
Find InstalledBinary for a command, trying abspath first then name.
Only matches binaries on the current machine.
Args:
cmd: Command list (e.g., ['/usr/bin/wget', '-p', 'url'])
machine_id: Current machine ID
Returns:
Binary ID as string if found, None otherwise
"""
if not cmd:
return None
from machine.models import InstalledBinary
bin_path_or_name = cmd[0] if isinstance(cmd, list) else cmd
# Try matching by absolute path first
binary = InstalledBinary.objects.filter(
abspath=bin_path_or_name,
machine_id=machine_id
).first()
if binary:
return str(binary.id)
# Fallback: match by binary name
bin_name = Path(bin_path_or_name).name
binary = InstalledBinary.objects.filter(
name=bin_name,
machine_id=machine_id
).first()
return str(binary.id) if binary else None
def create_model_record(record: Dict[str, Any]) -> Any:
"""
Generic helper to create/update model instances from hook JSONL output.
Args:
record: Dict with 'type' field and model data
Returns:
Created/updated model instance, or None if type unknown
"""
from machine.models import InstalledBinary, Machine
record_type = record.pop('type', None)
if not record_type:
return None
# Remove plugin metadata (not model fields)
record.pop('plugin', None)
record.pop('plugin_hook', None)
if record_type == 'InstalledBinary':
# InstalledBinary requires machine FK
machine = Machine.current()
record.setdefault('machine', machine)
# Required fields check
name = record.get('name')
abspath = record.get('abspath')
if not name or not abspath:
return None
obj, created = InstalledBinary.objects.update_or_create(
machine=machine,
name=name,
defaults={
'abspath': abspath,
'version': record.get('version', ''),
'sha256': record.get('sha256', ''),
'binprovider': record.get('binprovider', 'env'),
}
)
return obj
elif record_type == 'Machine':
# Machine config update (special _method handling)
method = record.pop('_method', None)
if method == 'update':
key = record.get('key')
value = record.get('value')
if key and value:
machine = Machine.current()
if not machine.config:
machine.config = {}
machine.config[key] = value
machine.save(update_fields=['config'])
return machine
return None
# Add more types as needed (Dependency, Snapshot, etc.)
else:
# Unknown type - log warning but don't fail
import sys
print(f"Warning: Unknown record type '{record_type}' from hook output", file=sys.stderr)
return None

18
archivebox/misc/jsonl.py
View File

@@ -174,16 +174,30 @@ def archiveresult_to_jsonl(result) -> Dict[str, Any]:
"""
Convert an ArchiveResult model instance to a JSONL record.
"""
return {
record = {
'type': TYPE_ARCHIVERESULT,
'id': str(result.id),
'snapshot_id': str(result.snapshot_id),
'extractor': result.extractor,
'status': result.status,
'output': result.output,
'output_str': result.output_str,
'start_ts': result.start_ts.isoformat() if result.start_ts else None,
'end_ts': result.end_ts.isoformat() if result.end_ts else None,
}
# Include optional fields if set
if result.output_json:
record['output_json'] = result.output_json
if result.output_files:
record['output_files'] = result.output_files
if result.output_size:
record['output_size'] = result.output_size
if result.output_mimetypes:
record['output_mimetypes'] = result.output_mimetypes
if result.cmd:
record['cmd'] = result.cmd
if result.cmd_version:
record['cmd_version'] = result.cmd_version
return record
def tag_to_jsonl(tag) -> Dict[str, Any]:

43
archivebox/plugins/accessibility/on_Snapshot__39_accessibility.js
View File

@@ -198,12 +198,12 @@ async function main() {
// Check if enabled
if (!getEnvBool('SAVE_ACCESSIBILITY', true)) {
console.log('Skipping accessibility (SAVE_ACCESSIBILITY=False)');
status = 'skipped';
const endTs = new Date();
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`STATUS=${status}`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status, url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'SAVE_ACCESSIBILITY=False',
}));
process.exit(0);
}
@@ -225,34 +225,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

28
archivebox/plugins/archive_org/on_Snapshot__13_archive_org.py
View File

@@ -121,33 +121,19 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

23
archivebox/plugins/chrome_navigate/on_Snapshot__30_chrome_navigate.js
View File

@@ -157,26 +157,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) console.log(`OUTPUT=${output}`);
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR=${error}`);
if (error) console.error(`ERROR: ${error}`);
console.log(`RESULT_JSON=${JSON.stringify({
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
})}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

34
archivebox/plugins/chrome_session/on_Crawl__00_validate_chrome.py → archivebox/plugins/chrome_session/on_Crawl__00_install_chrome.py
View File

@@ -1,23 +1,34 @@
#!/usr/bin/env python3
"""
Validation hook for Chrome/Chromium binary.
Install hook for Chrome/Chromium binary.
Runs at crawl start to verify Chrome is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects CHROME_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_chrome() -> dict | None:
"""Find Chrome/Chromium binary."""
"""Find Chrome/Chromium binary, respecting CHROME_BINARY env var."""
try:
from abx_pkg import Binary, AptProvider, BrewProvider, EnvProvider
# Try common Chrome/Chromium binary names
for name in ['google-chrome', 'chromium', 'chromium-browser', 'google-chrome-stable', 'chrome']:
binary = Binary(name=name, binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('CHROME_BINARY', '').strip()
if configured_binary:
# User specified a custom binary path or name
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
binary = Binary(name=bin_name, binproviders=[EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
@@ -27,6 +38,19 @@ def find_chrome() -> dict | None:
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
'binprovider': loaded.binprovider.name if loaded.binprovider else 'env',
}
else:
# Try common Chrome/Chromium binary names
for name in ['google-chrome', 'chromium', 'chromium-browser', 'google-chrome-stable', 'chrome']:
binary = Binary(name=name, binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'chrome',
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
'binprovider': loaded.binprovider.name if loaded.binprovider else 'env',
}
except Exception:
pass

0
archivebox/plugins/chrome_session/on_Crawl__00_validate_chrome_config.py → archivebox/plugins/chrome_session/on_Crawl__00_install_chrome_config.py
View File

36
archivebox/plugins/chrome_session/on_Snapshot__20_chrome_session.js
View File

@@ -380,39 +380,21 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (version) {
console.log(`VERSION=${version}`);
}
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) {
console.error(`ERROR=${error}`);
console.error(`ERROR: ${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
crawl_id: crawlId || null,
// Output clean JSONL (no RESULT_JSON= prefix)
const result = {
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
cmd_version: version,
output,
error: error || null,
output_str: output || error || '',
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
if (version) {
result.cmd_version = version;
}
console.log(JSON.stringify(result));
process.exit(status === 'succeeded' ? 0 : 1);
}

49
archivebox/plugins/consolelog/on_Snapshot__21_consolelog.js → archivebox/plugins/consolelog/on_Snapshot__21_consolelog.bg.js
View File

@@ -186,14 +186,8 @@ async function main() {
}
if (!getEnvBool('SAVE_CONSOLELOG', true)) {
console.log('Skipping (SAVE_CONSOLELOG=False)');
const result = {
extractor: EXTRACTOR_NAME,
status: 'skipped',
url,
snapshot_id: snapshotId,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
console.error('Skipping (SAVE_CONSOLELOG=False)');
console.log(JSON.stringify({type: 'ArchiveResult', status: 'skipped', output_str: 'SAVE_CONSOLELOG=False'}));
process.exit(0);
}
@@ -211,43 +205,26 @@ async function main() {
// Report success
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
console.log(`OUTPUT=${OUTPUT_FILE}`);
console.log(`STATUS=succeeded`);
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'succeeded',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output: OUTPUT_FILE,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: OUTPUT_FILE,
}));
process.exit(0);
} catch (e) {
const error = `${e.name}: ${e.message}`;
console.error(`ERROR=${error}`);
console.error(`ERROR: ${error}`);
const endTs = new Date();
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'failed',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
error,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: error,
}));
process.exit(1);
}
}

51
archivebox/plugins/dom/on_Snapshot__36_dom.js
View File

@@ -222,19 +222,23 @@ async function main() {
// Check if DOM is enabled (permanent skip - don't retry)
if (!getEnvBool('SAVE_DOM', true)) {
console.log('Skipping DOM (SAVE_DOM=False)');
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${new Date().toISOString()}`);
console.log(`STATUS=skipped`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status: 'skipped', url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'SAVE_DOM=False',
}));
process.exit(0); // Permanent skip - feature disabled
}
// Check if staticfile extractor already handled this (permanent skip)
if (hasStaticFileOutput()) {
console.log(`Skipping DOM - staticfile extractor already downloaded this`);
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${new Date().toISOString()}`);
console.log(`STATUS=skipped`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status: 'skipped', url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'staticfile already handled',
}));
process.exit(0); // Permanent skip - staticfile already handled
} else {
const result = await dumpDom(url);
@@ -255,34 +259,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

29
archivebox/plugins/extractor_utils.py
View File

@@ -105,7 +105,7 @@ class ExtractorResult:
# ... do extraction ...
result.output = 'example.com/index.html'
result.output_str = 'example.com/index.html'
result.status = 'succeeded'
result.finish()
@@ -121,7 +121,7 @@ class ExtractorResult:
self.cmd: list[str] = []
self.version: str = ''
self.output: str | Path | None = None
self.output_str: str = '' # Human-readable output summary
self.status: str = 'failed' # 'succeeded', 'failed', 'skipped'
self.stdout: str = ''
@@ -174,8 +174,8 @@ class ExtractorResult:
print(f"VERSION={self.version}")
# Print output path
if self.output:
print(f"OUTPUT={self.output}")
if self.output_str:
print(f"OUTPUT={self.output_str}")
# Print status
print(f"STATUS={self.status}")
@@ -192,22 +192,17 @@ class ExtractorResult:
for hint in self.hints:
print(f"HINT={hint}", file=sys.stderr)
# Print JSON result for structured parsing
# Print clean JSONL result for hooks.py to parse
result_json = {
'extractor': self.name,
'url': self.url,
'snapshot_id': self.snapshot_id,
'type': 'ArchiveResult',
'status': self.status,
'start_ts': self.start_ts.isoformat(),
'end_ts': self.end_ts.isoformat() if self.end_ts else None,
'duration': round(self.duration, 2),
'cmd': self.cmd,
'cmd_version': self.version,
'output': str(self.output) if self.output else None,
'returncode': self.returncode,
'error': self.error or None,
'output_str': self.output_str or self.error or '',
}
print(f"RESULT_JSON={json.dumps(result_json)}")
if self.cmd:
result_json['cmd'] = self.cmd
if self.version:
result_json['cmd_version'] = self.version
print(json.dumps(result_json))
def run_shell_command(

28
archivebox/plugins/favicon/on_Snapshot__11_favicon.py
View File

@@ -134,33 +134,19 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

35
archivebox/plugins/forumdl/on_Crawl__00_validate_forumdl.py → archivebox/plugins/forumdl/on_Crawl__00_install_forumdl.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for forum-dl.
Install hook for forum-dl.
Runs at crawl start to verify forum-dl binary is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects FORUMDL_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_forumdl() -> dict | None:
"""Find forum-dl binary."""
"""Find forum-dl binary, respecting FORUMDL_BINARY env var."""
try:
from abx_pkg import Binary, PipProvider, EnvProvider
binary = Binary(name='forum-dl', binproviders=[PipProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('FORUMDL_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'forum-dl'
binary = Binary(name=bin_name, binproviders=[PipProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'forum-dl',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_forumdl() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('FORUMDL_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'forum-dl'
# Check for forum-dl (required)
forumdl_result = find_forumdl()
@@ -67,7 +90,7 @@ def main():
# Provide overrides to install with chardet instead
print(json.dumps({
'type': 'Dependency',
'bin_name': 'forum-dl',
'bin_name': bin_name,
'bin_providers': 'pip,env',
'overrides': {
'pip': {
@@ -77,7 +100,7 @@ def main():
}
}
}))
missing_deps.append('forum-dl')
missing_deps.append(bin_name)
if missing_deps:
print(f"Missing dependencies: {', '.join(missing_deps)}", file=sys.stderr)

35
archivebox/plugins/gallerydl/on_Crawl__00_validate_gallerydl.py → archivebox/plugins/gallerydl/on_Crawl__00_install_gallerydl.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for gallery-dl.
Install hook for gallery-dl.
Runs at crawl start to verify gallery-dl binary is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects GALLERYDL_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_gallerydl() -> dict | None:
"""Find gallery-dl binary."""
"""Find gallery-dl binary, respecting GALLERYDL_BINARY env var."""
try:
from abx_pkg import Binary, PipProvider, EnvProvider
binary = Binary(name='gallery-dl', binproviders=[PipProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('GALLERYDL_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'gallery-dl'
binary = Binary(name=bin_name, binproviders=[PipProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'gallery-dl',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_gallerydl() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('GALLERYDL_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'gallery-dl'
# Check for gallery-dl (required)
gallerydl_result = find_gallerydl()
@@ -65,10 +88,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'gallery-dl',
'bin_name': bin_name,
'bin_providers': 'pip,env',
}))
missing_deps.append('gallery-dl')
missing_deps.append(bin_name)
if missing_deps:
print(f"Missing dependencies: {', '.join(missing_deps)}", file=sys.stderr)

35
archivebox/plugins/git/on_Crawl__00_validate_git.py → archivebox/plugins/git/on_Crawl__00_install_git.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for git binary.
Install hook for git binary.
Runs at crawl start to verify git is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects GIT_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_git() -> dict | None:
"""Find git binary."""
"""Find git binary, respecting GIT_BINARY env var."""
try:
from abx_pkg import Binary, EnvProvider
binary = Binary(name='git', binproviders=[EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('GIT_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'git'
binary = Binary(name=bin_name, binproviders=[EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'git',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_git() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('GIT_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'git'
result = find_git()
if result and result.get('abspath'):
@@ -63,10 +86,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'git',
'bin_name': bin_name,
'bin_providers': 'apt,brew,env',
}))
print(f"git binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

37
archivebox/plugins/git/on_Snapshot__12_git.py
View File

@@ -153,38 +153,23 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if binary:
print(f'CMD={binary} clone {url}')
if version:
print(f'VERSION={version}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'cmd_version': version,
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
if binary:
result['cmd'] = [binary, 'clone', '--depth=1', '--recursive', url, OUTPUT_DIR]
if version:
result['cmd_version'] = version
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

31
archivebox/plugins/headers/on_Snapshot__33_headers.js
View File

@@ -162,34 +162,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

53
archivebox/plugins/media/on_Crawl__00_validate_ytdlp.py → archivebox/plugins/media/on_Crawl__00_install_ytdlp.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for yt-dlp and its dependencies (node, ffmpeg).
Install hook for yt-dlp and its dependencies (node, ffmpeg).
Runs at crawl start to verify yt-dlp and required binaries are available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects YTDLP_BINARY, NODE_BINARY, FFMPEG_BINARY env vars.
"""
import os
import sys
import json
from pathlib import Path
def get_bin_name(env_var: str, default: str) -> str:
"""Get binary name from env var or use default."""
configured = os.environ.get(env_var, '').strip()
if configured:
if '/' in configured:
return Path(configured).name
return configured
return default
def find_ytdlp() -> dict | None:
"""Find yt-dlp binary."""
"""Find yt-dlp binary, respecting YTDLP_BINARY env var."""
try:
from abx_pkg import Binary, PipProvider, BrewProvider, AptProvider, EnvProvider
binary = Binary(name='yt-dlp', binproviders=[PipProvider(), BrewProvider(), AptProvider(), EnvProvider()])
bin_name = get_bin_name('YTDLP_BINARY', 'yt-dlp')
binary = Binary(name=bin_name, binproviders=[PipProvider(), BrewProvider(), AptProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'yt-dlp',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,15 +46,16 @@ def find_ytdlp() -> dict | None:
def find_node() -> dict | None:
"""Find node binary."""
"""Find node binary, respecting NODE_BINARY env var."""
try:
from abx_pkg import Binary, AptProvider, BrewProvider, EnvProvider
binary = Binary(name='node', binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
bin_name = get_bin_name('NODE_BINARY', 'node')
binary = Binary(name=bin_name, binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'node',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -53,15 +68,16 @@ def find_node() -> dict | None:
def find_ffmpeg() -> dict | None:
"""Find ffmpeg binary."""
"""Find ffmpeg binary, respecting FFMPEG_BINARY env var."""
try:
from abx_pkg import Binary, AptProvider, BrewProvider, EnvProvider
binary = Binary(name='ffmpeg', binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
bin_name = get_bin_name('FFMPEG_BINARY', 'ffmpeg')
binary = Binary(name=bin_name, binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'ffmpeg',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -85,6 +101,11 @@ def main():
missing_deps = []
# Get configured binary names
ytdlp_bin_name = get_bin_name('YTDLP_BINARY', 'yt-dlp')
node_bin_name = get_bin_name('NODE_BINARY', 'node')
ffmpeg_bin_name = get_bin_name('FFMPEG_BINARY', 'ffmpeg')
# Emit results for yt-dlp
if ytdlp_result and ytdlp_result.get('abspath'):
print(json.dumps({
@@ -113,10 +134,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'yt-dlp',
'bin_name': ytdlp_bin_name,
'bin_providers': 'pip,brew,apt,env',
}))
missing_deps.append('yt-dlp')
missing_deps.append(ytdlp_bin_name)
# Emit results for node
if node_result and node_result.get('abspath'):
@@ -147,13 +168,13 @@ def main():
# node is installed as 'nodejs' package on apt
print(json.dumps({
'type': 'Dependency',
'bin_name': 'node',
'bin_name': node_bin_name,
'bin_providers': 'apt,brew,env',
'overrides': {
'apt': {'packages': ['nodejs']}
}
}))
missing_deps.append('node')
missing_deps.append(node_bin_name)
# Emit results for ffmpeg
if ffmpeg_result and ffmpeg_result.get('abspath'):
@@ -183,10 +204,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'ffmpeg',
'bin_name': ffmpeg_bin_name,
'bin_providers': 'apt,brew,env',
}))
missing_deps.append('ffmpeg')
missing_deps.append(ffmpeg_bin_name)
if missing_deps:
print(f"Missing dependencies: {', '.join(missing_deps)}", file=sys.stderr)

53
archivebox/plugins/media/on_Snapshot__51_media.py
View File

@@ -218,22 +218,14 @@ def main(url: str, snapshot_id: str):
try:
# Check if yt-dlp is enabled
if not (get_env_bool('USE_YTDLP', True) and get_env_bool('SAVE_MEDIA', True)):
print('Skipping media (USE_YTDLP=False or SAVE_MEDIA=False)')
status = 'skipped'
end_ts = datetime.now(timezone.utc)
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'STATUS={status}')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": status, "url": url, "snapshot_id": snapshot_id})}')
print('Skipping media (USE_YTDLP=False or SAVE_MEDIA=False)', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'USE_YTDLP=False'}))
sys.exit(0)
# Check if staticfile extractor already handled this (permanent skip)
if has_staticfile_output():
print(f'Skipping media - staticfile extractor already downloaded this')
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={datetime.now(timezone.utc).isoformat()}')
print(f'STATUS={status}')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": status, "url": url, "snapshot_id": snapshot_id})}')
print('Skipping media - staticfile extractor already downloaded this', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'staticfile already exists'}))
sys.exit(0)
# Find binary
@@ -265,38 +257,23 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if cmd_str:
print(f'CMD={cmd_str}')
if version:
print(f'VERSION={version}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'cmd_version': version,
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
if binary:
result['cmd'] = [binary, url]
if version:
result['cmd_version'] = version
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

35
archivebox/plugins/mercury/on_Crawl__00_validate_mercury.py → archivebox/plugins/mercury/on_Crawl__00_install_mercury.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for postlight-parser binary.
Install hook for postlight-parser binary.
Runs at crawl start to verify postlight-parser is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects MERCURY_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_mercury() -> dict | None:
"""Find postlight-parser binary."""
"""Find postlight-parser binary, respecting MERCURY_BINARY env var."""
try:
from abx_pkg import Binary, NpmProvider, EnvProvider
binary = Binary(name='postlight-parser', binproviders=[NpmProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('MERCURY_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'postlight-parser'
binary = Binary(name=bin_name, binproviders=[NpmProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'postlight-parser',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_mercury() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('MERCURY_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'postlight-parser'
result = find_mercury()
if result and result.get('abspath'):
@@ -64,13 +87,13 @@ def main():
# postlight-parser is installed as @postlight/parser in npm
print(json.dumps({
'type': 'Dependency',
'bin_name': 'postlight-parser',
'bin_name': bin_name,
'bin_providers': 'npm,env',
'overrides': {
'npm': {'packages': ['@postlight/parser']}
}
}))
print(f"postlight-parser binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

35
archivebox/plugins/papersdl/on_Crawl__00_validate_papersdl.py → archivebox/plugins/papersdl/on_Crawl__00_install_papersdl.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for papers-dl.
Install hook for papers-dl.
Runs at crawl start to verify papers-dl binary is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects PAPERSDL_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_papersdl() -> dict | None:
"""Find papers-dl binary."""
"""Find papers-dl binary, respecting PAPERSDL_BINARY env var."""
try:
from abx_pkg import Binary, PipProvider, EnvProvider
binary = Binary(name='papers-dl', binproviders=[PipProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('PAPERSDL_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'papers-dl'
binary = Binary(name=bin_name, binproviders=[PipProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'papers-dl',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_papersdl() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('PAPERSDL_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'papers-dl'
# Check for papers-dl (required)
papersdl_result = find_papersdl()
@@ -65,10 +88,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'papers-dl',
'bin_name': bin_name,
'bin_providers': 'pip,env',
}))
missing_deps.append('papers-dl')
missing_deps.append(bin_name)
if missing_deps:
print(f"Missing dependencies: {', '.join(missing_deps)}", file=sys.stderr)

43
archivebox/plugins/parse_dom_outlinks/on_Snapshot__40_parse_dom_outlinks.js
View File

@@ -211,12 +211,12 @@ async function main() {
// Check if enabled
if (!getEnvBool('SAVE_DOM_OUTLINKS', true)) {
console.log('Skipping DOM outlinks (SAVE_DOM_OUTLINKS=False)');
status = 'skipped';
const endTs = new Date();
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`STATUS=${status}`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status, url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'SAVE_DOM_OUTLINKS=False',
}));
process.exit(0);
}
@@ -240,34 +240,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

41
archivebox/plugins/pdf/on_Snapshot__35_pdf.js
View File

@@ -230,10 +230,12 @@ async function main() {
// Check if staticfile extractor already handled this (permanent skip)
if (hasStaticFileOutput()) {
console.log(`Skipping PDF - staticfile extractor already downloaded this`);
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${new Date().toISOString()}`);
console.log(`STATUS=skipped`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status: 'skipped', url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'staticfile already handled',
}));
process.exit(0); // Permanent skip - staticfile already handled
} else {
const result = await printToPdf(url);
@@ -254,34 +256,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

35
archivebox/plugins/readability/on_Crawl__00_validate_readability.py → archivebox/plugins/readability/on_Crawl__00_install_readability.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for readability-extractor binary.
Install hook for readability-extractor binary.
Runs at crawl start to verify readability-extractor is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects READABILITY_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_readability() -> dict | None:
"""Find readability-extractor binary."""
"""Find readability-extractor binary, respecting READABILITY_BINARY env var."""
try:
from abx_pkg import Binary, NpmProvider, EnvProvider
binary = Binary(name='readability-extractor', binproviders=[NpmProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('READABILITY_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'readability-extractor'
binary = Binary(name=bin_name, binproviders=[NpmProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'readability-extractor',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_readability() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('READABILITY_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'readability-extractor'
result = find_readability()
if result and result.get('abspath'):
@@ -64,13 +87,13 @@ def main():
# readability-extractor is installed from GitHub
print(json.dumps({
'type': 'Dependency',
'bin_name': 'readability-extractor',
'bin_name': bin_name,
'bin_providers': 'npm,env',
'overrides': {
'npm': {'packages': ['github:ArchiveBox/readability-extractor']}
}
}))
print(f"readability-extractor binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

37
archivebox/plugins/readability/on_Snapshot__52_readability.py
View File

@@ -178,38 +178,23 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if binary:
print(f'CMD={binary} <html>')
if version:
print(f'VERSION={version}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'cmd_version': version,
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
if binary:
result['cmd'] = [binary, '<html>']
if version:
result['cmd_version'] = version
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

23
archivebox/plugins/redirects/on_Snapshot__31_redirects.js
View File

@@ -218,26 +218,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) console.log(`OUTPUT=${output}`);
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR=${error}`);
if (error) console.error(`ERROR: ${error}`);
console.log(`RESULT_JSON=${JSON.stringify({
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
})}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

49
archivebox/plugins/responses/on_Snapshot__24_responses.js → archivebox/plugins/responses/on_Snapshot__24_responses.bg.js
View File

@@ -288,14 +288,8 @@ async function main() {
}
if (!getEnvBool('SAVE_RESPONSES', true)) {
console.log('Skipping (SAVE_RESPONSES=False)');
const result = {
extractor: EXTRACTOR_NAME,
status: 'skipped',
url,
snapshot_id: snapshotId,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
console.error('Skipping (SAVE_RESPONSES=False)');
console.log(JSON.stringify({type: 'ArchiveResult', status: 'skipped', output_str: 'SAVE_RESPONSES=False'}));
process.exit(0);
}
@@ -313,43 +307,26 @@ async function main() {
// Report success
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
console.log(`OUTPUT=responses/`);
console.log(`STATUS=succeeded`);
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'succeeded',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output: 'responses/',
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: 'responses/',
}));
process.exit(0);
} catch (e) {
const error = `${e.name}: ${e.message}`;
console.error(`ERROR=${error}`);
console.error(`ERROR: ${error}`);
const endTs = new Date();
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'failed',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
error,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: error,
}));
process.exit(1);
}
}

41
archivebox/plugins/screenshot/on_Snapshot__34_screenshot.js
View File

@@ -226,10 +226,12 @@ async function main() {
// Check if staticfile extractor already handled this (permanent skip)
if (hasStaticFileOutput()) {
console.log(`Skipping screenshot - staticfile extractor already downloaded this`);
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${new Date().toISOString()}`);
console.log(`STATUS=skipped`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status: 'skipped', url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'staticfile already handled',
}));
process.exit(0); // Permanent skip - staticfile already handled
} else {
const result = await takeScreenshot(url);
@@ -250,34 +252,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

36
archivebox/plugins/search_backend_ripgrep/on_Crawl__00_validate_ripgrep.py → archivebox/plugins/search_backend_ripgrep/on_Crawl__00_install_ripgrep.py
View File

@@ -1,26 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for ripgrep binary.
Install hook for ripgrep binary.
Only runs if SEARCH_BACKEND_ENGINE is set to 'ripgrep'.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects RIPGREP_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_ripgrep() -> dict | None:
"""Find ripgrep binary."""
"""Find ripgrep binary, respecting RIPGREP_BINARY env var."""
try:
from abx_pkg import Binary, AptProvider, BrewProvider, EnvProvider
binary = Binary(name='rg', binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('RIPGREP_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'rg'
binary = Binary(name=bin_name, binproviders=[AptProvider(), BrewProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'rg',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -33,7 +46,7 @@ def find_ripgrep() -> dict | None:
def main():
"""Validate ripgrep binary and output JSONL."""
"""Find ripgrep binary and output JSONL."""
# Check if ripgrep search backend is enabled
search_backend = os.environ.get('SEARCH_BACKEND_ENGINE', '').lower()
@@ -42,6 +55,15 @@ def main():
# No-op: ripgrep is not the active search backend
sys.exit(0)
# Determine binary name from config
configured_binary = os.environ.get('RIPGREP_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'rg'
result = find_ripgrep()
if result and result.get('abspath'):
@@ -76,12 +98,12 @@ def main():
# Output Dependency request
print(json.dumps({
'type': 'Dependency',
'bin_name': 'rg',
'bin_name': bin_name,
'bin_providers': 'apt,brew,cargo,env',
}))
# Exit non-zero to indicate binary not found
print(f"ripgrep binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

43
archivebox/plugins/seo/on_Snapshot__38_seo.js
View File

@@ -152,12 +152,12 @@ async function main() {
// Check if enabled
if (!getEnvBool('SAVE_SEO', true)) {
console.log('Skipping SEO (SAVE_SEO=False)');
status = 'skipped';
const endTs = new Date();
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`STATUS=${status}`);
console.log(`RESULT_JSON=${JSON.stringify({extractor: EXTRACTOR_NAME, status, url, snapshot_id: snapshotId})}`);
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'skipped',
output_str: 'SAVE_SEO=False',
}));
process.exit(0);
}
@@ -178,34 +178,15 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) console.error(`ERROR: ${error}`);
if (error) {
console.error(`ERROR=${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
output_str: output || error || '',
}));
process.exit(status === 'succeeded' ? 0 : 1);
}

35
archivebox/plugins/singlefile/on_Crawl__00_validate_singlefile.py → archivebox/plugins/singlefile/on_Crawl__00_install_singlefile.py
View File

@@ -1,25 +1,39 @@
#!/usr/bin/env python3
"""
Validation hook for single-file binary.
Install hook for single-file binary.
Runs at crawl start to verify single-file (npm package) is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects SINGLEFILE_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_singlefile() -> dict | None:
"""Find single-file binary."""
"""Find single-file binary, respecting SINGLEFILE_BINARY env var."""
try:
from abx_pkg import Binary, NpmProvider, EnvProvider
binary = Binary(name='single-file', binproviders=[NpmProvider(), EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('SINGLEFILE_BINARY', '').strip()
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'single-file'
binary = Binary(name=bin_name, binproviders=[NpmProvider(), EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'single-file',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,6 +46,15 @@ def find_singlefile() -> dict | None:
def main():
# Determine binary name from config
configured_binary = os.environ.get('SINGLEFILE_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'single-file'
result = find_singlefile()
if result and result.get('abspath'):
@@ -63,10 +86,10 @@ def main():
else:
print(json.dumps({
'type': 'Dependency',
'bin_name': 'single-file',
'bin_name': bin_name,
'bin_providers': 'npm,env',
}))
print(f"single-file binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

55
archivebox/plugins/singlefile/on_Snapshot__37_singlefile.py
View File

@@ -245,23 +245,15 @@ def main(url: str, snapshot_id: str):
try:
# Check if SingleFile is enabled
if not get_env_bool('SAVE_SINGLEFILE', True):
print('Skipping SingleFile (SAVE_SINGLEFILE=False)')
status = 'skipped'
end_ts = datetime.now(timezone.utc)
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'STATUS={status}')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": status, "url": url, "snapshot_id": snapshot_id})}')
print('Skipping SingleFile (SAVE_SINGLEFILE=False)', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'SAVE_SINGLEFILE=False'}))
sys.exit(0)
# Check if staticfile extractor already handled this (permanent skip)
if has_staticfile_output():
print(f'Skipping SingleFile - staticfile extractor already downloaded this')
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={datetime.now(timezone.utc).isoformat()}')
print(f'STATUS=skipped')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": "skipped", "url": url, "snapshot_id": snapshot_id})}')
sys.exit(0) # Permanent skip - staticfile already handled
print('Skipping SingleFile - staticfile extractor already downloaded this', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'staticfile already exists'}))
sys.exit(0)
# Find binary
binary = find_singlefile()
@@ -287,38 +279,23 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if cmd_str:
print(f'CMD={cmd_str}')
if version:
print(f'VERSION={version}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'cmd_version': version,
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
if binary:
result['cmd'] = [binary, '--browser-headless', url, OUTPUT_FILE]
if version:
result['cmd_version'] = version
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

49
archivebox/plugins/ssl/on_Snapshot__23_ssl.js → archivebox/plugins/ssl/on_Snapshot__23_ssl.bg.js
View File

@@ -176,14 +176,8 @@ async function main() {
}
if (!getEnvBool('SAVE_SSL', true)) {
console.log('Skipping (SAVE_SSL=False)');
const result = {
extractor: EXTRACTOR_NAME,
status: 'skipped',
url,
snapshot_id: snapshotId,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
console.error('Skipping (SAVE_SSL=False)');
console.log(JSON.stringify({type: 'ArchiveResult', status: 'skipped', output_str: 'SAVE_SSL=False'}));
process.exit(0);
}
@@ -201,43 +195,26 @@ async function main() {
// Report success
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
console.log(`OUTPUT=${OUTPUT_FILE}`);
console.log(`STATUS=succeeded`);
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'succeeded',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output: OUTPUT_FILE,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: OUTPUT_FILE,
}));
process.exit(0);
} catch (e) {
const error = `${e.name}: ${e.message}`;
console.error(`ERROR=${error}`);
console.error(`ERROR: ${error}`);
const endTs = new Date();
const result = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
console.log(JSON.stringify({
type: 'ArchiveResult',
status: 'failed',
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
error,
};
console.log(`RESULT_JSON=${JSON.stringify(result)}`);
output_str: error,
}));
process.exit(1);
}
}

28
archivebox/plugins/title/on_Snapshot__32_title.js
View File

@@ -221,34 +221,18 @@ async function main() {
}
const endTs = new Date();
const duration = (endTs - startTs) / 1000;
// Print results
console.log(`START_TS=${startTs.toISOString()}`);
console.log(`END_TS=${endTs.toISOString()}`);
console.log(`DURATION=${duration.toFixed(2)}`);
if (output) {
console.log(`OUTPUT=${output}`);
}
console.log(`STATUS=${status}`);
if (error) {
console.error(`ERROR=${error}`);
console.error(`ERROR: ${error}`);
}
// Print JSON result
const resultJson = {
extractor: EXTRACTOR_NAME,
url,
snapshot_id: snapshotId,
// Output clean JSONL (no RESULT_JSON= prefix)
const result = {
type: 'ArchiveResult',
status,
start_ts: startTs.toISOString(),
end_ts: endTs.toISOString(),
duration: Math.round(duration * 100) / 100,
output,
error: error || null,
output_str: output || error || '',
};
console.log(`RESULT_JSON=${JSON.stringify(resultJson)}`);
console.log(JSON.stringify(result));
process.exit(status === 'succeeded' ? 0 : 1);
}

42
archivebox/plugins/wget/on_Crawl__00_validate_wget.py → archivebox/plugins/wget/on_Crawl__00_install_wget.py
View File

@@ -1,25 +1,43 @@
#!/usr/bin/env python3
"""
Validation hook for wget binary.
Install hook for wget binary.
Runs at crawl start to verify wget is available.
Outputs JSONL for InstalledBinary and Machine config updates.
Respects WGET_BINARY env var for custom binary paths.
"""
import os
import sys
import json
from pathlib import Path
def find_wget() -> dict | None:
"""Find wget binary using abx-pkg."""
"""Find wget binary using abx-pkg, respecting WGET_BINARY env var."""
try:
from abx_pkg import Binary, EnvProvider
binary = Binary(name='wget', binproviders=[EnvProvider()])
# Check if user has configured a custom binary
configured_binary = os.environ.get('WGET_BINARY', '').strip()
if configured_binary:
# User specified a custom binary path or name
if '/' in configured_binary:
# Absolute path - extract name from path
bin_name = Path(configured_binary).name
else:
# Just a binary name
bin_name = configured_binary
else:
# Default to 'wget'
bin_name = 'wget'
binary = Binary(name=bin_name, binproviders=[EnvProvider()])
loaded = binary.load()
if loaded and loaded.abspath:
return {
'name': 'wget',
'name': bin_name,
'abspath': str(loaded.abspath),
'version': str(loaded.version) if loaded.version else None,
'sha256': loaded.sha256 if hasattr(loaded, 'sha256') else None,
@@ -32,7 +50,15 @@ def find_wget() -> dict | None:
def main():
"""Validate wget binary and output JSONL."""
"""Find wget binary and output JSONL."""
# Determine binary name from config
configured_binary = os.environ.get('WGET_BINARY', '').strip()
if configured_binary and '/' in configured_binary:
bin_name = Path(configured_binary).name
elif configured_binary:
bin_name = configured_binary
else:
bin_name = 'wget'
result = find_wget()
@@ -65,15 +91,15 @@ def main():
sys.exit(0)
else:
# Output Dependency request
# Output Dependency request (uses configured bin_name)
print(json.dumps({
'type': 'Dependency',
'bin_name': 'wget',
'bin_name': bin_name,
'bin_providers': 'apt,brew,env',
}))
# Exit non-zero to indicate binary not found
print(f"wget binary not found", file=sys.stderr)
print(f"{bin_name} binary not found", file=sys.stderr)
sys.exit(1)

0
archivebox/plugins/wget/on_Crawl__00_validate_wget_config.py → archivebox/plugins/wget/on_Crawl__00_install_wget_config.py
View File

55
archivebox/plugins/wget/on_Snapshot__50_wget.py
View File

@@ -241,23 +241,15 @@ def main(url: str, snapshot_id: str):
try:
# Check if wget is enabled
if not get_env_bool('SAVE_WGET', True):
print('Skipping wget (SAVE_WGET=False)')
status = 'skipped'
end_ts = datetime.now(timezone.utc)
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'STATUS={status}')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": status, "url": url, "snapshot_id": snapshot_id})}')
print('Skipping wget (SAVE_WGET=False)', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'SAVE_WGET=False'}))
sys.exit(0)
# Check if staticfile extractor already handled this (permanent skip)
if has_staticfile_output():
print(f'Skipping wget - staticfile extractor already downloaded this')
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={datetime.now(timezone.utc).isoformat()}')
print(f'STATUS=skipped')
print(f'RESULT_JSON={json.dumps({"extractor": EXTRACTOR_NAME, "status": "skipped", "url": url, "snapshot_id": snapshot_id})}')
sys.exit(0) # Permanent skip - staticfile already handled
print('Skipping wget - staticfile extractor already downloaded this', file=sys.stderr)
print(json.dumps({'type': 'ArchiveResult', 'status': 'skipped', 'output_str': 'staticfile already exists'}))
sys.exit(0)
# Find binary
binary = find_wget()
@@ -285,38 +277,23 @@ def main(url: str, snapshot_id: str):
error = f'{type(e).__name__}: {e}'
status = 'failed'
# Print results
# Calculate duration
end_ts = datetime.now(timezone.utc)
duration = (end_ts - start_ts).total_seconds()
print(f'START_TS={start_ts.isoformat()}')
print(f'END_TS={end_ts.isoformat()}')
print(f'DURATION={duration:.2f}')
if cmd_str:
print(f'CMD={cmd_str}')
if version:
print(f'VERSION={version}')
if output:
print(f'OUTPUT={output}')
print(f'STATUS={status}')
if error:
print(f'ERROR={error}', file=sys.stderr)
print(f'ERROR: {error}', file=sys.stderr)
# Print JSON result
result_json = {
'extractor': EXTRACTOR_NAME,
'url': url,
'snapshot_id': snapshot_id,
# Output clean JSONL (no RESULT_JSON= prefix)
result = {
'type': 'ArchiveResult',
'status': status,
'start_ts': start_ts.isoformat(),
'end_ts': end_ts.isoformat(),
'duration': round(duration, 2),
'cmd_version': version,
'output': output,
'error': error or None,
'output_str': output or error or '',
}
print(f'RESULT_JSON={json.dumps(result_json)}')
if binary:
result['cmd'] = [binary, '--no-verbose', url]
if version:
result['cmd_version'] = version
print(json.dumps(result))
sys.exit(0 if status == 'succeeded' else 1)

549
archivebox/tests/test_hooks.py Executable file
View File

@@ -0,0 +1,549 @@
#!/usr/bin/env python3
"""
Unit tests for the ArchiveBox hook architecture.
Tests hook discovery, execution, JSONL parsing, background hook detection,
binary lookup, and install hook XYZ_BINARY env var handling.
Run with:
sudo -u testuser bash -c 'source .venv/bin/activate && python -m pytest archivebox/tests/test_hooks.py -v'
"""
import json
import os
import shutil
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
# Set up Django before importing any Django-dependent modules
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.settings')
class TestBackgroundHookDetection(unittest.TestCase):
"""Test that background hooks are detected by .bg. suffix."""
def test_bg_js_suffix_detected(self):
"""Hooks with .bg.js suffix should be detected as background."""
script = Path('/path/to/on_Snapshot__21_consolelog.bg.js')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertTrue(is_background)
def test_bg_py_suffix_detected(self):
"""Hooks with .bg.py suffix should be detected as background."""
script = Path('/path/to/on_Snapshot__24_responses.bg.py')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertTrue(is_background)
def test_bg_sh_suffix_detected(self):
"""Hooks with .bg.sh suffix should be detected as background."""
script = Path('/path/to/on_Snapshot__23_ssl.bg.sh')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertTrue(is_background)
def test_legacy_background_suffix_detected(self):
"""Hooks with __background in stem should be detected (backwards compat)."""
script = Path('/path/to/on_Snapshot__21_consolelog__background.js')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertTrue(is_background)
def test_foreground_hook_not_detected(self):
"""Hooks without .bg. or __background should NOT be detected as background."""
script = Path('/path/to/on_Snapshot__11_favicon.js')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertFalse(is_background)
def test_foreground_py_hook_not_detected(self):
"""Python hooks without .bg. should NOT be detected as background."""
script = Path('/path/to/on_Snapshot__50_wget.py')
is_background = '.bg.' in script.name or '__background' in script.stem
self.assertFalse(is_background)
class TestJSONLParsing(unittest.TestCase):
"""Test JSONL parsing in run_hook() output processing."""
def test_parse_clean_jsonl(self):
"""Clean JSONL format should be parsed correctly."""
stdout = '{"type": "ArchiveResult", "status": "succeeded", "output_str": "Done"}'
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 1)
self.assertEqual(records[0]['type'], 'ArchiveResult')
self.assertEqual(records[0]['status'], 'succeeded')
self.assertEqual(records[0]['output_str'], 'Done')
def test_parse_multiple_jsonl_records(self):
"""Multiple JSONL records should all be parsed."""
stdout = '''{"type": "ArchiveResult", "status": "succeeded", "output_str": "Done"}
{"type": "InstalledBinary", "name": "wget", "abspath": "/usr/bin/wget"}'''
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 2)
self.assertEqual(records[0]['type'], 'ArchiveResult')
self.assertEqual(records[1]['type'], 'InstalledBinary')
def test_parse_jsonl_with_log_output(self):
"""JSONL should be extracted from mixed stdout with log lines."""
stdout = '''Starting hook execution...
Processing URL: https://example.com
{"type": "ArchiveResult", "status": "succeeded", "output_str": "Downloaded"}
Hook completed successfully'''
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 1)
self.assertEqual(records[0]['status'], 'succeeded')
def test_parse_legacy_result_json_format(self):
"""Legacy RESULT_JSON= format should be parsed for backwards compat."""
stdout = 'RESULT_JSON={"status": "succeeded", "output": "Done"}'
output_json = None
records = []
for line in stdout.splitlines():
line = line.strip()
if line.startswith('RESULT_JSON='):
try:
data = json.loads(line[len('RESULT_JSON='):])
if output_json is None:
output_json = data
data['type'] = 'ArchiveResult'
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 1)
self.assertEqual(records[0]['type'], 'ArchiveResult')
self.assertEqual(records[0]['status'], 'succeeded')
def test_ignore_invalid_json(self):
"""Invalid JSON should be silently ignored."""
stdout = '''{"type": "ArchiveResult", "status": "succeeded"}
{invalid json here}
not json at all
{"type": "InstalledBinary", "name": "wget"}'''
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 2)
def test_json_without_type_ignored(self):
"""JSON objects without 'type' field should be ignored."""
stdout = '''{"status": "succeeded", "output_str": "Done"}
{"type": "ArchiveResult", "status": "succeeded"}'''
records = []
for line in stdout.splitlines():
line = line.strip()
if not line or not line.startswith('{'):
continue
try:
data = json.loads(line)
if 'type' in data:
records.append(data)
except json.JSONDecodeError:
pass
self.assertEqual(len(records), 1)
self.assertEqual(records[0]['type'], 'ArchiveResult')
class TestInstallHookEnvVarHandling(unittest.TestCase):
"""Test that install hooks respect XYZ_BINARY env vars."""
def setUp(self):
"""Set up test environment."""
self.work_dir = Path(tempfile.mkdtemp())
self.test_hook = self.work_dir / 'test_hook.py'
def tearDown(self):
"""Clean up test environment."""
shutil.rmtree(self.work_dir, ignore_errors=True)
def test_binary_env_var_absolute_path_handling(self):
"""Install hooks should handle absolute paths in XYZ_BINARY."""
# Test the logic that install hooks use
configured_binary = '/custom/path/to/wget2'
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
self.assertEqual(bin_name, 'wget2')
def test_binary_env_var_name_only_handling(self):
"""Install hooks should handle binary names in XYZ_BINARY."""
# Test the logic that install hooks use
configured_binary = 'wget2'
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
self.assertEqual(bin_name, 'wget2')
def test_binary_env_var_empty_default(self):
"""Install hooks should use default when XYZ_BINARY is empty."""
configured_binary = ''
if configured_binary:
if '/' in configured_binary:
bin_name = Path(configured_binary).name
else:
bin_name = configured_binary
else:
bin_name = 'wget' # default
self.assertEqual(bin_name, 'wget')
class TestHookDiscovery(unittest.TestCase):
"""Test hook discovery functions."""
def setUp(self):
"""Set up test plugin directory."""
self.test_dir = Path(tempfile.mkdtemp())
self.plugins_dir = self.test_dir / 'plugins'
self.plugins_dir.mkdir()
# Create test plugin structure
wget_dir = self.plugins_dir / 'wget'
wget_dir.mkdir()
(wget_dir / 'on_Snapshot__50_wget.py').write_text('# test hook')
(wget_dir / 'on_Crawl__00_install_wget.py').write_text('# install hook')
chrome_dir = self.plugins_dir / 'chrome_session'
chrome_dir.mkdir()
(chrome_dir / 'on_Snapshot__20_chrome_session.js').write_text('// test hook')
consolelog_dir = self.plugins_dir / 'consolelog'
consolelog_dir.mkdir()
(consolelog_dir / 'on_Snapshot__21_consolelog.bg.js').write_text('// background hook')
def tearDown(self):
"""Clean up test directory."""
shutil.rmtree(self.test_dir, ignore_errors=True)
def test_discover_hooks_by_event(self):
"""discover_hooks() should find all hooks for an event."""
# Use the local implementation since we can't easily mock BUILTIN_PLUGINS_DIR
hooks = []
for ext in ('sh', 'py', 'js'):
pattern = f'*/on_Snapshot__*.{ext}'
hooks.extend(self.plugins_dir.glob(pattern))
hooks = sorted(set(hooks), key=lambda p: p.name)
self.assertEqual(len(hooks), 3)
hook_names = [h.name for h in hooks]
self.assertIn('on_Snapshot__20_chrome_session.js', hook_names)
self.assertIn('on_Snapshot__21_consolelog.bg.js', hook_names)
self.assertIn('on_Snapshot__50_wget.py', hook_names)
def test_discover_hooks_sorted_by_name(self):
"""Hooks should be sorted by filename (numeric prefix ordering)."""
hooks = []
for ext in ('sh', 'py', 'js'):
pattern = f'*/on_Snapshot__*.{ext}'
hooks.extend(self.plugins_dir.glob(pattern))
hooks = sorted(set(hooks), key=lambda p: p.name)
# Check numeric ordering
self.assertEqual(hooks[0].name, 'on_Snapshot__20_chrome_session.js')
self.assertEqual(hooks[1].name, 'on_Snapshot__21_consolelog.bg.js')
self.assertEqual(hooks[2].name, 'on_Snapshot__50_wget.py')
class TestGetExtractorName(unittest.TestCase):
"""Test get_extractor_name() function."""
def test_strip_numeric_prefix(self):
"""Numeric prefix should be stripped from extractor name."""
# Inline implementation of get_extractor_name
def get_extractor_name(extractor: str) -> str:
parts = extractor.split('_', 1)
if len(parts) == 2 and parts[0].isdigit():
return parts[1]
return extractor
self.assertEqual(get_extractor_name('10_title'), 'title')
self.assertEqual(get_extractor_name('26_readability'), 'readability')
self.assertEqual(get_extractor_name('50_parse_html_urls'), 'parse_html_urls')
def test_no_prefix_unchanged(self):
"""Extractor without numeric prefix should be unchanged."""
def get_extractor_name(extractor: str) -> str:
parts = extractor.split('_', 1)
if len(parts) == 2 and parts[0].isdigit():
return parts[1]
return extractor
self.assertEqual(get_extractor_name('title'), 'title')
self.assertEqual(get_extractor_name('readability'), 'readability')
class TestHookExecution(unittest.TestCase):
"""Test hook execution with real subprocesses."""
def setUp(self):
"""Set up test environment."""
self.work_dir = Path(tempfile.mkdtemp())
def tearDown(self):
"""Clean up test environment."""
shutil.rmtree(self.work_dir, ignore_errors=True)
def test_python_hook_execution(self):
"""Python hook should execute and output JSONL."""
hook_path = self.work_dir / 'test_hook.py'
hook_path.write_text('''#!/usr/bin/env python3
import json
print(json.dumps({"type": "ArchiveResult", "status": "succeeded", "output_str": "Test passed"}))
''')
result = subprocess.run(
['python3', str(hook_path)],
cwd=str(self.work_dir),
capture_output=True,
text=True,
)
self.assertEqual(result.returncode, 0)
output = json.loads(result.stdout.strip())
self.assertEqual(output['type'], 'ArchiveResult')
self.assertEqual(output['status'], 'succeeded')
def test_js_hook_execution(self):
"""JavaScript hook should execute and output JSONL."""
# Skip if node not available
if shutil.which('node') is None:
self.skipTest('Node.js not available')
hook_path = self.work_dir / 'test_hook.js'
hook_path.write_text('''#!/usr/bin/env node
console.log(JSON.stringify({type: 'ArchiveResult', status: 'succeeded', output_str: 'JS test'}));
''')
result = subprocess.run(
['node', str(hook_path)],
cwd=str(self.work_dir),
capture_output=True,
text=True,
)
self.assertEqual(result.returncode, 0)
output = json.loads(result.stdout.strip())
self.assertEqual(output['type'], 'ArchiveResult')
self.assertEqual(output['status'], 'succeeded')
def test_hook_receives_cli_args(self):
"""Hook should receive CLI arguments."""
hook_path = self.work_dir / 'test_hook.py'
hook_path.write_text('''#!/usr/bin/env python3
import sys
import json
# Simple arg parsing
args = {}
for arg in sys.argv[1:]:
if arg.startswith('--') and '=' in arg:
key, val = arg[2:].split('=', 1)
args[key.replace('-', '_')] = val
print(json.dumps({"type": "ArchiveResult", "status": "succeeded", "url": args.get("url", "")}))
''')
result = subprocess.run(
['python3', str(hook_path), '--url=https://example.com'],
cwd=str(self.work_dir),
capture_output=True,
text=True,
)
self.assertEqual(result.returncode, 0)
output = json.loads(result.stdout.strip())
self.assertEqual(output['url'], 'https://example.com')
class TestInstallHookOutput(unittest.TestCase):
"""Test install hook output format compliance."""
def setUp(self):
"""Set up test environment."""
self.work_dir = Path(tempfile.mkdtemp())
def tearDown(self):
"""Clean up test environment."""
shutil.rmtree(self.work_dir, ignore_errors=True)
def test_install_hook_outputs_installed_binary(self):
"""Install hook should output InstalledBinary JSONL when binary found."""
hook_output = json.dumps({
'type': 'InstalledBinary',
'name': 'wget',
'abspath': '/usr/bin/wget',
'version': '1.21.3',
'sha256': None,
'binprovider': 'apt',
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'InstalledBinary')
self.assertEqual(data['name'], 'wget')
self.assertTrue(data['abspath'].startswith('/'))
def test_install_hook_outputs_dependency(self):
"""Install hook should output Dependency JSONL when binary not found."""
hook_output = json.dumps({
'type': 'Dependency',
'bin_name': 'wget',
'bin_providers': 'apt,brew,env',
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'Dependency')
self.assertEqual(data['bin_name'], 'wget')
self.assertIn('apt', data['bin_providers'])
def test_install_hook_outputs_machine_config(self):
"""Install hook should output Machine config update JSONL."""
hook_output = json.dumps({
'type': 'Machine',
'_method': 'update',
'key': 'config/WGET_BINARY',
'value': '/usr/bin/wget',
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'Machine')
self.assertEqual(data['_method'], 'update')
self.assertEqual(data['key'], 'config/WGET_BINARY')
class TestSnapshotHookOutput(unittest.TestCase):
"""Test snapshot hook output format compliance."""
def test_snapshot_hook_basic_output(self):
"""Snapshot hook should output clean ArchiveResult JSONL."""
hook_output = json.dumps({
'type': 'ArchiveResult',
'status': 'succeeded',
'output_str': 'Downloaded 5 files',
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'ArchiveResult')
self.assertEqual(data['status'], 'succeeded')
self.assertIn('output_str', data)
def test_snapshot_hook_with_cmd(self):
"""Snapshot hook should include cmd for binary FK lookup."""
hook_output = json.dumps({
'type': 'ArchiveResult',
'status': 'succeeded',
'output_str': 'Archived with wget',
'cmd': ['/usr/bin/wget', '-p', '-k', 'https://example.com'],
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'ArchiveResult')
self.assertIsInstance(data['cmd'], list)
self.assertEqual(data['cmd'][0], '/usr/bin/wget')
def test_snapshot_hook_with_output_json(self):
"""Snapshot hook can include structured metadata in output_json."""
hook_output = json.dumps({
'type': 'ArchiveResult',
'status': 'succeeded',
'output_str': 'Got headers',
'output_json': {
'content-type': 'text/html',
'server': 'nginx',
'status-code': 200,
},
})
data = json.loads(hook_output)
self.assertEqual(data['type'], 'ArchiveResult')
self.assertIsInstance(data['output_json'], dict)
self.assertEqual(data['output_json']['status-code'], 200)
def test_snapshot_hook_skipped_status(self):
"""Snapshot hook should support skipped status."""
hook_output = json.dumps({
'type': 'ArchiveResult',
'status': 'skipped',
'output_str': 'SAVE_WGET=False',
})
data = json.loads(hook_output)
self.assertEqual(data['status'], 'skipped')
def test_snapshot_hook_failed_status(self):
"""Snapshot hook should support failed status."""
hook_output = json.dumps({
'type': 'ArchiveResult',
'status': 'failed',
'output_str': '404 Not Found',
})
data = json.loads(hook_output)
self.assertEqual(data['status'], 'failed')
class TestPluginMetadata(unittest.TestCase):
"""Test that plugin metadata is added to JSONL records."""
def test_plugin_name_added(self):
"""run_hook() should add plugin name to records."""
# Simulate what run_hook() does
script = Path('/archivebox/plugins/wget/on_Snapshot__50_wget.py')
plugin_name = script.parent.name
record = {'type': 'ArchiveResult', 'status': 'succeeded'}
record['plugin'] = plugin_name
record['plugin_hook'] = str(script)
self.assertEqual(record['plugin'], 'wget')
self.assertIn('on_Snapshot__50_wget.py', record['plugin_hook'])
if __name__ == '__main__':
unittest.main()