alex/ArchiveBox
1
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-06 07:47:53 +10:00
Code Issues Packages Projects Releases Wiki Activity

switch to external plugins

Browse Source
This commit is contained in:
Nick Sweeting
2026-03-15 03:45:51 -07:00
parent 07dc880d0b
commit ecb1764590
256 changed files with 516 additions and 31272 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
archivebox/workers/orchestrator.py
View File

@@ -3,13 +3,16 @@ Orchestrator for managing worker processes.
The Orchestrator polls the Crawl queue and spawns CrawlWorkers as needed.
Architecture:
Orchestrator (polls Crawl queue)
└── CrawlWorker(s) (one per active Crawl)
└── SnapshotWorker(s) (one per Snapshot, up to limit)
└── Hook Processes (sequential, forked by SnapshotWorker)
Orchestrator (takes list of specific crawls | polls for pending queued crawls forever) spawns:
└── CrawlWorker(s) (one per active Crawl)
└── SnapshotWorker(s) (one per Snapshot, up to limit)
└── Hook Processes (sequential, forked by SnapshotWorker)
e.g on_Snapshot__23_save_pdf.js
on_Snapshot__24_save_screenshot.js
...
Usage:
# Default: runs forever (for use as subprocess of server)
orchestrator = Orchestrator(exit_on_idle=False)
orchestrator.runloop()