Commit Graph

4977 Commits

Author SHA1 Message Date
Nick Sweeting
57e11879ec cleanup archivebox tests 2026-03-15 22:09:56 -07:00
Nick Sweeting
9de084da65 bump package versions 2026-03-15 20:47:28 -07:00
Nick Sweeting
bc21d4bfdb type and test fixes 2026-03-15 20:12:27 -07:00
Nick Sweeting
3889eb4efa Tighten config and admin typing 2026-03-15 19:49:52 -07:00
Nick Sweeting
44cabac8d0 fix typing 2026-03-15 19:47:36 -07:00
Nick Sweeting
4756697a17 Use ruff pyright and ty for linting 2026-03-15 19:43:59 -07:00
Nick Sweeting
49436af869 Tighten CLI and admin typing 2026-03-15 19:33:15 -07:00
Nick Sweeting
5381f7584c Tighten API typing and add return values 2026-03-15 19:24:54 -07:00
Nick Sweeting
95a105feb9 small fixes 2026-03-15 19:22:06 -07:00
Nick Sweeting
f932054915 add stricter locking around stage machine models 2026-03-15 19:21:41 -07:00
Nick Sweeting
311e4340ec Fix add CLI input handling and lint regressions 2026-03-15 19:04:13 -07:00
Nick Sweeting
5f0cfe5251 add new persona tests 2026-03-15 18:46:45 -07:00
Nick Sweeting
934e02695b fix lint 2026-03-15 18:45:29 -07:00
Nick Sweeting
f97725d16f Mark version as 0.9.10rc0 (pre-release) per PEP 440
Uses rc suffix so docs system correctly identifies 0.9.x as pre-release.
Remove the rc suffix when ready to declare 0.9.x stable.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 18:44:31 -07:00
Nick Sweeting
70c9358cf9 Improve scheduling, runtime paths, and API behavior 2026-03-15 18:31:56 -07:00
Nick Sweeting
7d42c6c8b5 bump versions and fix docs 2026-03-15 17:43:07 -07:00
Nick Sweeting
e598614b05 Avoid filesystem lookups in snapshot admin list 2026-03-15 17:18:53 -07:00
Nick Sweeting
21a0a27091 Remove 7 dead functions and 4 unused imports from hooks.py
Dead functions: extract_step, run_hooks, is_parser_plugin,
get_all_plugin_icons, discover_plugin_templates, find_binary_for_cmd,
create_model_record, get_parser_plugins

Dead imports: re, signal, subprocess, django.utils.timezone

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 16:34:20 -07:00
Nick Sweeting
002de811e2 bump dep versions 2026-03-15 15:28:19 -07:00
Nick Sweeting
f0b255914d bump dep versions 2026-03-15 14:57:01 -07:00
Nick Sweeting
0ac83c8799 Wait for crawl hook records before advancing 2026-03-15 14:15:04 -07:00
Nick Sweeting
1d16038ceb Relax archive output readiness check 2026-03-15 13:31:05 -07:00
Nick Sweeting
2585ef5870 Use npm package for readability extractor installs 2026-03-15 13:09:18 -07:00
Nick Sweeting
957387fd88 Fix plugin hook env and extractor retries 2026-03-15 12:39:27 -07:00
Nick Sweeting
1fc860e901 Remove legacy binary override coercion 2026-03-15 11:45:04 -07:00
Nick Sweeting
f92ca93ae9 Skip puppeteer browser download during package install 2026-03-15 11:39:43 -07:00
Nick Sweeting
7c55259ed0 Update title HTML test for search export 2026-03-15 11:17:58 -07:00
Nick Sweeting
86fdc3be1e Refresh worker config from resolved plugin installs 2026-03-15 11:07:55 -07:00
Nick Sweeting
47f540c094 Resolve crawl provider dependencies lazily 2026-03-15 10:18:49 -07:00
Nick Sweeting
d4be507a6b Keep provider plugins enabled under whitelists 2026-03-15 09:49:45 -07:00
Nick Sweeting
82bfd7e655 Filter binary hooks by allowed providers 2026-03-15 09:32:32 -07:00
Nick Sweeting
941135d6d0 Bound URL fixture archive wait 2026-03-15 09:07:25 -07:00
Nick Sweeting
50901e5367 Align worker config propagation expectations 2026-03-15 08:47:00 -07:00
Nick Sweeting
9ef8a1b23a Stabilize secret-backed plugin CI 2026-03-15 08:37:41 -07:00
Nick Sweeting
31e883ec53 Stabilize plugin and crawl integration tests 2026-03-15 08:16:52 -07:00
Nick Sweeting
bfc1e76ff5 Update extractor tests for plugin output dirs 2026-03-15 07:32:11 -07:00
Nick Sweeting
b62064f63e Avoid recursive crawl timeout regressions 2026-03-15 07:09:15 -07:00
Nick Sweeting
5fb3709281 Run recursive crawl tests to completion 2026-03-15 06:55:35 -07:00
Nick Sweeting
68b9f75dab Stabilize recursive crawl CI coverage 2026-03-15 06:49:40 -07:00
Nick Sweeting
760cf9d6b2 Stabilize CI against expanded plugin surface 2026-03-15 06:31:41 -07:00
Nick Sweeting
1f792d7199 Restore CLI compat and plugin dependency handling 2026-03-15 06:06:18 -07:00
Nick Sweeting
6b482c62df Restore top-level list command compatibility 2026-03-15 05:04:31 -07:00
Nick Sweeting
c4d30a853f Restore index-only snapshot output links 2026-03-15 04:58:46 -07:00
Nick Sweeting
cc3e72b92f Preserve tags for index-only adds 2026-03-15 04:54:55 -07:00
Nick Sweeting
58f801c220 Fix update orphan import and host-aware tests 2026-03-15 04:51:06 -07:00
Nick Sweeting
ea94029759 Update deprecated GitHub Pages actions 2026-03-15 04:39:26 -07:00
Nick Sweeting
4fa701fafe Update abx dependencies and plugin test harness 2026-03-15 04:37:32 -07:00
Nick Sweeting
ecb1764590 switch to external plugins 2026-03-15 03:46:23 -07:00
Nick Sweeting
07dc880d0b Harden AddView config overrides to admin-only 2026-03-15 03:45:57 -07:00
Nick Sweeting
4cd50dbc58 Add Debian and Homebrew package build automation (#1771)
## Summary

This PR adds automated build and release workflows for Debian (.deb) and
Homebrew packages, replacing the legacy PPA-based distribution model. It
includes build scripts, packaging configurations, and updated GitHub
Actions workflows to generate and publish packages on release.

## Related issues

Improves package distribution and installation experience for Linux and
macOS users.

## Changes these areas

- [x] Internal architecture
- [x] Feature behavior (installation methods)

## Details

### New Build Infrastructure

**Debian Packages:**
- Added `bin/build_deb.sh` - Builds .deb packages using nfpm with
support for multiple architectures (amd64, arm64)
- Added `pkg/debian/nfpm.yaml` - nfpm configuration defining package
metadata, dependencies, and contents
- Added `pkg/debian/archivebox` - Wrapper script that activates the
virtualenv and runs archivebox CLI
- Added `pkg/debian/install.sh` - Post-install setup script that creates
virtualenv and installs dependencies
- Added `pkg/debian/archivebox.service` - Systemd service file for
running archivebox as a service
- Added `pkg/debian/scripts/postinstall.sh` and `preremove.sh` - Package
lifecycle scripts
- Added `bin/release_deb.sh` - Uploads built .deb packages to GitHub
Releases

**Homebrew Packages:**
- Added `bin/build_brew.sh` - Generates Homebrew formula using
homebrew-pypi-poet, auto-populating dependencies and checksums
- Added `brew_dist/archivebox.rb` - Homebrew formula template with
virtualenv support and service definition
- Added `bin/release_brew.sh` - Pushes updated formula to the
homebrew-archivebox tap repository

### Updated Workflows

**`.github/workflows/debian.yml`:**
- Changed trigger from `push` to `release: [published]` for
release-based builds
- Updated to Ubuntu 24.04 and added multi-architecture build matrix
(amd64, arm64)
- Replaced legacy stdeb-based build with nfpm
- Added artifact upload and GitHub Release integration
- Added test job that installs and verifies the .deb package

**`.github/workflows/homebrew.yml`:**
- Changed trigger from `push` to `release: [published]`
- Replaced bottle building with formula generation using
homebrew-pypi-poet
- Added artifact upload and automatic push to homebrew-archivebox tap
- Added test job that installs and verifies the formula

### Updated Installation Instructions

- Modified `bin/setup.sh` to download and install from GitHub Releases
.deb instead of PPA
- Updated `README.md` to document the new .deb installation method
- Updated `bin/build.sh` and `bin/release.sh` to include new build and
release scripts

## Test Plan

- GitHub Actions workflows will test package builds on release events
- Debian workflow includes automated test job that installs the .deb and
verifies `archivebox version`
- Homebrew workflow includes automated test job that installs the
formula and verifies the CLI
- Manual testing can verify installation via: `sudo apt install
./dist/archivebox*.deb` or `brew install archivebox` (after tap setup)

https://claude.ai/code/session_01Vx1EsNrNySgsc8Y67dGzCn
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/archivebox/archivebox/pull/1771"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds automated Debian (.deb) and Homebrew packaging with CI + release
orchestration, replacing the PPA and making installs via `apt` and
`brew` straightforward on Linux and macOS. Packages are thin wrappers
around pip to reduce dependencies; also fixes Docker release tags when
run via the orchestrator.

- **New Features**
- Debian `.deb` via `nfpm` (amd64/arm64): venv at
`/opt/archivebox/venv`, `/usr/bin/archivebox` wrapper (exports PATH)
used by the `systemd` service (no `--setup`); postinstall creates the
`archivebox` user, pins to the package version, enforces Python ≥ 3.13
in `install.sh`, and restarts the service after upgrades. Package
depends only on `python3`/`pip`/`venv`, recommends `git`/`curl`/`wget`.
CI pre-seeds the venv with a local wheel, runs init/status/add on native
`amd64`/`arm64`, and uploads to Releases only after tests pass; release
upload now fails loudly if the Release is missing when triggered by a
release event (skips gracefully on manual runs). `preremove.sh` stops
the service during upgrades to avoid stale venv binaries, and only
disables/removes the venv on full removal.
- Homebrew formula auto-generated with `homebrew-pypi-poet`: virtualenv
install with a single dependency on `python@3.13`, a `post_install` that
initializes in `var/archivebox` with `DATA_DIR` set, and a `caveats`
message showing the data directory. CI generates from a local sdist on
push and from PyPI on release, tests on macOS and Linuxbrew, then pushes
to the `homebrew-archivebox` tap.
- GitHub Actions: `release.yml` orchestrates `pip` → `.deb`/brew →
docker; Docker workflow treats `workflow_call` as a release so
semver/latest tags are published. Tests hardened (e.g., `tests/out`),
README `.deb` install detects arch via `dpkg` and fetches the latest
version via the GitHub API.

- **Migration**
- Ubuntu/Debian: install from Releases (`sudo apt install
/tmp/archivebox.deb`), replacing the PPA.
- macOS/Linuxbrew: `brew tap archivebox/archivebox && brew install
archivebox`.

<sup>Written for commit 36b4055304.
Summary will update on new commits.</sup>

<!-- End of auto-generated description by cubic. -->
2026-03-14 22:43:27 -07:00