Introduction

A read-only FUSE filesystem that presents a re-tagged, reorganized view of your music library — without modifying or duplicating a single byte of the original audio. Fix tags, art, and folder structure in a SQLite store; the mount shows a clean library while your files stay exactly as they are.

What it's for

• A clean view of a messy library. Your files keep their on-disk chaos; the mount presents one consistent, template-driven tree for players and media managers.
• Tag editing without touching files. Edit the SQLite store (directly, or via the beets plugin, Picard plugin, or Lidarr integration) and the mounted view updates live — no remount, no rewrite, no re-rip anxiety.
• Lossless-by-construction experimentation. Change your tags, try a different organization scheme, new cover art — the originals are physically read-only to the mount. Backing up a current library is as easy as copying the db file.
• Hash-stable by construction. The mount never rewrites a byte, so each backing file's checksum is exactly what it was the day it arrived — anything verified by hash keeps verifying, and anything you're seeding keeps seeding, however aggressively you retag and reorganize the view on top.

Note: This project was built with AI. The general workflow was to use the superpowers skills to provide a framework. Claude Opus was used to write plans and specs which were then implemented by another model, primarily MiMo v2.5.

One of my goals in building this project was to "vibe code" something that was decisively not slop. I believe I've realized that objective and I hope that you take the project on its merits.

If you disagree, please let me know! I'd love to know where I came up short so I can improve things.

Status

All five formats ship with embedded cover art and binary-tag preservation. The serve path has been through a performance/concurrency hardening pass for real-world player and media-manager access against large libraries on HDD/SSD/NFS, and the parsers are continuously fuzzed. beets, Picard, and Lidarr plugins ship in contrib/. See the CHANGELOG for history.

Deeper reading: the architecture reference for how it works, the contributor guide for the development workflow.

Quick start

cargo install musefs    # compiles from source — needs a Rust toolchain,
                        # libfuse3-dev and pkg-config; prebuilt binaries
                        # and container images: see Installing

musefs scan ~/Music --db library.db        # ingest your library
mkdir -p ~/mnt/music
musefs mount ~/mnt/music --db library.db \
    --template '$albumartist/$album/$title'
# mount blocks until unmounted: fusermount3 -u ~/mnt/music (or Ctrl-C)

~/mnt/music now serves your library as Album Artist/Album/Title.flac — with each file's metadata generated fresh from the database, spliced in front of your original, untouched audio.

Installation

Three ways to get musefs: a prebuilt binary (no toolchain needed), building from source, or a container image. Whichever you pick, mounting needs a 64-bit FUSE-capable OS (Linux, FreeBSD, macOS) — see Platform support.

Important: Linux and FreeBSD are E2E tested. I don't have anything running macOS to test on, if you run this on one let me know if it works, or especially if it doesn't!

At present AMD64, AARCH64, and RISC-V 64 are supported. If you'd like 32-bit support please open an issue.

Prebuilt binaries

Each tagged release attaches static/portable Linux binaries for six targets:

The *-musl build is statically linked, so it runs on any Linux host of that architecture regardless of libc — glibc distros (Debian/Ubuntu/Fedora) included, not just Alpine/musl. For mixed or containerized deployments it is the simplest choice: one binary you can drop onto a glibc host and an Alpine image alike.

Download the tarball for your target from the latest release, verify it, and extract:

sha256sum -c musefs-<version>-<target>.tar.gz.sha256
tar -xzf musefs-<version>-<target>.tar.gz   # yields ./musefs

Runtime requirements: the binaries mount via FUSE's fusermount3 helper, so the target needs the FUSE userspace tools and /dev/fuse:

• Debian/Ubuntu: apt-get install fuse3
• Alpine: apk add fuse3

No glibc/libfuse install is needed for the musl binaries beyond fuse3.

Note: On Ubuntu 24.04+ (libfuse ≥ 3.17) the fusermount3 AppArmor profile only permits unprivileged mounts under whitelisted prefixes ($HOME/**, /mnt, /media, /tmp, …). Mounting elsewhere fails with fusermount3: mount failed: Permission denied — see Mounting for the whitelist and the fix.

Building from source

cargo install musefs compiles the latest release; building needs a stable Rust toolchain (2024 edition) plus the FUSE headers (libfuse3-dev) and pkg-config. To install the latest development version instead:

cargo install --git https://github.com/Sohex/musefs musefs

The same fuse3 runtime requirement as the prebuilt binaries applies.

The binary uses jemalloc as its global allocator by default (it bounds resident memory for the long-lived mount daemon under heavy concurrent reads). Distribution packagers or anyone debugging memory with valgrind/heaptrack can build against the system allocator instead with cargo build -p musefs --no-default-features (or cargo install musefs --no-default-features).

Platform support

On platforms without kernel passthrough, --mode structure-only still serves the original bytes, just through the daemon instead of the kernel.

Filename case-folding is platform-aware: --case-insensitive <true|false> defaults to true on macOS and false on Linux/FreeBSD. When enabled, filenames are compared case-insensitively — case-variant directories merge into one (first-seen casing wins) and case-variant files get a numeric suffix (e.g. Song (2)); case-insensitive mounts refresh via a full rebuild rather than the incremental fast path.

Running in containers

Container images

Each tagged release also publishes multi-arch images to the GitHub Container Registry:

docker pull selects the CPU architecture automatically. Use the -musl / :musl tags when slotting musefs into an Alpine-based stack; the default (glibc) tags suit everything else. Floating :latest / :musl track the most recent stable release only — prereleases publish only version-pinned tags.

Running musefs on the host is the simplest, best-supported option — it is an ordinary FUSE daemon and the image exists mainly to colocate musefs with containerized media managers (e.g. Lidarr). If you do containerize, mind the gotchas below.

Required flags

musefs mounts via FUSE, so the container needs /dev/fuse and the matching capability:

docker run --rm \
  --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor=unconfined \
  -v /path/to/library:/library:ro \
  -v /path/to/store:/store \
  ghcr.io/sohex/musefs:latest scan /library --db /store/musefs.db

Without --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor=unconfined the mount cannot be established.

Note: The apparmor flag may or may not be necessary depending on how your system is configured.

Note that CAP_SYS_ADMIN is a broadly privileged capability — it grants far more than FUSE mounting (mounting arbitrary filesystems, and more). It is unavoidable for an in-container FUSE mount — even rootless Podman cannot drop it; without --cap-add SYS_ADMIN the mount fails with fusermount3: mount failed: Permission denied. Under rootless Podman the capability is confined to the container's user namespace rather than the host, so its blast radius is smaller, but it is still required. Running musefs on the host needs no such capability at all.

Runs as a non-root user

The images run as a dedicated unprivileged user (default uid/gid 1000), not root — musefs mounts via the setuid fusermount3 helper and needs no root of its own. Consequences for the commands above:

• The bind-mounted store volume must be writable by that uid. Either chown 1000:1000 /path/to/store on the host, or add --user $(id -u):$(id -g) to run as your own uid. The library volume is mounted :ro, so its ownership does not matter.
• To bake an image whose user matches your host account (so no chown or --user is needed), build from source with --build-arg MUSEFS_UID=$(id -u) --build-arg MUSEFS_GID=$(id -g).
• The images include user_allow_other in /etc/fuse.conf, so a non-root --allow-other / --owner / --group mount (needed to share the mount across containers or users, below) passes musefs's pre-flight check. See Ownership and permissions.

The mount-visibility gotcha (read this before sharing the mount)

A FUSE mount made inside a container lives in that container's mount namespace. By default neither the host nor other containers can see it, so pointing a second container (your media manager) at musefs's output does not work out of the box. To share the mount you propagate it between containers through a host directory: musefs binds that directory with rshared and mounts itself there, and the consumer binds the same directory with rslave so the mount propagates in. The host directory must itself be a shared mount.

# A host directory both containers bind to, marked shared so mounts propagate.
mkdir -p /srv/musefs-mnt
mount --bind /srv/musefs-mnt /srv/musefs-mnt
mount --make-rshared /srv/musefs-mnt

# A named volume for the store, writable by the image's unprivileged user.
podman volume create musefs-store

# musefs container: bind rshared, mount musefs there with --allow-other.
podman run -d --name musefs \
  --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor=unconfined \
  -v /path/to/library:/library:ro -v musefs-store:/store \
  --mount type=bind,source=/srv/musefs-mnt,destination=/mnt/musefs,bind-propagation=rshared \
  ghcr.io/sohex/musefs:latest mount /mnt/musefs --db /store/musefs.db --allow-other

# consumer container: bind the same host path rslave; the mount propagates in.
podman run -d --name player \
  --mount type=bind,source=/srv/musefs-mnt,destination=/music,bind-propagation=rslave \
  ghcr.io/sohex/yourmediamanager:latest

Use a named volume (or an already-writable host path) for the store: a bind from a root-owned host directory is read-only to the image's unprivileged user and musefs aborts before mounting. --allow-other is required because the consumer container runs as a different uid than the musefs container; without it the consumer gets Permission denied on the mount. See Ownership and permissions.

Note: Some hardened kernels block cross-uid access to an unprivileged user's FUSE mount even with --allow-other — for example when the fuse module's allow_sys_admin_access parameter is N, or unprivileged user namespaces are restricted. If the consumer still gets Permission denied, set /sys/module/fuse/parameters/allow_sys_admin_access to Y, or run musefs and the consumer under the same uid.

Both the glibc and musl images carry the fuse3 userspace tools; pick :musl if your other containers are Alpine-based, otherwise the default tags are fine.

Sharing a host mount into a container

Running musefs on the host instead of in a container is simpler and needs no CAP_SYS_ADMIN. Mark the mount point as shared and mount musefs there with --allow-other, then bind it into the consumer container with rslave so the host's musefs mount propagates in:

# On the host: mark the mount point shared, then mount musefs with --allow-other.
mkdir -p /srv/musefs-mnt
mount --bind /srv/musefs-mnt /srv/musefs-mnt
mount --make-rshared /srv/musefs-mnt
musefs mount /srv/musefs-mnt --db /store/musefs.db --allow-other &

podman run -d \
  --mount type=bind,source=/srv/musefs-mnt,destination=/music,bind-propagation=rslave \
  ghcr.io/sohex/yourmediamanager:latest
# the container reads the re-tagged view at /music, byte-for-byte live

rslave is what keeps this working across restarts: a plain bind only captures whatever is mounted when the container starts, so it shows an empty directory if musefs mounts later and a stale view after a musefs restart.

Scanning

musefs --version (or -V) prints the build version; --help on the root or any subcommand lists its flags.

Scan

musefs scan /path/to/music --db library.db             # additive ingest
musefs scan /path/to/music --db library.db --force     # reseed existing rows

scan probes each audio file (FLAC, MP3, M4A/M4B, Ogg, WAV), recording its audio byte range, tags, and embedded art in the store. Bare scan is additive: it leaves already tracked rows alone, while --force re-seeds existing rows from disk. (Refreshing already-tracked files and pruning gone ones is the job of musefs revalidate — see Maintenance.) It takes one or more files or directories, and --jobs N controls probe parallelism. --follow-symlinks walks symlinked files and directories (off by default, so symlinks are logged and skipped). --quiet (-q) suppresses the per-target summary for scripting; scan failures still surface on stderr (raise detail with -v/-vv, or RUST_LOG=info).

scan (with or without --force) shows a live progress indicator: on an interactive terminal, a discovery spinner followed by a determinate bar (position, percent, ETA, current file); on a non-interactive stderr (piped or logged), throttled processed N/M (P%) lines. --quiet (-q) suppresses the progress indicator and the per-target summary. Each summary line ends with the elapsed time. Anything logged during the scan (skip warnings, per-file failures) is printed above the bar, which is lifted out of the way and redrawn underneath, so warnings stay readable and scroll back intact.

The per-target summary reads scanned N: … already present Z, skipped X, failed Y. already present counts files bare scan skipped because they were already tracked. skipped counts every file that isn't a supported audio format — cover art, .cue / .log / .nfo sidecars, and anything else non-audio — so a large skipped number (hundreds or thousands on a big library) is expected, not an error. A per-extension breakdown of the skip count is logged at end of scan at info (e.g. skipped 42: jpg=20, cue=10, log=8, <none>=4, so it needs -v or RUST_LOG=info), letting you tell expected sidecars from anything genuinely unexpected. failed is the one to watch: those are audio files musefs recognised by extension but could not parse, or could not store. Its own breakdown by reason is logged at end of scan too — failed 38: unparseable=30, io=5, oversize=2, rejected=1 — at warn, so it is visible without -v; a further walk errors N: unreadable=9, symlink=3 line accounts for directories and entries the walk itself could not read (those are counted in neither skipped nor failed, since no file was ever queued for them).

A rejected bucket in that breakdown means the store refused a file's rows on a constraint — the tag, art or track values it parsed were not something the schema accepts. Each one is logged with its path and the constraint text, and the rest of the library scans normally; nothing partial is stored for a rejected file, so the mount never shows a track quietly missing its tags. These are worth reporting: unlike oversize, which names a documented limit, a rejected file is a shape musefs did not anticipate.

Per-file skip messages are capped at ten per reason per scan; the rest drop to debug (-vv / RUST_LOG=debug) so an unreadable subtree or a share that vanished mid-scan cannot emit one line per file. The end-of-scan breakdowns carry the full counts either way. Format dispatch is by extension only — there is no content sniffing and no fallback to another parser, so a file whose contents don't match its extension (e.g. a FLAC named .mp3) is handed to the wrong parser, fails, and is counted here rather than retried. Renaming files across formats makes them vanish from the mount; fix the extension and rescan.

Non-standard containers are tolerated where the audio is still recoverable: a FLAC that carries one or more ID3 tags in front of the fLaC marker parses, and any tags or cover art in that ID3 header are ingested as a fallback for what the FLAC itself does not carry — see Leading ID3 tags.

If any file fails (failed Y with Y > 0), scan exits 2 even though the batch otherwise completes and the parseable files are ingested — so a pipeline like musefs scan … && musefs mount … stops on a partial or total ingest failure rather than mounting an incomplete library. A successful scan exits 0; a hard error (a missing target, an unreadable DB) still exits 1. The exit code is the only machine-detectable signal; per-file failures otherwise surface only on stderr. The full exit-code contract, and how to raise log detail on those failures, are in Logging & troubleshooting.

Content checksums and move re-identification

--checksum=none|fingerprint|full (env MUSEFS_CHECKSUM, default fingerprint) controls what content checksums scan computes and stores.

• none — no checksums (legacy behavior).
• fingerprint — compute a cheap fingerprint for each file, derived from the probe's parsed output (tags, audio bounds, embedded art). This is the default: it rides the existing probe at essentially no extra I/O cost and is sufficient for routine move detection.
• full — fingerprint plus an eager full-file SHA-256. Use this when you want collision-proof retargeting or a forensic content identity for every file.

Two flags govern how a fingerprint match is confirmed before retargeting a moved file:

• --fast (env MUSEFS_FAST) — fingerprint match is always sufficient; never reads the full file even when a stored content_hash exists.
• --strict (env MUSEFS_STRICT) — require a full-hash match; if the matched candidate has no stored content_hash, refuse the retarget and insert a fresh row instead. The default (neither flag) auto-escalates: full-hash the new file when the candidate already has a content_hash, and trust the fingerprint alone when it does not.

--fast and --strict are mutually exclusive.

Move re-identification workflow. After moving or reorganizing your backing library, run a normal musefs scan on the new locations. For each file not already in the store, the scanner looks up rows whose fingerprint matches and whose old path is gone, and retargets the unique match in place — its id, tags, and art are preserved. Move recovery only applies to rows that were fingerprinted before the move (rows scanned under --checksum=none have no fingerprint and cannot be retargeted until a later fingerprint-tier pass). Run scan after a move and ideally before any revalidate — revalidate only refreshes already tracked rows, so a moved file must be re-seeded before the maintenance pass can see it. Use revalidate --prune only when you are ready to drop missing rows.

Mounting & path templates

Mount

musefs mount /path/to/mountpoint --db library.db \
    --template '$albumartist/$album/$title' \
    --default-fallback Unknown \
    --fallback albumartist='Unknown Artist' \
    --mode synthesis        # or: structure-only

mount blocks until the filesystem is unmounted (fusermount3 -u, or Ctrl-C).

mount never creates the store — unlike scan, it requires a populated DB to already exist and exits non-zero otherwise. Interactively this is invisible (the scan → mount quick start always seeds it first), but it bites automation: a mount started at boot before anything has scanned hard-fails (and crash-loops under Restart=). Seed the store with an initial scan, or order the mount after it — see contrib/systemd.

Mounting at an arbitrary path may be denied by AppArmor. On distros that ship an AppArmor profile for fusermount3 (Ubuntu 24.04+ / libfuse ≥ 3.17), unprivileged FUSE mounts are only allowed when the mountpoint is under a whitelisted prefix — the shipped profile permits $HOME/**, /mnt, /media, /tmp, /cvmfs, $XDG_RUNTIME_DIR, plus flatpak dirs. Mounting elsewhere (e.g. a data volume at /data/...) fails with fusermount3: mount failed: Permission denied, and the kernel audit log shows apparmor="DENIED" operation="mount" … profile="fusermount3". The mountpoint's own ownership is irrelevant — AppArmor rejects the mount() syscall first. Fix it by mounting under a permitted prefix, or by whitelisting your prefix in /etc/apparmor.d/local/fusermount3 (the shipped profile ends with include if exists <local/fusermount3>).

Two modes:

• synthesis (default) — files carry metadata freshly generated from the store, spliced ahead of the original audio bytes.
• structure-only — files are served byte-for-byte as they are on disk; only the directory tree is virtual.

Edit tags or art in the database while mounted (another scan, a beets/Picard/Lidarr sync, raw SQL) and the view refreshes automatically.

Run musefs <command> --help for the full flag list.

Raising log detail

A mount logs at warn by default, which means it says nothing while it is working normally. The lines that explain what it did — the post-mount summary naming the store, mountpoint, and file/directory counts, and whether reads are being served by the kernel or by the daemon — are info, so add -v (or set RUST_LOG=info) when a mount is not behaving as expected:

musefs mount /path/to/mountpoint --db library.db -v

See Logging & troubleshooting for the full level model, the serve-path warn limiter, and reading the log under systemd or in a container.

Path templates

Paths come from a beets-style template (matched case-insensitively; any tag key in the store works):

• $field / ${field} — substitute a tag field (e.g. $artist, $album, $title, $tracknumber, $date, $genre).
• ${albumartist|artist} — fallback chain: the first present field wins, before the --default-fallback value (default Unknown) is used.
• A missing field resolves in order: the field's value, then a per-field fallback from --fallback FIELD=VALUE (repeatable, e.g. --fallback albumartist='Unknown Artist'), then --default-fallback. Per-field fallbacks let one field default differently from the rest.
• --skip-on-missing — drop a track from the mount entirely when a top-level template field stays unresolved, instead of substituting --default-fallback. Per-field --fallback chains and [ … ] sections are unaffected (a field resolved via its fallback counts as present, and section fields stay optional). Handy when an external tool tags only some tracks, e.g. --template '$!{beets_path}' --skip-on-missing hides tracks beets left without a beets_path (such as deduplicated albums).
• [ … ] — conditional section: the bracketed text is emitted only when at least one field inside it is present. So $album[ - CD $disc] yields Album - CD 2, or just Album on a single-disc release. Write $[ / $] for literal brackets.
• $!{field} — path field: the value's / are kept as directory separators (each segment sanitized; empty/./.. dropped). Lets an external tool precompute a whole relative path into one tag and mount it as --template '$!{beets_path}'.

Anything else is literal. Name collisions get a deterministic (2), (3), … suffix. Every rendered component is capped at 255 bytes (NAME_MAX, truncated on a UTF-8 boundary, extension preserved), and a plain field whose value is exactly . or .. is dropped rather than creating an unusable directory. The default template is $albumartist/$album/$title.

Brackets and braces must be balanced: an unclosed [ section or an unterminated ${ / $!{ field is rejected at mount time with an error naming the problem, rather than silently folding the rest of the template into the open construct. To check a template before committing to a mount, add --dry-run: it validates the template, prints a sample of the paths the mount would expose along with the total file and directory counts, then exits without mounting.

Tuning & metrics

Tuning

The defaults are sensible for most setups, including the two measured storage wins — daemon-level backing read-ahead (--read-ahead-budget-mib, the single biggest win for NFS/remote) and keeping the kernel page cache across opens (--keep-cache, on by default, ~3× faster reopen on HDD/NFS). The kernel-level read-ahead / background knobs have little measurable effect (see the storage-tunables benchmarks for the methodology and numbers).

Filename case-folding (--case-insensitive) is platform behaviour rather than a performance knob — see Platform support.

Memory footprint

--read-ahead-budget-mib is the only direct byte-budget knob, but on a sizeable library it is not the biggest number. Two other components dominate, and they grow at different moments:

• The virtual tree — one node per rendered file and directory, plus the inode allocator's path map — is built at mount, held resident for the lifetime of the mount, and scales with the track count, not with I/O.
• Per-worker SQLite read connections. Metadata ops run on a worker pool (2× the CPU count by default; see --workers), and each worker lazily opens its own read-only connection the first time it serves a query. A full library enumeration — the first thing any media indexer does — touches every worker, so the daemon reaches its true steady state only after that first walk. This scales with the worker count, not the track count (bounded: each connection carries a 512 KiB page-cache cap plus roughly as much again in per-connection state). --workers is the lever: on a many-core host that serves few concurrent readers, --workers 8 keeps the read path fully concurrent while capping this component at 8 connections. The cap is real: pool tasks run behind a panic boundary, so a worker is never retired and replaced by one that would open a connection of its own.

Measured on Linux, release build, 64 workers (32 CPUs):

Size a host against the post-enumeration steady state — ~3 KB of RAM per track at this worker count, so ~310 MB at 100,000 tracks and ~3.1 GB at a million — on top of the read-ahead budget (64 MiB by default). The tree-build peak, the previous sizing figure, no longer exceeds the post-build steady state; what an operator actually observes over time is the higher post-walk plateau. The tree's own cost is dominated by rendered names and paths rather than by audio, so a deeply nested --template (more path components, longer names) raises it and a flatter one lowers it.

The current row was measured on a host with transparent hugepages set to always, which inflates it: khugepaged collapses the allocator's sparsely-used ranges into fully-resident 2 MB pages, worth about +130 MB on this corpus. On such a host (check /sys/kernel/mm/transparent_hugepage/enabled), starting the daemon with _RJEM_MALLOC_CONF=thp:never in the environment — or setting the system policy to madvise — brought the post-enumeration figure down to 477 MB (~2.4 KB per track) with no measured cost.

Since v1.3.0 the tree stores each name once and shares that one allocation with every index keyed on it, which cut the tree's own resident cost from 1.71 KB to 1.28 KB per track (−25%) on a 200,000-track build. Rendered paths are now shared the same way: the inode allocator's key and the refresh snapshot's entry for a track are one allocation rather than two, so the tree carries one fewer whole path per track. That is worth −78 B/track (−7%) on a short $artist/$album/$title corpus and −173 B/track (−14%) when the same three levels carry long names — the saving is one full path, so it grows with the template's depth and with name length.

musefs-core/tests/tree_footprint.rs re-measures the per-track cost on every cargo test run and fails above a ceiling, so a regression surfaces without a full-mount run; set MUSEFS_TREE_FOOTPRINT_TRACKS to re-measure at library scale.

Gauges in the metrics surface below size a live mount:

• musefs_process_resident_bytes (Linux) is the whole-process RSS as the OS charges it — the number to size against, and the only one that sees every allocator in the process.
• musefs_sqlite_memory_bytes is what SQLite holds across all connections. SQLite is C and allocates through libc, so on a jemalloc build the musefs_alloc_* gauges exclude it entirely — musefs_alloc_resident_bytes is the Rust heap only, not an RSS proxy. Watch the process gauge for capacity and the difference between it and the allocator gauges for C-side growth.
• musefs_tree_nodes (live virtual-tree inodes: files plus directories) and musefs_inode_paths (paths interned by the inode allocator — up to 2× the live node count between prunes, since retired paths are kept until they outnumber live ones) track the tree's share.

Metrics

musefs mount optionally exposes runtime telemetry through a synthetic .musefs-metrics/ directory at the mount root:

musefs mount /mnt/music --db library.db --expose-metrics   # or: MUSEFS_EXPOSE_METRICS=1
cat /mnt/music/.musefs-metrics/metrics

# HELP musefs_uptime_seconds Seconds since the mount started.
# TYPE musefs_uptime_seconds gauge
musefs_uptime_seconds 60
# HELP musefs_handles_open Open file handles in the core slab.
# TYPE musefs_handles_open gauge
musefs_handles_open 3
# HELP musefs_cache_header_hits_total Raw header-cache key hits; a hit may still trigger a content-version rebuild.
# TYPE musefs_cache_header_hits_total counter
musefs_cache_header_hits_total 100

Serve-path failure warnings are rate-limited (a burst of 10 per 30-second window; the rest drop to debug), so a mount that is failing fast logs far less than it fails. musefs_serve_warns_suppressed_total counts what the limiter dropped, which is the signal that separates "the serve path is quiet" from "the serve path is failing faster than it can log". Suppression is bursty by construction, so watch the counter's rate rather than any single sample, and raise the log level to debug to see the individual failures behind it.

--expose-metrics (default off) is a runtime flag that gates the virtual file; it is unrelated to the compile-time metrics cargo feature, which adds syscall counters (opens, preads, etc.) to the output. The jemalloc allocator stats require a build with the jemalloc feature, which is the default.

The metrics file advertises st_size == 0 (like /proc), so use an EOF-aware reader — cat, head -c, or the Prometheus textfile collector — not a stat-and-read-by-size approach.

Maintenance

Refreshing the store (musefs revalidate)

musefs revalidate is the maintenance pass over a library you have already scanned. It re-probes only the tracks whose backing file changed on disk (by size / mtime / ctime) and refreshes their structural serving data — audio byte range, content checksums, and FLAC structural blocks — while preserving the curated tags, art, and binary tags in the store. Unchanged files are skipped, and files not yet in the store are ignored (ingesting new files is scan's job — see Scanning).

musefs revalidate /path/to/music --db library.db          # refresh changed rows
musefs revalidate /path/to/music --db library.db --prune  # also delete gone tracks

By default revalidate never deletes anything. Pass --prune to delete tracks whose backing file is gone from disk (scoped to the revalidated root) and garbage-collect any art left unreferenced. Pruning is opt-in because it removes a track's curated metadata along with its row, so a transient mount blip or an unplugged drive can't silently drop your edits.

It shares scan's probe flags — --jobs N, --follow-symlinks, --quiet / -q, and --checksum (which also backfills missing checksums on a changed row) — and shows the same live progress indicator. The per-target summary reads revalidated N: U updated, C unchanged, P pruned, F failed.

When to run it

Run revalidate after the contents of your backing library change on disk — files re-encoded, retagged at the source, or deleted — to bring the store's structural data back in sync without disturbing your curated metadata. To pick up newly-added files run scan (which is additive); to drop rows for deleted files add --prune. After moving files, run scan first so they retarget, then revalidate --prune — see Move re-identification.

Compacting the store (musefs vacuum)

The SQLite store only grows as you use it: deleting tracks (beets/Lidarr prunes), garbage-collecting orphaned art, and the schema migration all leave free pages behind that are not automatically reclaimed. Because embedded art is stored inline (up to ~16 MiB per image), a library that has churned art can carry significant dead space.

musefs vacuum compacts the store and reports how much it reclaimed:

musefs vacuum --db library.db        # or: MUSEFS_DB=library.db musefs vacuum

vacuumed library.db: 412.7 MiB → 318.2 MiB (reclaimed 94.5 MiB)

It runs SQLite's VACUUM followed by a WAL checkpoint, rewriting the database into a compact form.

Run it while unmounted

VACUUM needs a write lock on the store and rewrites the whole file. Run it when nothing else is using the database — no mount, no scan. If the store is in use, the command fails with an actionable error rather than fighting for the lock:

error: the store is in use — unmount the filesystem or stop any scan before vacuuming

Notes

• Full rewrite. Each run rewrites the entire database and transiently needs free disk space roughly equal to the store size (it builds a complete copy before swapping). Running it again on an already-compact store is safe and reports (already compact).
• May upgrade the schema. Like every musefs command that opens the store for writing, vacuum migrates an older store to the current schema version before compacting.

Ownership, permissions & config

Ownership and permissions

By default the mount presents the launching process's uid/gid and read-only permission bits (555 dirs, 444 files), and is reachable only by the user who performed the mount (and root).

To present a different owner — e.g. a media-server service account — and let that account actually reach the mount, pass --owner/--group (or --allow-other). Either makes musefs mount with allow_other and default_permissions: other users can traverse the mount, and the kernel enforces the presented owner/mode bits instead of ignoring them.

The default 444/555 bits are world-readable, so any account can read once allow_other is on. To restrict the mount to the presented owner/group, drop the world bits (e.g. --file-mode 440 --dir-mode 550) — only then does --owner/--group gate access rather than merely label it.

Non-root mounts need user_allow_other. When you are not root, libfuse refuses an allow_other mount unless /etc/fuse.conf contains a line user_allow_other. musefs checks this before mounting and fails with an explanatory error if it is missing; add the line to /etc/fuse.conf, or run musefs as root. (This is libfuse/system policy, not a musefs restriction.) The published container images already include this line, so non-root allow_other mounts work out of the box there.

--allow-other grants other users — but not root. A FUSE mount made with allow_other (not allow_root) is reachable by other unprivileged users, yet root specifically cannot traverse or stat it when it is owned by another user. This surprises root-run tooling (Ansible, boot scripts):

• mountpoint -q <mnt> / stat <mnt> run as root report it as not a mountpoint — they try to stat through the mount and get EACCES. Detect the mount from root with findmnt <mnt> or /proc/mounts instead, which read the mount table rather than the filesystem.
• Don't have root manage the mountpoint directory while it is mounted: a root task that re-asserts the directory (e.g. Ansible file: state=directory) fails with EACCES/EEXIST on every run after the first. Create the directory before mounting, or run such tasks as the mounting user.

Configuring with environment variables

Every scalar mount and scan flag can also be set with a MUSEFS_* environment variable — uppercase the long flag and turn dashes into underscores (e.g. --poll-interval-ms → MUSEFS_POLL_INTERVAL_MS, the mount mountpoint → MUSEFS_MOUNTPOINT). An explicit flag always overrides its env var, which overrides the default. Boolean flags (e.g. MUSEFS_KEEP_CACHE, MUSEFS_REVALIDATE, MUSEFS_FOLLOW_SYMLINKS, MUSEFS_QUIET, MUSEFS_ALLOW_OTHER, MUSEFS_CASE_INSENSITIVE, MUSEFS_EXPOSE_METRICS, MUSEFS_FAST, MUSEFS_STRICT) accept a case-insensitive boolish value — true/false, yes/no, on/off, 1/0 — and reject anything else. The repeatable --fallback and the scan targets are command-line only. See contrib/systemd/musefs.conf.example for a commented example covering the common settings.

These variables are read the same way no matter how musefs is launched: exported into the shell before running the binary directly (MUSEFS_DB=… musefs mount), set via a systemd EnvironmentFile= or Environment= directive, or passed into a container with -e/--env-file. The configuration surface is identical across all three; the sections below just show the per-deployment wiring.

Running as a systemd user service

To run musefs on the host at login, drop-in units live in contrib/systemd/: a musefs.service mount daemon, an optional musefs-scan.timer for periodic re-scans, and a commented musefs.conf.example holding every MUSEFS_* setting. Copy the units to ~/.config/systemd/user/, copy the config to ~/.config/musefs/musefs.conf, edit MUSEFS_MOUNTPOINT and MUSEFS_DB, then systemctl --user enable --now musefs.service. See the systemd integration guide for the full walkthrough and the PATH / linger gotchas.

Logging & troubleshooting

Every musefs crate reports through the log facade; the musefs binary installs the sink and writes it to stderr. Nothing is logged to the mount or to the store, so the log is wherever that process's stderr goes — your terminal, the journal, or the container log.

Raising log detail

The default floor is warn: enough that a failure is never silent, quiet enough to leave running. -v raises it, and repeats stack:

-v / --verbose is global — musefs -v mount … and musefs mount … -v are equivalent.

RUST_LOG wins. The -v count only supplies the default filter, so an explicit RUST_LOG in the environment takes precedence and -v then has nothing to do. That is the lever to reach for when you cannot edit the command line (a systemd unit, a container ExecStart), and it is also the only way to filter per crate:

RUST_LOG=info musefs mount /mnt/music --db library.db
RUST_LOG=warn,musefs_fuse=debug musefs mount /mnt/music --db library.db

Targets are crate names with underscores: musefs, musefs_cli, musefs_core, musefs_db, musefs_fuse, musefs_format. There is no MUSEFS_* variable for verbosity — unlike the mount and scan flags, this one is RUST_LOG or -v only.

Two kinds of output ignore the log level entirely: scan/revalidate progress and per-target summaries (suppressed with --quiet / -q, see Scanning), and a handful of CLI diagnostics printed directly to stderr — warning: mountpoint … is not empty, the --file-mode / --dir-mode write-bit warnings, and the final musefs: <error> on a hard failure.

They share stderr with the log, but they do not collide with it: the progress bar is drawn through a target the log sink suspends around every record, so a warning emitted mid-scan clears the bar, prints, and the bar is redrawn beneath it. Raising verbosity during an interactive scan is safe.

What each level buys

warn — the default

• Serve-path failures behind the mount: read(…) failed: …, lookup(…) failed: …, and read(…) rejected: EAGAIN load-shedding. These go through the warn limiter.
• Scan-time refusals: skipping <path>: <reason>, skipping <path>: no parseable audio metadata. These are capped per reason — see Scanning.
• The end-of-scan failure summaries: failed 38: unparseable=30, io=5, oversize=2, rejected=1 (these reasons, plus panicked, partition the failed count the CLI reports and the exit-2 signal keys on; empty buckets are omitted) and walk errors 12: unreadable=9, symlink=3 for entries the walk never queued. Neither fires on a healthy library.
• Degraded-but-correct fallbacks: incremental tree mutation failed …; falling back to full rebuild, poll_refresh failed, inval_inode(…) failed.
• Tag keys dropped during Vorbis synthesis (track N: dropping tag key … (not a valid field name)). Like the serve-path failures above, these go through the warn limiter.
• An in-place store schema upgrade: upgrading store schema at … from version 1 to version 3; this is irreversible …. Once per store per schema bump — it is at warn precisely so it lands in your scrollback before you ever try to run an older musefs against that store.
• scan --revalidate deprecation.
• error sits above it, reserved for a caught panic in a synthesis or scan worker, a scan aborted mid-ingest, and recovered lock poisoning.

-v (info) — "why is my mount doing this"

This is the level worth knowing about. The lines that explain a mount's behaviour are all info, so by default they are invisible:

• The pre-mount line naming the store, mountpoint, and template, then the post-mount summary once the session is actually serving:

  musefs mounted at /mnt/music: 41230 files in 3812 directories (Synthesis)
  

  An empty or wrong --db is otherwise indistinguishable from a correctly serving one — the mount succeeds either way; the counts are the tell.

• The passthrough decision. In structure-only mode on Linux 6.9+ musefs registers the backing fd with the kernel so reads bypass the daemon; when that is not available it says so once and serves reads itself:

  FUSE passthrough unavailable; serving reads through the daemon: <error>
  StructureOnly mount without CAP_SYS_ADMIN: kernel passthrough unavailable; reads will be served by the daemon
  StructureOnly mount: kernel passthrough is Linux-only; reads will be served by the daemon
  

  This is the single best explanation for "why is my structure-only mount slower than I expected".

• changelog gap; falling back to full refresh — the mount slept past more external edits than the changelog ring retains, so it rebuilt the tree wholesale instead of applying a diff. Correct, just more expensive.

• The per-extension breakdown of a scan's skip count (skipped 42: jpg=20, cue=10, …). This sits at info rather than warn because cover art and .cue/.log sidecars are the normal contents of a music library, so it fires on every healthy scan — and a warning that always fires teaches you to filter warnings, which would cost you the failure summary that matters. The skipped N total is in the CLI's per-target summary at any level; -v gates only the breakdown.

• store schema at … is now at version 3 (took 0.4s) — the completion half of the upgrade announcement above.

If a mount is behaving oddly and you only reach for one thing, reach for -v.

-vv (debug)

• Everything the warn limiter dropped, tagged (over warn budget).
• Routine tree-shape misses that never warn: lookup(…) failed: no such inode: … and friends (ENOENT / EISDIR / ENOTDIR), which a kernel path probe and a stale inode after a refresh generate constantly.
• opendir(…) over the 1024-handle cap: serving it statelessly — the degraded readdir path a parallel walk can push a mount onto. The operator-facing signal for this is the musefs_dir_handle_rejections_total counter in the metrics surface; the log line is per-occurrence.
• Scan-walk detail: duplicate backing targets, symlink handling.
• Scan skips past their per-reason cap. After the first 10 of a given reason, one line says further "<reason>" skips are logged at debug … and the rest go here; the end-of-scan summary still has the totals.

The serve-path warn limiter

Serve-path failure warns are rate-limited process-wide: 10 warns per 30 s window. Over budget, a message is downgraded to debug with (over warn budget) appended and counted; the first warn of the next window carries the tally:

read(1042) failed: backing file changed since scan: /music/a.flac (317 similar serve-path warnings suppressed in the last 30s)

Read that line carefully — the suppressed failures are not necessarily the same failure as the one printed. The count is a volume signal, not a multiplicity of that message.

The limiter exists because the failure mode it guards is per-file: a library enumeration over a corpus whose backing files have moved would otherwise emit one warn per file — hundreds of thousands of lines, tens of MB of log, for a single walk. One budget is shared across the whole serve path: the reply_errno warn arm (the failures behind lookup, getattr, open, read, readdir), the read load-shedding EAGAIN line, and the synthesis warns — dropped Vorbis tag keys, over-cap art, art-blob read failures — which are emitted per cache miss and so scale the same way. Scan-time warns are capped separately and per reason; see Scanning.

Sharing the budget does not merge the log targets: each message is still attributed to the module that raised it (musefs_fuse, musefs_core::reader, musefs_core::mapping), so per-crate RUST_LOG filtering works as described above.

To see every suppressed failure, run at -vv (or RUST_LOG=debug): nothing is discarded, only downgraded. For the rate rather than the events, the musefs_serve_warns_suppressed_total counter on the metrics surface counts every downgraded warn since start — a scrape landing between bursts sees a quiet log either way, so the counter is what distinguishes a healthy serve path from a throttled one.

Reading the logs under systemd

The user units send stderr to the journal:

journalctl --user -u musefs -f          # follow the mount
journalctl --user -u musefs-scan -e     # the last of a scan run

ExecStart in the shipped unit is a bare musefs mount driven by MUSEFS_* environment variables, so the least invasive way to raise detail is the environment rather than the command line — add RUST_LOG=info to ~/.config/musefs/musefs.conf, or set it in a drop-in (systemctl --user edit musefs):

[Service]
Environment=RUST_LOG=info

Then systemctl --user restart musefs. Adding -v works too, but it means editing ExecStart itself.

Reading the logs in a container

stderr is the container's log stream, so nothing special is needed to capture it:

docker logs -f musefs        # podman logs -f musefs

Raise detail with -e RUST_LOG=info on docker run / podman run, or by appending -v to the command after the image name. Note that the most common container failure — a missing --device /dev/fuse or --cap-add SYS_ADMIN — is not a log line at all: it is a hard error, musefs: mounting at /mnt/musefs: …, and the container exits 1. See Running in containers.

Exit codes

Exit 2 is what makes a partial ingest machine-detectable, so musefs scan … && musefs mount … stops instead of mounting an incomplete library — the per-file failures otherwise surface only on stderr. It overlaps clap's usage-error code, but a usage error fails before any work starts, so the two are never ambiguous in a pipeline that got as far as running. See Scanning for what failed counts.

When the log is not the answer

• A file that will not open or errors on read is usually a backing file that changed since its last scan — see the FAQ and run musefs revalidate.
• For steady-state behaviour (handle counts, cache hit rates, read errors, degraded-readdir rejections) the counters in .musefs-metrics/ are the surface to watch; the log reports events, not rates.

FAQ

Does musefs ever write to my audio files? No. The mount is read-only and the scanner only reads. The served files are assembled on the fly: generated metadata plus positioned reads of your originals. Nothing is ever copied or rewritten.

Where do my edited tags live? In the SQLite store (--db). Edit it with the beets or Picard plugins, the Lidarr integration, or with plain SQL — the schema is a documented, stable contract (see the SQLite store).

Do edits show up without remounting? Yes. The mount polls the database (debounced) and picks up external commits automatically, with stable inodes across refreshes — even files held open keep working.

Can I write through the mount? No — and it's not planned. Out-of-band editing against the store is the design: it's what guarantees your originals can never be corrupted.

Is it fast enough for a big library on a NAS? That's the design target: synthesized headers are cached, blocking reads run on a worker pool so a slow disk never stalls the filesystem, and read-ahead, cache TTLs, and poll intervals are all tunable. In structure-only mode on kernel 6.9+, reads can bypass the daemon entirely via FUSE passthrough (needs CAP_SYS_ADMIN).

A file in the mount won't open / reads error — why? The most common cause is a backing file that changed since its last scan (musefs refuses to serve a file whose size, mtime, or ctime drifted, rather than splice at stale offsets). Run musefs revalidate to re-probe it.

Supported formats

musefs synthesizes fresh metadata for each supported container while serving the original audio bytes verbatim. Each format has its own page for the exact synthesis behavior and lossy edges.

FLAC

How musefs scans and synthesizes native FLAC files (.flac). FLAC inside an Ogg container is a different beast — see Ogg. For the segment model these layouts plug into, see the segment model.

What round-trips

• All text tags. Canonical keys (title, artist, albumartist, date, tracknumber, …) map to their conventional Vorbis field names via the shared vocabulary (musefs-format/src/tagmap.rs); any other field round-trips verbatim by its own name. Multi-value fields keep their order. User-defined keys that are not legal Vorbis field names (empty, containing =, control characters, or non-ASCII bytes — i.e. outside ASCII 0x20–0x7D minus =) are dropped on synthesis and logged; they cannot round-trip by name.
• Binary metadata blocks. APPLICATION and CUESHEET blocks are captured at scan time as binary tags (an APPLICATION payload includes its 4-byte application id) and re-emitted on synthesis, streamed from the DB rather than held in memory.
• Embedded pictures. Each PICTURE block round-trips with its MIME type, picture type, description, and dimensions; image bytes are stored content-addressed and streamed at read time.
• Structural blocks. STREAMINFO and SEEKTABLE are preserved bit-exact. They are captured into the read-only structural_blocks store at scan time (external tools must not edit them) and re-emitted on synthesis.

Leading ID3 tags

FLAC does not define an ID3 tag, but files that put one (or several) in front of the fLaC marker exist in the wild — usually a blank header left behind by a converter. musefs scans them rather than skipping them:

• The tag run is stepped over to find the fLaC marker, and the recorded audio offset counts it, so the audio segment still reads the right range out of the backing file. A tag is stepped over by its declared size, whatever its version: the ID3v2 header has the same shape in every version, and the spec's own rule for a version a reader does not understand is to ignore the tag. A header musefs cannot parse the frames of is therefore still skipped correctly — only its contents go unread. What is required is that the header matches the spec's detection pattern ($49 44 33 yy yy xx zz zz zz zz, both version bytes below $FF, every size byte below $80) and that its declared length lands where the next thing in the file starts — the following ID3v2 header, or the fLaC marker for the last tag in the run. A header that fails either test contradicts the file it is in, and the file is skipped rather than guessed at.
• The tag's text frames and APIC pictures are ingested as a fallback: the FLAC's own VORBIS_COMMENT / PICTURE blocks win, and the ID3 tag only supplies keys the FLAC does not define. The rule is all-or-nothing per key, so a multi-value field never mixes values from both sources. A file whose tags live only in the ID3 header is therefore still ingested with its tags.
• A 128-byte trailing ID3v1 tag is trimmed from the audio length, so it is not served as audio. This check runs only for files that carry a leading ID3v2 tag: a stock FLAC pays no extra read, and audio that happens to spell TAG 128 bytes from the end is never mistaken for a trailer.
• Neither tag survives into the synthesized file — see the lossy edge below.

Lossy edges

• A leading ID3v2 tag is not preserved: it is metadata, and the synthesized file is regenerated metadata in front of untouched audio. Its contents are ingested first (above), so they come back as Vorbis comments and PICTURE blocks — the served file is a stock FLAC that starts at fLaC. This matches how MP3's original ID3v2 tag is treated.
• A trailing ID3v1 tag is dropped without being read. Unlike the leading tag, its fields are never parsed into tags — it is only excluded from the audio length so its 128 bytes are not served as audio. This mirrors MP3, where ID3v1 is likewise not read; populate those tags through the store instead.
• PADDING blocks are dropped — the synthesized file carries no padding.
• Metadata blocks of unknown/reserved types are dropped at scan time.
• A PICTURE block whose picture type falls outside the standard 0–20 range is clamped to 0 (Other) at scan time, matching the store's track_art.picture_type CHECK. This shared PICTURE parser also serves FLAC-in-Ogg, so the same clamp applies there.
• The VORBIS_COMMENT vendor string is replaced with musefs's own.
• Vorbis field names are case-insensitive by spec; musefs re-emits canonical keys under their conventional uppercase names and upper-cases unknown field names. A field stored as MixedCase comes back as MIXEDCASE — same field to a conforming reader, different bytes.

How synthesis works

flac::synthesize_layout (musefs-format/src/flac.rs) builds the layout in this order — an inline metadata region, DB-streamed payloads, then the untouched audio:

 offset 0
 ┌──────────────────────────────────────────────┐ ┐
 │ █ "fLaC" marker                      (Inline) │ │
 │ █ STREAMINFO / SEEKTABLE, bit-exact  (Inline) │ │ generated
 │ █ VORBIS_COMMENT rebuilt from DB     (Inline) │ │ metadata
 │ ▒ APPLICATION / CUESHEET bodies   (BinaryTag) │ │ region
 │ █ PICTURE framing + ▒ image bytes  (ArtImage) │ │
 ├──────────────────────────────────────────────┤ ┘
 │ ░ audio frames, verbatim       (BackingAudio) │
 └──────────────────────────────────────────────┘
 EOF     █ inline-generated   ▒ DB-streamed   ░ untouched backing

1. Inline — the fLaC marker plus the preserved structural blocks (STREAMINFO, SEEKTABLE, sorted by block type) and a VORBIS_COMMENT block regenerated entirely from the DB tag rows.
2. BinaryTag — one segment per stored APPLICATION/CUESHEET block, streamed from the DB at read time.
3. ArtImage — one PICTURE block per linked art row; the block framing is inline, the image bytes stream from the blob store.
4. BackingAudio — the original audio frames, served by positioned reads at the stored audio_offset/audio_length.

Structural blocks normally come from the structural_blocks store. A database scanned before that store existed has no rows there; synthesis then falls back to re-reading the file's front for every preserved block (carrying APPLICATION/CUESHEET inline and suppressing the streamed binary tags so nothing is emitted twice). A re-scan upgrades the track to the streamed path.

Quirks & invariants

• The audio frames are never touched: the backing segment starts exactly at the scanned audio offset, and the byte-identical-audio property is asserted by musefs-format/tests/proptest_flac.rs and the mutagen interop suite (musefs-core/tests/interop_emit.rs).
• Synthesis re-parses its own inline output in tests (flac_tag_roundtrip_is_stable): the regenerated front must be a valid FLAC metadata region whose computed audio boundary equals the layout's header length.
• Block-body sizes are bounded at parse time (MAX_BLOCK_BODY); a crafted file cannot force a huge allocation.
• The parser now rejects (at scan and synthesis) any FLAC whose metadata does not begin with exactly one 34-byte STREAMINFO block; a crafted store providing malformed structural rows fails synthesis with a controlled error rather than emitting decoder-rejected output.

MP3

How musefs scans and synthesizes MP3 files (.mp3) and their ID3v2 metadata. For the segment model these layouts plug into, see the segment model. The ID3v2 builder described here is shared with WAV's embedded id3 chunk — see WAV.

What round-trips

• Canonical text tags (title, artist, albumartist, date, tracknumber, …) map to their standard ID3v2 text frames (TIT2, TPE1, TPE2, TDRC, TRCK, …) via the shared vocabulary (musefs-format/src/tagmap.rs). NUL-separated multi-value frames yield one tag row per value and are re-emitted NUL-separated in a single frame.
• Vocabulary TXXX keys (ReplayGain fields, MusicBrainz album/artist ids) round-trip through TXXX frames with their fixed, exact-case descriptions (e.g. MusicBrainz Album Id).
• Unmapped standard text frames round-trip keyed by their own frame id: a TSSE (or a legacy v2.3 TYER) comes back as the same frame inside the synthesized tag.
• Other user-defined keys round-trip as TXXX frames keyed by their own description, original casing preserved.
• Comments and lyrics (COMM/USLT): one tag row per frame. A frame with a placeholder language (XXX/und/empty) and no descriptor folds to the shared comment/lyrics key; one carrying a real language or descriptor is keyed id3:COMM:<lang>:<desc> / id3:USLT:<lang>:<desc> so per-language or description-keyed frames stay distinct, and both fields are restored on synthesis.
• Ratings and play counts: a POPM frame is promoted at scan time to rating (the raw 0–255 byte) and playcount (omitted when 0) text tags, and rebuilt as a POPM frame on synthesis.
• MusicBrainz track id: a UFID frame with the http://musicbrainz.org owner is promoted to musicbrainz_trackid and rebuilt with the same owner.
• Opaque binary frames, byte-exact: PRIV, GEOB, SYLT, MCDI, URL (W***) frames, non-MusicBrainz UFIDs, and unknown frames are captured verbatim (frame id + raw body) and re-emitted streamed from the DB (BinaryTag segments) — never held in memory.
• Embedded pictures (APIC): MIME type, picture type, and description round-trip; image bytes are stored content-addressed and streamed.

Lossy edges

• The synthesized tag is always ID3v2.4, regardless of the source tag's version (v2.2/v2.3 tags are parsed but never re-emitted as such).
• A COMM/USLT frame folded to the shared comment/lyrics key (placeholder language, no descriptor) is re-emitted with language XXX and an empty descriptor, so a source und placeholder comes back as XXX. Frames carrying a real language or descriptor are preserved (see above).
• POPM: the owner ("email to user") field is dropped by design. Multiple POPM frames collapse to one (first rating wins, last parseable play count wins); counters above u32::MAX clamp to 4 bytes.
• ID3v1 is not read. A file whose only tag is ID3v1 scans with no tags (populate the DB via beets/Picard instead). A trailing ID3v1 tag is also excluded from the audio region, so the synthesized file does not carry it.
• The audio locator validates the ID3v2 major version (2–4) and rejects synchsafe size bytes with the high bit set, producing a controlled Malformed error rather than mask-decoding an invalid offset. Tags using unsynchronisation or an extended header still scan — their declared size already covers the audio boundary.
• Scan-time tag extraction is skipped entirely — by a deliberate denial-of-service guard, see below — for tags using unsynchronisation, an extended header, non-zero frame flags (compression/encryption), malformed synchsafe size fields, or containing CHAP/CTOC chapter frames. Such files still mount and serve; they just contribute no scanned tags.
• ID3v2.2 binary frames are not extracted (3-char ids; text and art still parse). APIC width/height are not recorded at scan time.
• An APIC picture type outside the standard 0–20 range (the id3 crate's Undefined(u8) variant can exceed 20) is clamped to 0 (Other) at scan time, matching the store's track_art.picture_type CHECK.

How synthesis works

mp3::synthesize_layout (musefs-format/src/mp3.rs) emits a fresh ID3v2.4 tag followed by the untouched audio:

 offset 0
 ┌──────────────────────────────────────────────┐ ┐
 │ █ ID3v2.4 header (10 bytes)          (Inline) │ │
 │ █ text / TXXX / COMM / USLT frames   (Inline) │ │ generated
 │ █ rebuilt POPM / UFID frames         (Inline) │ │ ID3v2.4
 │ █ frame header + ▒ opaque body    (BinaryTag) │ │ tag
 │ █ APIC framing + ▒ image bytes     (ArtImage) │ │
 ├──────────────────────────────────────────────┤ ┘
 │ ░ MPEG audio incl. Xing/LAME,  (BackingAudio) │
 │ ░ verbatim                                    │
 └──────────────────────────────────────────────┘
 EOF     █ inline-generated   ▒ DB-streamed   ░ untouched backing

1. Inline — the 10-byte tag header, all text/TXXX/COMM/USLT frames, and the rebuilt POPM/UFID frames. Frame sizes are synchsafe-bounded; oversized frames fail synthesis rather than emit a corrupt tag.
2. Per picture: inline APIC framing + an ArtImage segment streaming the image bytes.
3. Per opaque binary frame: an inline frame header + a BinaryTag segment streaming the body from the DB (empty payloads are skipped — they would fail layout validation).
4. BackingAudio — the audio region located at scan time: everything after the leading ID3v2 tag and before a trailing ID3v1 tag, anchored by an MPEG frame-sync check. The Xing/LAME info frame is an MPEG frame, so it travels with the audio untouched.

Quirks & invariants

• The OOM guard (id3v2_alloc_safe): the id3 parser crate eagerly allocates a frame's declared size (v2.3 sizes are plain 32-bit — up to 4 GiB), so musefs validates every frame bound itself before handing a buffer to the crate, and refuses tags it cannot validate. Found and locked in by the mp3 fuzz target; the conservative skips listed under "Lossy edges" are this guard.
• Byte-identical audio and tag round-trip stability are asserted by musefs-format/tests/proptest_mp3.rs and the mutagen interop suite (musefs-core/tests/interop_emit.rs).

M4A

How musefs scans and synthesizes MP4-container audio (.m4a, .m4b). Only unfragmented files with exactly one audio (soun) track are accepted; chapter tracks (text, sbtl) may accompany it, and any other track — video above all — is skipped at scan time with an error naming the handler types found. For the segment model these layouts plug into, see the segment model.

What round-trips

• Canonical text tags map to their standard ilst atoms (©nam, ©ART, aART, ©alb, ©day, …) via the shared vocabulary (musefs-format/src/tagmap.rs).
• Vocabulary freeform keys (ReplayGain fields, MusicBrainz album/artist ids, ISRC, COPYRIGHT, …) round-trip through ---- freeform atoms under the com.apple.iTunes mean, matched case-insensitively.
• Other text freeform atoms round-trip keyed by their verbatim name, original casing preserved.
• Track and disc numbers, with totals: the binary trkn/disk atoms are decoded to tracknumber/discnumber as "N" or "N/M" (the "N of M" total, matching ID3 TRCK/TPOS) and rebuilt as binary atoms with the total filled in.
• Integer atoms: tmpo/cpil/pgap map to the canonical bpm/ compilation/gapless keys (shared with ID3 TBPM/TCMP and Vorbis) and are rebuilt as type-21 integer atoms.
• Multi-value atoms: every data sub-box of an atom is read (the iTunes multiple-data convention), so a multi-valued atom round-trips all its values, not just the first.
• Opaque binary freeform atoms, byte-exact: a ---- atom whose payload is binary-typed is captured verbatim under the key ----:<mean>:<name> (so the mean survives) and re-emitted streamed from the DB (BinaryTag segment).
• Cover art: every data child of a covr atom (the iTunes multiple-artwork convention) is ingested; synthesis emits one covr atom with one data child per stored art row, in order, image bytes streamed.
• Chapters, both conventions. A QuickTime chapter track — the second text/sbtl track that is the reason .m4b exists — is kept verbatim like any other structural moov child, and its chunk offsets are relocated alongside the audio track's. A Nero chapter list (moov/udta/chpl) is copied through byte-for-byte into the regenerated udta; it holds timestamps and titles, never a file offset, so relocating mdat cannot invalidate it. ffmpeg writes both by default (-movflags +disable_chpl suppresses the chpl), and players differ on which form they read, so both are preserved.

Lossy edges

• A text freeform atom under a mean other than com.apple.iTunes is re-emitted with the com.apple.iTunes mean (the scan keys text freeform by name only). Binary freeform atoms keep their mean via the ----:<mean>:<name> key.
• Binary ilst atoms outside the handled set (trkn/disk, the tmpo/cpil/pgap integer atoms, and ---- freeform) are dropped at scan time, since they are not re-emitted on synthesis.
• covr ingestion accepts only JPEG (type 13) and PNG (type 14) artwork; other type codes are skipped. MP4 has no picture-type or description fields: scanned art becomes "front cover" with an empty description, and any non-PNG stored art is emitted with the JPEG type code.
• A covr image or binary ---- value larger than its size cap is skipped at scan time — before the image is materialized out of a potentially large moov — and logged (a warn line on stderr) so the lossy drop is explained rather than silent.

How synthesis works

mp4::synthesize_layout (musefs-format/src/mp4.rs) regenerates the moov box and serves [ftyp][regenerated moov][mdat header][mdat payload]:

 offset 0
 ┌──────────────────────────────────────────────┐ ┐
 │ █ ftyp, copied verbatim              (Inline) │ │
 │ █ moov: kept structural children,    (Inline) │ │ regenerated
 │ █   every track's stco/co64 += Δ              │ │ front
 │ █ fresh udta/meta/ilst framing       (Inline) │ │
 │ █ ---- framing + ▒ freeform body  (BinaryTag) │ │
 │ █ covr framing + ▒ image bytes     (ArtImage) │ │
 │ █ chpl, copied from the old udta     (Inline) │ │
 │ █ mdat header                        (Inline) │ │
 ├──────────────────────────────────────────────┤ ┘
 │ ░ mdat payload, verbatim       (BackingAudio) │
 └──────────────────────────────────────────────┘
 EOF     █ inline-generated   ▒ DB-streamed   ░ untouched backing
         Δ = new mdat payload offset − old

1. The scan keeps moov's structural children and drops its old udta, save for a chpl chapter list, which is carried through. A fresh udta/meta/ilst is built from the DB: inline box framing, with each opaque ---- value and each cover image spliced in as streamed BinaryTag/ArtImage segments. Every enclosing box size accounts for the streamed lengths, so the spliced bytes land exactly where the sizes say.
2. The mdat payload is served verbatim (BackingAudio), merely relocated: every chunk offset in stco (32-bit) or co64 (64-bit), in every track, shifts by one constant delta. One delta suffices because a file has a single mdat, so a chapter track's chunks relocate exactly as the audio track's do. Only offset values are patched, never box sizes, so the new moov size is computable before the delta — no circular dependency. A 32-bit stco offset that would overflow fails synthesis rather than corrupt, and a track with neither stco nor co64 fails it too rather than being left unpatched.
3. A moov that sits after mdat (common for faststart-less files) is handled by a streaming reader that skips the mdat payload — the potentially hundreds-of-MB payload is never read at resolve time.

Quirks & invariants

• The structural metadata read at resolve time is capped (MAX_MP4_METADATA_BYTES, 256 MiB); a file declaring more is refused with a controlled error instead of ballooning memory.
• MP4 box sizes are 32-bit: oversized synthesized metadata (e.g. enormous art) fails with TooLarge at the format boundary rather than emitting a truncated size field.
• Byte-identical audio and structural validity are asserted by musefs-format/tests/proptest_mp4.rs, an offset-patching oracle test (mp4_oracle.rs), and the mutagen interop suite (musefs-core/tests/interop_emit.rs).

Ogg (Opus / Vorbis / FLAC-in-Ogg)

How musefs scans and synthesizes Ogg files (.ogg, .oga, .opus) carrying an Opus, Vorbis, or FLAC logical bitstream. Multiplexed and chained Ogg is detected and skipped at scan time: within the header region every page must share the first page's serial, and only the first page may carry beginning-of-stream. For the segment model these layouts plug into, see the segment model. Native FLAC files are covered by FLAC.

The Ogg invariant

Original Ogg packet payload bytes are preserved during synthesis; page sequence numbers and CRCs may be patched intentionally. Synthesis regenerates the logical bitstream's header pages (to embed fresh tags and art), which changes the header page count; the audio pages that follow are served verbatim except that each page header's sequence number is shifted by a constant delta and its CRC recomputed in place. The served audio byte length is unchanged — renumbering patches, never recopies.

Verified by musefs-format/tests/proptest_ogg.rs (crate feature fuzzing), read_at integration tests comparing source and synthesized audio payloads (musefs-core/src/reader.rs test modules), and the mutagen interop suite (musefs-core/tests/interop_emit.rs).

What round-trips

• All text tags. VorbisComments are rebuilt from the DB through the same builder as FLAC: canonical keys map to their conventional field names via the shared vocabulary (musefs-format/src/tagmap.rs); any other field round-trips verbatim by its own name, in order, multi-values included. User-defined keys outside the Vorbis field-name grammar (empty, containing =, control characters, or non-ASCII — outside ASCII 0x20–0x7D minus =) are dropped on synthesis and logged.
• Embedded pictures, with MIME type, picture type, description, and dimensions — in both art encodings (see below).
• Codec headers. The identification packet (OpusHead, Vorbis identification, the OggFLAC STREAMINFO carrier) and any trailing header packets (e.g. the Vorbis setup packet) are preserved; only the comment metadata is regenerated.

Lossy edges

• The VorbisComment vendor string is replaced with musefs's own.
• Vorbis field names are case-insensitive by spec; canonical keys come back under their conventional uppercase names and unknown field names are upper-cased on synthesis.
• Ogg carries no binary-tag slot: only text comments and pictures exist, so there is nothing else to preserve.
• Embedded pictures are parsed through FLAC's PICTURE block reader, so a picture type outside the standard 0–20 range is clamped to 0 (Other) at scan time, matching the store's track_art.picture_type CHECK.
• An embedded picture that cannot be decoded is skipped on its own, and the drop is logged (a warn line on stderr) so it is explained rather than silent. The rest of the file's pictures still ingest. Base64 decoding tolerates ASCII whitespace, so a METADATA_BLOCK_PICTURE wrapped in the older 76-column MIME style still decodes — Vorbis comment values are length-prefixed, so wrapping is unnecessary, but rejecting it cost the whole picture.
• Embedded picture descriptions are right-padded with up to two trailing spaces. The FLAC PICTURE block is built with its description padded so the prefix length — 32 + mime.len() + description.len(), i.e. everything before the image bytes — is a multiple of 3 (picture_prefix, musefs-format/src/ogg/mod.rs), which is what makes base64(prefix ++ image) == base64(prefix) ++ base64(image) and lets the image's base64 be served as an independent, incrementally-streamable substring (the art split above). Padding the description is the safe place to do it — the MIME type must stay a valid type. So a synthesized picture's description can differ from the original by up to two trailing spaces; this applies to Opus/Vorbis and OggFLAC alike, since both build the block body the same way.

How synthesis works

ogg::synthesize_layout (musefs-format/src/ogg/mod.rs) produces:

 offset 0
 ┌──────────────────────────────────────────────┐ ┐
 │ █ identification page, preserved     (Inline) │ │ regenerated
 │ █ comment page(s) rebuilt from DB    (Inline) │ │ header
 │ ▒   art windows, base64/raw    (OggArtSlice)  │ │ pages
 │ █ trailing header pages, preserved   (Inline) │ │ (repaginated)
 ├──────────────────────────────────────────────┤ ┘
 │ ░ audio pages: payload verbatim,   (OggAudio) │
 │ ░ page seq += Δ, CRC repatched in place       │
 └──────────────────────────────────────────────┘
 EOF     █ inline-generated   ▒ DB-streamed
         ░ backing pages (headers patched in place, payload untouched)
         Δ = synthesized header page count − original

1. Inline — the regenerated header pages: the preserved identification packet, a comment packet rebuilt from the DB, and the preserved trailing header packets, repaginated with correct CRCs.
2. The art split. Opus and Vorbis embed art as base64 METADATA_BLOCK_PICTURE comments (the decoded bytes are a FLAC PICTURE block body): each image is an OggArtSlice run — a window of base64(image) encoded incrementally at read time from the blob store, never materialized whole. Artwork is streamed at synthesis time: page CRCs are computed from page-bounded ArtSource windows, and the full image and its base64 copy are never materialized. FLAC-in-Ogg instead carries one native FLAC PICTURE block packet per image (raw OggArtSlice runs, no base64); the last metadata packet's last-block flag and packet 0's 16-bit following-packet count are recomputed to match. Art exceeding MAX_ART_BYTES (16 MiB − 64 KiB) is rejected by the store's CHECK, with a resolve-time cap backstopping a writer that disables check enforcement.
3. OggAudio — one compact segment covering all original audio pages, with the page-count delta to apply to every sequence number.

At read time there is no in-memory page index: the page containing a requested offset is found by a bounded backward scan (CRC-validated), then pages are walked forward with each header patched algebraically and payload bytes served by exact positioned reads. A one-page memo on the resolved file short-circuits the scan for sequential reads. A page walk that overruns the scanned audio bounds is a hard Malformed error — corrupt or misaligned data is refused, not served. Synthesized page sequence numbers wrap modulo 2³² (matching Ogg's u32 sequence field), so files whose audio pages have very high sequence numbers serve correctly rather than failing the read.

The forward page-walk reads (serve_ogg_window) flow through the shared backing read-ahead buffer (BackingReader, see backing read-ahead) just like PCM BackingAudio reads, so a sequential Ogg stream amortizes backing latency the same way. The read-ahead cache holds raw backing bytes keyed by absolute offset, so it is orthogonal to header patching: the algebraic CRC/sequence rewrite happens on the bytes after they are read, and the cache never sees a patched page. (The backward find_page_start scan and its CRC check stay on the raw fd — they are short, non-sequential probes that the forward-streaming window would not help.)

CRC patching: the linear-CRC trick

This is the neatest thing in the Ogg path. Every Ogg page carries a CRC-32 over its entire contents — header and payload, with the 4-byte CRC field treated as zero during the computation (musefs-format/src/ogg/crc.rs). Renumbering shifts every audio page's sequence number by Δ, which changes 4 header bytes (offsets 18..22). Naively, repairing the CRC means re-checksumming the whole page — including the up-to-64 KB payload that musefs has gone out of its way never to pull into memory.

It doesn't have to. The Ogg CRC uses init 0, no input/output reflection, and no final XOR, which makes it linear over GF(2): for two equal-length messages, crc32(A ⊕ B) == crc32(A) ⊕ crc32(B). Take A = the original page and B = a delta page the same length as the original but all zeros except bytes 18..22, which hold old_seq ⊕ new_seq. Then A ⊕ B is exactly the renumbered page, so:

new_crc = old_crc ⊕ crc32(DELTA)

and the payload — identical in A and A ⊕ B — cancels out entirely. The patched CRC depends only on the old CRC (already in the header) and the 4-byte sequence delta. The payload is never read.

Computing crc32(DELTA) also avoids walking the page. The 18 leading zero bytes leave the running CRC at 0 (TABLE[0] = 0, so each step is a no-op), so the computation starts directly from the 4-byte seq delta, then only has to "advance the CRC over" the trailing zeros (the rest of the header plus the whole payload length, read straight from the segment table). That advance is crc_shift_zeros — the CRC-32 of appending n zero bytes. Appending one zero byte is a fixed linear map on the 32-bit CRC state, so appending n of them is that 32×32 GF(2) matrix raised to the n-th power. The power-of-two matrices are precomputed once as a const, so a shift by n costs n.count_ones() matrix applies — O(log n) with no per-call setup, at every page size.

The net effect is that patch_page_header_algebraic (musefs-format/src/ogg/page.rs) repairs each served audio page's header from just its 27 + seg_count header bytes, in work bounded independent of payload size — and the audio payload stays untouched on disk, spliced in verbatim by positioned reads. That is what lets the Ogg invariant ("renumbering patches, never recopies") hold at serve time without a per-page in-memory index.

Quirks & invariants

• Page and header sizes are bounded at parse and serve time (MAX_OGG_PAGE_BYTES, MAX_OGG_HEADER_BYTES in musefs-core/src/ogg_index.rs); a crafted file cannot force unbounded allocation. The ogg, ogg_page, b64, and vorbiscomment fuzz targets hammer these paths.
• The incremental base64 encoder is windowed by output offset: any byte range of the encoded form can be produced from the corresponding slice of raw image bytes (musefs-format/src/ogg/b64.rs).
• The serve path's determinism does not depend on the memo: a content change rebuilds the resolved file and starts with a fresh, empty memo.

WAV

How musefs scans and synthesizes RIFF/WAVE files (.wav). WAV has no single native tag standard, so musefs writes metadata twice: a broad-compatibility LIST/INFO chunk and a full-fidelity embedded id3 chunk. For the segment model these layouts plug into, see the segment model. The ID3v2 tag inside the id3 chunk is built by the same code as MP3's — MP3's round-trip and lossy-edge rules apply to it wholesale.

What round-trips

• All text tags, via the embedded id3 chunk (full ID3v2.4, exactly as for MP3: canonical frames, TXXX extension slot, frame-id passthrough).
• The INFO subset, twice. Seven canonical keys also get a native LIST/INFO subchunk for ID3-unaware readers: title→INAM, artist→IART, album→IPRD, date→ICRD, genre→IGNR, comment→ICMT, tracknumber→ITRK.
• Binary ID3 frames and promoted tags (POPM→rating/playcount, MusicBrainz UFID→musicbrainz_trackid, opaque PRIV/GEOB/… byte-exact) — classification identical to MP3, only the chunk extraction differs.
• Embedded pictures: APIC frames inside the id3 chunk, MIME + picture type + description preserved, image bytes streamed.
• Structural chunks: fmt (required) and fact (when present) are preserved from the original front.

At scan time, tags are merged per field from both surfaces with id3 taking precedence and INFO filling gaps; only chunk headers are walked — the data payload is never read.

Lossy edges

• Non-structural chunks are dropped. The synthesized front carries only fmt , fact, the new LIST/INFO, and the new id3 chunk: cue points (cue ), broadcast-wave metadata (bext), sampler loops (smpl), and any other chunk from the original front are not reproduced.
• The INFO chunk carries only the seven-field vocabulary above; readers that understand only INFO see just those fields. Everything still rides in the id3 chunk.
• All of MP3's ID3 lossy edges apply to the id3 chunk: ID3v2.4-only output, placeholder-language COMM/USLT reset to XXX, POPM owner dropped, ID3v1 ignored, the OOM-guard skips (the authoritative list lives in MP3's lossy edges).
• Tags trailing a very large data payload are not seen. When the data payload pushes any LIST/INFO or id3 chunk beyond the scan probe ceiling (64 MiB), the file is still ingested — the data chunk header gives the audio bounds without reading the payload — but those trailing tag chunks are not read at scan time. Front-positioned metadata is unaffected.

How synthesis works

wav::synthesize_layout (musefs-format/src/wav.rs) regenerates the entire RIFF front, then serves the untouched payload:

 offset 0
 ┌──────────────────────────────────────────────┐ ┐
 │ █ RIFF/WAVE framing                  (Inline) │ │
 │ █ fmt  (+ fact), preserved           (Inline) │ │ regenerated
 │ █ LIST/INFO chunk (7-field subset)   (Inline) │ │ RIFF front
 │ █ id3  chunk: ID3v2.4 text frames    (Inline) │ │ (metadata
 │ █   frame header + ▒ opaque body  (BinaryTag) │ │  written
 │ █   APIC framing + ▒ image bytes   (ArtImage) │ │  twice)
 ├──────────────────────────────────────────────┤ ┘
 │ ░ data chunk payload, verbatim (BackingAudio) │
 └──────────────────────────────────────────────┘
 EOF     █ inline-generated   ▒ DB-streamed   ░ untouched backing

1. Inline — RIFF/WAVE framing, the preserved fmt (and fact) chunks, the rebuilt LIST/INFO chunk, and the embedded id3 chunk's text frames. Every chunk length is known up front, so the RIFF size and each chunk size field are byte-exact — no placeholder sizes.
2. Inside the id3 chunk: APIC framing inline with ArtImage segments streaming image bytes, and BinaryTag segments streaming opaque ID3 frame bodies, exactly as in MP3 synthesis.
3. BackingAudio — the original data chunk payload, served verbatim by positioned reads.

RIFF form-size enforcement

Every RIFF/WAVE file declares a form size at bytes 4..8 (riff_size). The form covers bytes 8 through 8 + riff_size and must encompass all top-level chunks (fmt , data, LIST, id3 , …). musefs enforces this at parse time:

• riff_wave_start parses the RIFF size and returns form_end = 8 + riff_size.
• locate_audio and locate_audio_at_ceiling reject any file where form_end exceeds the physical file or where the data chunk payload extends past form_end.
• Streaming or concatenated WAVs that write riff_size = 0 or 0xFFFFFFFF are rejected, but only incidentally: there is no explicit sentinel check. riff_size = 0 yields form_end = 8, which is smaller than any file carrying a data payload, and 0xFFFFFFFF yields a form_end larger than any real file — both fall foul of the bounds checks above. Detecting and honouring those sentinels explicitly is a deferred follow-up.

Quirks & invariants

• A file must have both a fmt chunk and a data chunk to scan; the declared data size must lie within the file.
• The ID3-in-WAV path inherits MP3's allocation-bomb guard (id3v2_alloc_safe): a crafted id3 chunk cannot OOM the scanner — this exact vector was found by the wav fuzz target.
• Byte-identical audio and front re-parseability are asserted by musefs-format/tests/proptest_wav.rs and the mutagen interop suite (musefs-core/tests/interop_emit.rs).

Integrations

External tools write tags and art into the musefs SQLite store; a live mount reflects their edits without copying audio. Each integration has its own page.

• beets — the musefs beets plugin
• Picard — the MusicBrainz Picard plugin
• Lidarr — Custom Script integration
• systemd — running musefs as a user/system service
• python-musefs — the shared store-contract library behind the plugins

The plugin packages have their own changelog at contrib/CHANGELOG.md.

beets-musefs

A beets plugin that syncs your beets metadata (tags + cover art) into a musefs SQLite store, so a live musefs mount shows a re-tagged view of your library without rewriting any audio.

How it fits together

• The plugin owns the tags (and cover art, when beets has it) of each track, keyed by the file's canonical real path.
• The structural columns (audio offsets, size, mtime) can only come from musefs probing the file, so the plugin runs musefs scan for you (via the bin config) before syncing — it never tries to compute those itself.
• beet musefs scans the library and then syncs; the import/write hooks scan just the touched file and then sync. musefs's auto-refresh shows changes live — no remount, and no separate scan step.

Install

pip install beets-musefs

This pulls in the shared python-musefs runtime library from PyPI automatically — both packages are published, so no working-tree install is needed.

Use via pluginpath (no package install)

The plugin itself doesn't need to be installed — point beets at the plugin's beetsplug directory and it loads at runtime. It still needs the shared python-musefs runtime library importable, so install that first:

pip install python-musefs

beets adds pluginpath entries directly to the beetsplug package path, so it must be the beetsplug dir itself (not its parent). In your beets config.yaml:

pluginpath: /path/to/musefs/contrib/beets/beetsplug
plugins: musefs
musefs:
  db: ~/musefs.db          # path to the musefs SQLite store (required)
  bin: musefs              # musefs executable for auto-scan; use a full path if
                           # not on $PATH, e.g. /path/to/musefs/target/release/musefs
  # autoscan: yes          # default; runs `musefs scan` for you. Set `no` to
  #                        # manage scanning yourself (hooks then best-effort).
  # fields:                # optional: map extra beets fields to musefs keys
  #   comments: comment

Development install (from a checkout)

To hack on the plugin or run the test suite against your working tree, install both packages editable from the repo so imports resolve to the local source:

pip install -e contrib/python-musefs       # shared library
pip install -e "contrib/beets[test]"       # plugin + test deps

Workflow (test drive)

# Sync beets metadata into the store. Auto-scans the library first (creating the
# DB if needed) — no separate `musefs scan` step.
beet musefs                      # everything
beet musefs albumartist:"Boards of Canada"   # a subset (scans just those files)
beet musefs -n                   # dry run: report counts, write nothing
beet musefs --revalidate         # prune rows whose backing file is gone

# Mount the re-tagged view.
musefs mount ~/mnt --db ~/musefs.db \
    --template '$albumartist/$album/$tracknumber - $title'

# ...or mirror your beets library layout exactly, via the computed beets_path tag.
musefs mount ~/mnt --db ~/musefs.db --template '$!{beets_path}'

Imports and tag write-backs auto-sync via event hooks: beet import and beet modify -w … record the touched items and reconcile them once the command finishes — when each file's path is final (beets has no move event, and a write fires before its move). The reconcile scans the new path and writes its tags, but it never prunes — pruning is a deliberate act (see below). A move therefore leaves the old path's row behind until you run beet musefs --revalidate. A metadata-only beet modify (no -w) doesn't fire a hook — re-run beet musefs. With autoscan: no, run musefs scan yourself first; the hooks then skip gracefully if the DB is missing.

Never writing to your backing audio files

If your backing files must stay byte-for-byte untouched — you're seeding them as a torrent, the library is immutable, or you simply want beets to drive the musefs view without ever rewriting a tag — configure beets to never write to disk:

import:
    copy: no
    move: no
    write: no

write: no is enough on its own: every stock beets plugin gates its file writes on import.write. The musefs plugin reads canonical metadata from the beets database, not from the files, and musefs scan ingests/synthesizes embedded art itself — so write: no loses nothing in the musefs view.

A few plugins ignore that gate or are redundant in this mode:

• scrub — deletes all tags from files directly via mutagen, ignoring import.write; its auto-import hook would wipe tags from your backing files. Don't enable it.
• embedart — embeds cover art into the audio files. Redundant: musefs already presents embedded art in the virtual files (scan ingestion plus the plugin's overlay of the album's artpath).
• zero — only acts during a file write, so it is inert with write: no (nothing to do, but nothing to worry about either).

Notes

• Field coverage: every tag beets writes to a file (its _media_tag_fields) is synced — ReplayGain, MusicBrainz IDs, comment, lyrics, grouping, isrc, multi-valued artists, and any custom field — under canonical musefs keys. Read-only file facts (bitrate, length, …) are never written as tags.
• Merge, not replace: beets' values win for the fields it manages; any other tag already embedded in the file is preserved in the view.
• Deletions stick: the plugin records the keys it manages per track in a musefs_managed beets flexattr (stored in the beets DB only — never in your audio files or the musefs store). Remove a tag in beets and it is removed from the view and stays gone across re-scans.
• --restore-backing (or restore_backing: yes): when you remove a tag in beets, let the file's original embedded value reappear instead of disappearing.
• Caveat: sticky deletion relies on autoscan: yes (the default), which re-derives the file's embedded tags before each sync. With autoscan: no, a deletion only takes effect after your next manual musefs scan.
• Cover art: taken from the album's artpath (beets' external cover file). beets art wins when present; otherwise any art musefs scan ingested from embedded pictures is preserved.
• Computed path (beets_path): each sync also writes a beets_path text tag holding the track's beets library-relative path (from your paths: config, via item.destination), with the file extension removed — musefs re-appends it. Mount with --template '$!{beets_path}' (the $!{} path field keeps / as directory separators) to mirror your beets layout, including layouts musefs's own template engine can't express. Set write_path: no in the musefs: config to skip it. Do not add an extension in a template that consumes beets_path. See the computed-tag workflow in the architecture overview.
• Pruning is a deliberate act. The plugin never prunes on its own. Pruning track rows whose backing file is gone from disk (renames/moves/deletes) is owned entirely by musefs revalidate --prune, reachable from beets as beet musefs --revalidate (which forwards to the pruning revalidate pass). Plain beet musefs and the passive end-of-command reconcile (beet import / beet modify -w) only sync, so a transient backing-storage loss — an unmounted network share, an offline drive, a momentary realpath divergence — can never mass-delete plugin metadata. Run beet musefs --revalidate (or musefs revalidate --prune) while the library is available to clear stale rows left by a move or an on-disk delete.
• Removals are not auto-pruned. beet remove / beet remove -d does not prune the store; run beet musefs --revalidate afterwards to drop the rows whose backing file is now gone. A bare beet remove (which keeps the file on disk) leaves a servable row in place even then — musefs can still serve those bytes.
• Orphaned art: replacing art can orphan old blobs; musefs revalidate --prune garbage-collects them.
• Schema version: the plugin refuses to run if the DB's user_version differs from the version it targets — rebuild after upgrading musefs.

Tests

The tests live under tests/ and use a local virtualenv with beets + pytest.

cd contrib/beets
uv venv                                   # create .venv (once)
source .venv/bin/activate
uv pip install -e ../python-musefs        # shared library (editable, from the working tree)
uv pip install -r requirements.txt        # beets + pytest

python -m pytest                          # unit + integration (no Rust binary)
python -m pytest -m musefs_bin            # path-matching gate vs the real `musefs` binary
python -m pytest -m e2e                   # full beets -> mount -> playback end-to-end

The musefs_bin gate shells out to the real musefs binary, so build it first from the repo root (cargo build) and run it against a fresh build. The e2e tier additionally needs ffmpeg and /dev/fuse + fusermount3: it generates audio, imports it with beets, retags, syncs, mounts via FUSE, and verifies the mount's tags and byte-identical audio (including a move-reconcile case). Both tiers are deselected from the default run and skip cleanly if their tools are absent.

musefs-picard

A MusicBrainz Picard plugin that syncs your Picard metadata (tags + front cover) into a musefs SQLite store, so a live musefs mount shows a re-tagged view of your library without rewriting any audio.

How it fits together

Picard has no way to redirect its Save to a database, so this plugin adds a context-menu action instead: match/edit as usual, then right-click your selection → "Sync to musefs" instead of pressing Save. The plugin:

1. runs musefs scan on each selected file to create/refresh its track row and structural columns (the offsets only musefs can compute), then
2. writes Picard's tags and front cover into the store, keyed by the file's canonical real path.

musefs's auto-refresh surfaces the change at the mount with no remount. The audio file is never saved by Picard.

Install (local / development)

Picard loads "folder plugins" from its plugins directory. Copy (or symlink) the musefs/ folder there:

• Linux: ~/.config/MusicBrainz/Picard/plugins/
• macOS: ~/Library/Preferences/MusicBrainz/Picard/plugins/
• Windows: %APPDATA%\MusicBrainz\Picard\plugins\

cp -r contrib/picard/musefs ~/.config/MusicBrainz/Picard/plugins/

The musefs/_common/ subfolder is the vendored python-musefs library, copied in so the plugin folder is self-contained (Picard does not install plugin dependencies). It is committed; you don't need to do anything to use it. If you change the shared library, re-run python contrib/python-musefs/vendor_to_picard.py and commit the refreshed copy — CI's drift guard enforces it.

Then enable musefs sync in Options → Plugins, and configure it in Options → musefs sync:

• musefs DB path — path to the musefs SQLite store (required).
• musefs binary — the musefs executable (PATH name or full path), used to auto-create rows. Default musefs.
• Run musefs scan before syncing — autoscan toggle (default on). With it off, run musefs scan yourself first or the sync errors on a missing DB.
• Extra field map — optional key=value list mapping additional or custom Picard tag names to musefs store keys (applied verbatim, last-wins, on top of the automatic full-tag-set sync), e.g. mymood=mood.

MUSEFS_DB and MUSEFS_BIN environment variables override the DB/binary settings (handy for testing).

Workflow

1. musefs mount ~/mnt --db ~/musefs.db --template '$albumartist/$album/$tracknumber - $title'
2. In Picard, match/cluster an album as usual.
3. Right-click the album/files → Sync to musefs.
4. Browse ~/mnt — the files show Picard's tags and cover, audio byte-identical.

Notes

• Front cover only: the first front-cover image Picard holds is synced. Picard art wins when present; otherwise any art musefs scan ingested from the file's embedded picture is preserved. Re-syncing a file with no Picard art lets the embedded picture re-seed when autoscan is on (musefs scan re-reads the file); with autoscan off, existing art is left untouched.
• Tags are fully replaced with Picard's view on every sync.
• Field coverage: every populated Picard tag is synced under its canonical musefs (on-disk) key — all MusicBrainz IDs, sort and performer/credit fields, movement, totals, and any custom field; multi-values expand and per-role performers fold to Name (Role). Picard's hidden ~ internals (length, rating, …) are never written.
• Orphaned art: replacing art can orphan old blobs; musefs revalidate --prune garbage-collects them.
• Schema version: the plugin refuses to run if the DB's user_version differs from the version it targets — rebuild the store after upgrading musefs.

Tests

cd contrib/picard
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

python -m pytest                 # unit + integration (no Picard, no Rust binary)
python -m pytest -m musefs_bin   # path-matching gate vs the real `musefs` binary

The musefs_bin gate shells out to the real musefs binary, so build it first from the repo root (cargo build). It is deselected from the default run and skips cleanly if the binary is absent.

Real-Picard (pytest-qt) tests

The adapter (musefs/__init__.py) is exercised against a real Picard + PyQt5 install, headless. Picard isn't a clean pip wheel, so use the distro package and bind a uv venv to the system Python it targets:

sudo apt-get install -y picard                              # Picard at /usr/lib/picard + system PyQt5
uv venv --system-site-packages --python "$(which python3)"  # match apt Picard's C-ext interpreter
uv pip install -e 'contrib/picard[test]'                    # test extra includes pytest-qt
PYTHONPATH=/usr/lib/picard QT_QPA_PLATFORM=offscreen \
  .venv/bin/python -m pytest contrib/picard/tests -v

These tests importorskip("picard"), so on a machine without Picard they skip cleanly and only the Qt-free _core tests run.

Manual smoke test (full GUI round-trip)

1. cargo build and create a store: musefs scan /path/to/album --db /tmp/m.db.
2. Copy the plugin into Picard's plugins dir; enable it; set DB path /tmp/m.db.
3. Load the album in Picard, change a tag (e.g. title), add a front cover.
4. Right-click → Sync to musefs; confirm the status bar / log reports synced=N.
5. musefs mount /tmp/mnt --db /tmp/m.db and verify the mounted file carries the new tag and cover, with byte-identical audio.

lidarr-musefs

A Lidarr integration that syncs Lidarr's metadata into a musefs SQLite store, so a live musefs mount shows a re-tagged view of your library without Lidarr ever copying, moving, or rewriting backing audio bytes.

Lidarr stays the downloader, matcher, and metadata source; its destination tree becomes a placeholder of symlinks that exists only so Lidarr can track files. Point Navidrome, Plex, Jellyfin, or other consumers at the musefs mount instead.

How it fits together

The package installs two console scripts that plug into Lidarr's hooks:

• musefs-lidarr-import (Import Using Script) — replaces Lidarr's own copy/move when it imports a download: it creates the destination entry as a symlink (or hardlink) to the downloaded file and fails closed — it never falls back to copying bytes.
• musefs-lidarr-sync (Custom Script notification) — fires after an import or rename: it queries Lidarr's API for the affected tracks' metadata (title, artist/albumartist, album, track/disc numbers, release date, MusicBrainz ids, genres) plus each album's cover art, runs musefs scan on the files to create/refresh their track rows (the structural columns only musefs can compute), and writes the tags and art into the store. Transient API failures (network errors, timeouts, 5xx) are retried with backoff so a blip or a Lidarr restart mid-import doesn't silently drop the sync.

musefs's auto-refresh surfaces each sync at the mount with no remount. Both scripts build on the shared python-musefs store-contract library.

Install

Install the package — with its python-musefs dependency — into the environment Lidarr uses to run custom scripts, so both scripts are on Lidarr's PATH:

pip install lidarr-musefs

This pulls in the shared python-musefs dependency from PyPI automatically. To install from a checkout instead (e.g. for development), install both editable so imports resolve to the local source:

pip install -e contrib/python-musefs
pip install -e contrib/lidarr

You also need the musefs binary reachable by the sync script (see MUSEFS_BIN below) and a musefs store/mount of your own — see the main README.

Required Lidarr settings

• Settings -> Media Management -> Import Using Script: enabled.
• Import Script Path: musefs-lidarr-import.
• Metadata Provider -> Write Audio Tags: Never.
• File Date: None.
• Linux permission management: disabled.

Do not rely on Lidarr's built-in "Use Hardlinks instead of Copy" for this workflow. Lidarr uses a hardlink-or-copy transfer mode internally, so a hardlink failure can copy bytes. musefs-lidarr-import creates the destination entry itself and fails closed.

musefs-lidarr-sync --doctor verifies these settings over the API (see Doctor).

Lidarr Custom Script

Configure a Custom Script notification (Settings -> Connect):

• On Release Import: enabled.
• On Rename: enabled.
• On Album Delete: enabled.
• On Artist Delete: enabled.
• Path: musefs-lidarr-sync.

Test events exit successfully without touching files or the database. TrackRetag events are skipped with a warning because they fire after Lidarr writes tags.

Environment

Both scripts are configured through environment variables, set in the environment Lidarr launches scripts with.

Import script:

MUSEFS_LIDARR_LINK_MODE=symlink   # default; use hardlink only if symlinks are unsuitable

Sync script:

MUSEFS_DB=/path/to/musefs.db      # the musefs SQLite store (required)
MUSEFS_BIN=musefs                 # musefs executable; full path if not on PATH
MUSEFS_LIDARR_URL=http://localhost:8686
MUSEFS_LIDARR_API_KEY=your-api-key
MUSEFS_LIDARR_AUTOSCAN=1          # default; runs `musefs scan` before each sync

API keys are redacted from logs and errors.

Manual backfill

To sync every track file Lidarr already knows about (e.g. on first setup):

musefs-lidarr-sync --all

Manual backfill requires MUSEFS_LIDARR_URL and MUSEFS_LIDARR_API_KEY. It runs the doctor preflight first (skip with --skip-lidarr-preflight), then queries all Lidarr artists and syncs their known track files into the musefs DB.

Migrating an existing Lidarr library

The forward path above (new import → import script symlink → sync) works cleanly on a fresh import. Re-homing a pre-existing Lidarr library onto the musefs symlink tree runs into several Lidarr behaviors; this is the working order (observed on Lidarr v1, lsio image). None of it is a musefs bug — these are Lidarr quirks an integrator only hits here.

1. Reassign the artists to the new (musefs) root folder.
2. Clear the stale trackfile records before re-importing. If the artists' existing trackfiles still reference the old root, re-import fails with NotParentException (/old/root/... is not a child of /new/root) — Lidarr's RemoveExistingTrackFiles chokes computing the relative path. Delete the stale trackfile records first.
   • The empty-root deletion guard: Lidarr blocks trackfile deletion while the new root folder is empty ("Artist's root folder is empty", a mass-deletion safety guard) — a chicken-and-egg with the symlinks not existing yet. Drop a placeholder file in the root until the first symlinks land, then remove it.
   • Batch the bulk delete: DELETE /api/v1/trackfile/bulk returns 500 on large batches (~200 ids); send ~25 ids per call.
3. Re-import. The import script creates the destination symlinks.
4. Backfill the store: musefs-lidarr-sync --all.

Point musefs scan at the backing directory, not the symlink tree. The default (--follow-symlinks off) is exactly right here: the store should key off the real files, while Lidarr's symlink tree is just its own tracking view.

Doctor

To verify your Lidarr settings are musefs-safe:

musefs-lidarr-sync --doctor

The doctor checks Lidarr's API for:

• writeAudioTags = no
• fileDate = none
• setPermissionsLinux = false

If MUSEFS_LIDARR_URL and MUSEFS_LIDARR_API_KEY are not configured, doctor and sync fail because the integration cannot verify safe settings or build complete per-track metadata.

--doctor is a runtime / post-deploy check, not an offline one: it makes a live Lidarr API call, so it needs MUSEFS_LIDARR_URL + MUSEFS_LIDARR_API_KEY and a reachable Lidarr instance. Run it after deployment, not at container build time — offline it fails with connection-refused even when the toolchain itself is wired up correctly. There is no offline "are the binary and plugins installed/wired" check; to confirm installation at build time, test that the musefs-lidarr-import / musefs-lidarr-sync scripts and the musefs binary are importable/on PATH.

Smoke test

1. Build and install musefs.
2. Install python-musefs and lidarr-musefs into the environment Lidarr uses for custom scripts.
3. Configure Import Using Script and Custom Script as described above.
4. Import a small album.
5. Confirm Lidarr's destination entry is a symlink by default.
6. Run musefs mount /tmp/mnt --db "$MUSEFS_DB".
7. Confirm the mount shows Lidarr metadata.
8. Confirm the source file's bytes and mtime did not change.

Notes

• Tags are fully replaced with Lidarr's view on every sync (scanner-written binary tags always survive — see the external-writer contract).
• Cover art: each album's Lidarr cover is fetched and written as the front cover, replacing the track's art rows on every sync (an over-cap or unreachable cover is skipped, leaving any scanner-ingested art in place).
• Schema version: the sync refuses to run if the DB's user_version differs from the version it targets — rebuild the store after upgrading musefs.
• Deletions prune by MusicBrainz id, scoped to rows this plugin owns. On an Album/Artist delete, the sync removes the matching store rows (musicbrainz_albumid / musicbrainz_artistid) so the mount stops presenting them. The backing audio is never touched — pruning only drops the store rows, not the files Lidarr keeps in the backing directory. A delete event for a release with no MusicBrainz id cannot be mapped and is logged and skipped.
• Ownership marker. Every track the sync writes is stamped with a musefs_lidarr_managed=1 tag, and a delete only removes rows carrying that marker. Without it, a musicbrainz_albumid the scanner seeded from a file's own native tags is indistinguishable from one Lidarr wrote, so an unrelated Lidarr delete could drop an unmanaged track's metadata. The marker is a normal text tag, so it does appear in served files (e.g. as a MUSEFS_LIDARR_MANAGED Vorbis comment / a TXXX frame / an iTunes freeform atom). A track imported under an older plugin version (before the marker existed) is treated as unmanaged and is left in place on delete — re-sync it to stamp the marker.
• CI coverage: a fast smoke (real Lidarr exec path + mocked API) gates PRs, and a full real-instance download-client import e2e gates the Python releases — see the Python plugins guide.

Tests

cd contrib/lidarr
python -m venv .venv && source .venv/bin/activate
pip install -e ../python-musefs    # shared library (editable, from the working tree)
pip install -e ".[test]"

python -m pytest                   # unit + integration (no Rust binary)
python -m pytest -m musefs_bin     # path-matching gate vs the real `musefs` binary

The musefs_bin gate shells out to the real musefs binary, so build it first from the repo root (cargo build). It is deselected from the default run and skips cleanly if the binary is absent.

Running musefs as a systemd user service

These units run musefs on the host (the recommended deployment) under your own user account — no root, no CAP_SYS_ADMIN.

Files

• musefs.service — the mount daemon (musefs mount); blocks until stopped.
• musefs-scan.service + musefs-scan.timer — optional periodic musefs revalidate --prune.
• musefs.conf.example — every MUSEFS_* setting, commented with defaults.

Install

mkdir -p ~/.config/systemd/user ~/.config/musefs
cp musefs.service musefs-scan.service musefs-scan.timer ~/.config/systemd/user/
cp musefs.conf.example ~/.config/musefs/musefs.conf
$EDITOR ~/.config/musefs/musefs.conf   # set MUSEFS_MOUNTPOINT and MUSEFS_DB
systemctl --user daemon-reload
systemctl --user enable --now musefs.service

Enable the periodic re-scan too (edit the library path in musefs-scan.service first):

systemctl --user enable --now musefs-scan.timer

Hardening

These units run under the --user manager, which constrains what systemd sandboxing is possible. The two units differ sharply:

• musefs-scan.service is fully sandboxed. The scanner creates no FUSE mount, so it takes a strong sandbox (ProtectSystem=true, SystemCallFilter=, plus namespace and seccomp restrictions). ProtectSystem=true (not strict) keeps system directories read-only while leaving your library and MUSEFS_DB writable, so a custom DB location needs no ReadWritePaths= edit. A few directives that require capability-bounding-set drops (CapabilityBoundingSet=, PrivateDevices, ProtectKernelModules, ProtectKernelLogs, ProtectClock) are omitted: the unprivileged user manager cannot apply them, and the process is already capability-less, so nothing is lost. Inspect with systemd-analyze --user security musefs-scan.service.

• musefs.service is intentionally not sandboxed, and cannot be. musefs mounts via the setuid fusermount3 helper. NoNewPrivileges=true — and nearly every other systemd hardening directive, since installing a seccomp filter for an unprivileged process forces the kernel no_new_privs flag — disables the setuid escalation, and the mount then fails with fusermount3: mount failed: Operation not permitted. The unit comment explains this in full.

Notes

• The store must exist before the mount starts. musefs mount never creates the DB — it requires a populated store and exits non-zero otherwise, so a mount unit that starts before anything has scanned hard-fails and (with Restart=on-failure) crash-loops. Seed the store with an initial musefs scan before enable --now musefs.service. If you generate the store from another unit, order this one after it with a drop-in (systemctl --user edit musefs):

  [Unit]
  After=musefs-initial-scan.service
  Requires=musefs-initial-scan.service
  

  (The musefs-scan.timer is a periodic re-scan, not the initial seed.)

• Binary location. The --user manager does not inherit your shell's PATH. The units set PATH for a cargo install binary in ~/.cargo/bin; if musefs is elsewhere, edit the Environment=PATH= line (or make ExecStart an absolute path).

• %h vs ~. Unit files expand %h to your home directory; the musefs.conf EnvironmentFile does not expand %h or ~ — use absolute paths there, and never paste ~/... into a unit directive (it is taken literally).

• Settings. musefs.conf.example is a commented example of the common MUSEFS_* mount/scan variables (every scalar mount/scan flag has a MUSEFS_* form — uppercase the long flag, dashes to underscores). Explicit flags override env vars; --fallback and scan targets are command-line only (set them in ExecStart).

• Inline overrides. Prefer systemctl --user edit musefs to add Environment= lines in a drop-in; it survives reinstalls.

• Headless servers. A --user timer only fires while your user manager runs. For a daily scan when you are not logged in: loginctl enable-linger <user>.

• Logs. journalctl --user -u musefs -f. The units log at warn by default; raise it with Environment=RUST_LOG=info in a drop-in or in musefs.conf — see Logging & troubleshooting.

python-musefs

The shared store-contract library behind the beets, Picard, and Lidarr musefs plugins. It is the single source of truth for how a plugin writes the musefs SQLite store: the schema-version check, the tags / art / track_art writes, sha256 art content-addressing, the realpath_key path normalization, the musefs scan shell-out (run_scan), and the per-file sync write-loop (Record / sync_files).

Field mapping stays in each plugin — beets expands multi-valued genres/composers into one tag each, Picard takes the first value — so this library deliberately does not own it.

Writing a plugin

A plugin turns host metadata (a beets item, a Picard track, a Lidarr release) into musefs store writes. This library owns every store-touching step except the field mapping: you supply the per-file tag and art values, and it handles the schema check, the scan shell-out, content-addressing, and the write loop.

The write flow

The canonical order is connect → check_schema_version → run_scan → build Records → sync_files → commit → prune_missing. The caller owns the transaction — nothing here commits for you.

from musefs_common import (
    SCAN_TIMEOUT_SECONDS,
    ArtImage,
    Record,
    check_schema_version,
    connect,
    prune_missing,
    realpath_key,
    run_scan,
    sync_files,
)


def sync(db_path, files, *, musefs_bin="musefs"):
    # `run_scan` creates the DB if absent and fills the structural columns a
    # plugin cannot compute (format, audio offset/length, backing size/mtime).
    # On a brand-new store it must precede `connect`, which has nothing to open
    # until the scan has created the file.
    result = run_scan(musefs_bin, db_path, files, timeout=SCAN_TIMEOUT_SECONDS)
    if result.partial:
        # Exit 2: the batch committed, but some file could not be ingested.
        # Warn and sync the rest — don't abort over one unparseable file.
        print(result.warning())

    conn = connect(db_path)
    try:
        check_schema_version(conn)  # raises SchemaMismatch on a version skew

        records = [
            Record(
                key=realpath_key(path),  # MUST equal the scanned row's backing_path
                pairs=[("artist", artist), ("title", title)],
                art=[ArtImage(data=cover, mime="image/jpeg")] if cover else None,
            )
            for path, artist, title, cover in host_metadata(files)
        ]

        stats = sync_files(conn, records)  # full-replace of plugin text tags
        conn.commit()  # the caller commits

        prune_missing(conn)  # drop rows whose backing file vanished
        conn.commit()
        return stats
    finally:
        conn.close()

For a dry run, pass dry_run=True to sync_files and conn.rollback() instead of committing — SyncStats still reports what would change.

run_scan reads the scanner's three-state exit code and returns a ScanResult: 0 is a clean run, and 2 is a partial success — the batch completed and committed, but at least one file could not be ingested (see Scanning). A partial run is not a failure, so it sets result.partial rather than raising; result.warning() renders the one-line, non-fatal message an adapter should surface before going on to sync the files that did land. Any other exit code is a hard failure and raises ScanError (kind ∈ {"not_found", "timeout", "failed"}), as do a missing binary and a timeout. check_schema_version raises SchemaMismatch; a host adapter formats its own user-facing message from the exception attributes (see the beets plugin's _scan_user_error).

The Record shape

One Record per file is your primary output. Its fields:

ArtImage(data, mime, picture_type=3, description="") is one picture: data is raw bytes, picture_type is the ID3/FLAC type (3 = front cover). Images larger than MAX_ART_BYTES are dropped and counted in SyncStats.skipped_art.

If every record lands in skipped, the keys and the scan target disagree — both must canonicalize the same way, so scan the real files (not a symlink farm) and build keys with realpath_key.

Merge vs. replace, and sticky deletes

sync_files(..., merge=False) (the default) replaces every plugin-owned text tag on each track: it clears all value_blob IS NULL rows and rewrites them from record.pairs. Scanner-written binary tags always survive.

sync_files(..., merge=True) merges: only the keys named in record.pairs and record.delete_keys are touched; other scan-seeded text tags stay. Use merge when your plugin owns a subset of the tags and must not clobber the rest. The store does not remember which keys you manage — you track your managed-key set out of band (the contract is explicit that the store is not the place for plugin state).

Merge-mode key matching is case-insensitive (lower(key) = lower(?)): Vorbis keys render case-insensitively, so a scan that seeds a tag in the file's native case (e.g. LABEL) is correctly replaced when your plugin canonicalizes to lowercase (label), rather than leaving the original row behind as a duplicate.

When the user removes a tag in the host, merge mode needs to delete the now-orphaned store row. The beets plugin solves this with an accumulating managed-key set (the musefs_managed pattern), worth copying:

• Persist, per file, the set of keys you have ever written (beets uses a flexattr; any per-file host metadata works).
• On each sync, delete_keys = previous_managed − keys_written_now, and the new persisted set is previous_managed ∪ keys_written_now.
• A key you stop writing becomes a tombstone: it keeps getting deleted on every sync until you write it again. Persist the managed set only after the store commit succeeds, so a failed sync doesn't lose the record of what you owe.

See contrib/beets/beetsplug/_core.py (build_records / persist_managed) for the reference implementation.

Store invariants you must respect

The full external-writer contract is in the external-writer contract. The rules that bite plugin authors:

• Write only tags, art, and track_art. The scanner owns the structural columns of tracks and all of structural_blocks; never compute them — run musefs scan (i.e. run_scan). CHECK constraints reject malformed structural shapes at commit, so you cannot persist them anyway.
• Binary tags survive a sync. merge_tags / replace_tags scope their deletes to text rows (value_blob IS NULL), so the write loop never wipes scanner-written binary tags. You may write binary tags yourself too — a binary row carries its payload in value_blob and must leave value empty (the only CHECK on the row).
• Content-address art through upsert_art (sha256 de-dup) rather than inserting art rows by hand; sync_files does this for you.
• Art rows are immutable. A trigger rejects in-place updates of an art row's content columns (data, sha256, mime, byte_len, width, height). To change a track's art, insert a new content-addressed row via upsert_art and relink it via replace_track_art.
• Path layout is just a tag. To drive a reorganized mount, write your computed relative path into a custom tag (e.g. beets_path) and mount with --template '$!{beets_path}'. musefs sanitizes each path segment, so a writer cannot inject traversal.

API reference

Everything in __all__, imported from the top-level musefs_common package.

Connection & schema

• connect(db_path) → sqlite3.Connection — open with a 5s busy timeout and foreign_keys = ON.
• check_schema_version(conn) — raise SchemaMismatch unless the store's user_version equals EXPECTED_USER_VERSION.

Scanning

• run_scan(binary, db_path, target, *, timeout=None) → ScanResult — shell out to musefs scan; target is one path or an iterable, all scanned under one process. Creates the DB if absent. Raises ScanError on a hard failure.
• ScanResult(binary, target, verb, returncode, partial, stderr) — a completed run. partial marks the exit-2 partial success (the batch committed; some file failed to ingest) and warning() renders its non-fatal message.

Building records

• Record(key, pairs=[], art=None, delete_keys=None) — one file's sync inputs (see The Record shape).
• ArtImage(data, mime, picture_type=3, description="") — one embedded picture.
• realpath_key(path) — canonical path string matching the scanner's backing_path; accepts str/bytes, returns str.

Writing

• sync_files(conn, records, *, dry_run=False, stats=None, merge=False) → SyncStats — the write loop; caller owns the transaction. Pass stats to accumulate into a caller-seeded instance.
• sync_one(conn, record, stats, *, dry_run=False, merge=False) — sync a single record into a caller-supplied SyncStats.
• SyncStats — synced / skipped / art_linked / skipped_art / skipped_invalid counters, plus .summary(). A record whose tags or art violate a store CHECK constraint is rolled back and skipped (not raised), bumping skipped_invalid and appending (record.key, message) to the invalid list — one malformed record never aborts the batch.

Lower-level store helpers (called for you by sync_files; use directly only for a custom write loop)

• track_id_for_path(conn, key) → track id or None.
• merge_tags(conn, track_id, managed_pairs, delete_keys) — per-key replace of plugin-managed text tags, leaving unmanaged text rows intact.
• replace_tags(conn, track_id, pairs) — replace all plugin-owned text tags.
• upsert_art(conn, data, mime) → art id — content-address data by sha256, inserting only if new.
• replace_track_art(conn, track_id, arts) — replace a track's track_art rows; arts is [(art_id, picture_type, description), …].
• sniff_mime(data, path) — image mime from magic bytes, falling back to file extension.
• prune_missing(conn, track_ids=None, *, unreadable=None) → count — delete tracks whose backing file is confirmed gone (every track, or just track_ids). Only a FileNotFoundError from os.stat counts as gone: a path that cannot be stat'd for any other reason (a parent directory's permissions, an unreachable network or removable mount) keeps its row, because the cascade would take that track's plugin-written tags and art with it. Pass a list as unreadable to collect (track_id, backing_path, message) for every row kept that way, so a pass that pruned nothing can be told from one that could not look.
• delete_tracks(conn, track_ids) → count — unconditionally delete the given track rows (intent-based, unlike prune_missing's on-disk existence check); their tags and track_art rows cascade away.

Reading

• track_ids_for_paths(conn, keys) → {key: id} — bulk backing_path → track id; keys with no matching row are omitted. Chunked under SQLite's parameter cap, so arbitrarily large lookups are safe (the bulk track_id_for_path).
• track_ids_by_tag(conn, key, value) → [id, …] — track ids whose plugin-owned text tag (key, value) matches (scanner-written binary tags never match); maps a source's "I deleted this album/artist" signal back to the rows it tagged.
• tags_for_track(conn, track_id) → [TagRow, …] ordered by key then ordinal, covering both plugin-owned text tags and scanner-written binary tags.
• TagRow(key, value, value_blob) — one read-back tag row. Text tags have value_blob is None; binary tags have value == "" and value_blob bytes.

Constants

• EXPECTED_USER_VERSION — schema user_version this library targets.
• MAX_ART_BYTES — per-image art cap; larger images are skipped.
• SCAN_TIMEOUT_SECONDS — default wall-clock cap for one run_scan.

Exceptions

• SchemaMismatch(found) — schema-version skew; .found is the DB's version. The message names which side is behind and the fix: a store newer than the plugin means upgrading the plugin, an older one means rescanning with musefs to migrate it.
• ScanError(kind, *, binary, target, …) — a musefs scan failure; .kind ∈ {"not_found", "timeout", "failed"}, with context attributes for messaging.

Consumers

• beets depends on this package via pip (contrib/beets/pyproject.toml).

• Picard cannot pip-install plugin dependencies, so the package is vendored into contrib/picard/musefs/_common/ by vendor_to_picard.py. After any change here, re-run:

  python contrib/python-musefs/vendor_to_picard.py
  

  The Picard test tests/test_vendor_sync.py fails if the committed copy drifts.

• Lidarr depends on this package via pip (contrib/lidarr/pyproject.toml).

Schema coupling

musefs_common/schema.py (SCHEMA_SQL, USER_VERSION) is generated from the Rust migrations in musefs-db/src/schema.rs — do not edit it by hand. EXPECTED_USER_VERSION (in constants.py) derives from it. When the Rust schema bumps, regenerate and re-vendor:

MUSEFS_REGEN_SCHEMA_PY=1 cargo test -p musefs-db schema_py
python contrib/python-musefs/vendor_to_picard.py

A musefs-db unit test fails if the generated file drifts. This is all independent of the package's own __version__ (its release SemVer).

Tests

cd contrib/python-musefs
python -m venv .venv && source .venv/bin/activate
pip install -e ".[test]"
python -m pytest -v
ruff check . && ruff format --check .

Architecture overview

This is the technical reference for musefs internals: how a virtual file is assembled, how the workspace is layered, what the SQLite store guarantees, and how external edits become visible without a remount. For usage, see the User Guide; for the development workflow, see Contributing; for per-format behavior, see the format docs.

Design overview

musefs is a read-only passthrough FUSE filesystem with one cardinal invariant: original audio bytes are never copied or modified. A served file is not a transcoded or rewritten copy — it is assembled on the fly by splicing a freshly generated metadata region in front of positioned reads of the untouched backing file. The SQLite store is the source of truth for tags, art, and each file's audio byte range; the backing directory is the source of truth for the audio itself.

Crate layout

A strict layered Cargo workspace; dependencies point one way only:

musefs-db   ─┐                 SQLite store: schema/migrations, tracks/tags/art access
musefs-format┘← (db)           format byte-surgery: metadata synthesis + RegionLayout
        ↑
musefs-core ← (db, format)     orchestration: virtual tree, resolution, scanning, refresh
        ↑
musefs-fuse ← (core)           thin FUSE adapter (fuser)
        ↑
musefs-cli  ← (core, fuse, db) clap commands library (scan/mount logic)
musefs      ← (cli)            thin binary entrypoint; published as `musefs`

musefs-core is the integration layer — cross-cutting logic belongs there. musefs-fuse, musefs-cli, and the musefs binary crate are deliberately thin; the FUSE adapter's job is translating kernel requests into core calls (and dispatching blocking reads onto a worker pool with per-thread reusable buffers, so a slow backing read never stalls the FUSE dispatch thread).

The workspace also carries musefs-latencyfs, a dev/bench-only crate (publish = false): a latency-injecting passthrough FUSE filesystem used by the benchmarks harness to simulate slow backing stores. It is not part of the shipping dependency graph (core uses it only as a dev-dependency).

The serving model

The segment model

A synthesized virtual file is described by a RegionLayout (musefs-format/src/layout.rs): an ordered list of Segments whose lengths sum to the served file size. Six variants:

• Inline(Vec<u8>) — generated framing/text bytes (an ID3v2 tag, FLAC metadata blocks, a RIFF front), fully materialized at resolve time.
• ArtImage { art_id, len } — embedded cover art; only the length lives in the layout. Image bytes stream from the DB blob in chunks at read time and are never buffered whole. This invariant also holds for Ogg synthesis, where page CRCs are computed from page-bounded ArtSource windows (previously the documented exception).
• BackingAudio { offset, len } — a run of the original file's audio frames, served by positioned reads (read_exact_at) against the backing file.
• OggAudio { offset, len, seq_delta } — original Ogg audio pages served with each page's sequence number shifted by seq_delta and its CRC recomputed in place (a resized header changes the page count). The byte length is unchanged — renumbering patches, never recopies.
• OggArtSlice { art_id, offset, len, base64, art_total } — a window of an embedded picture served lazily from the blob store; when base64, the window is base64-encoded incrementally at read time.
• BinaryTag { payload_id, len } — an opaque binary tag payload (e.g. an ID3 PRIV frame body or a FLAC APPLICATION block body) streamed from the DB at read time.

read_at (musefs-core/src/reader.rs) serves a byte range by walking the segments and splicing: inline bytes are copied, art and binary-tag payloads are read from the DB in chunks, backing audio comes from positioned reads of the original file, and Ogg pages are renumbered and CRC-patched in flight. This is how the cardinal invariant holds end to end. Layouts that stream any payload from the DB by rowid — binary tags and art (ArtImage / OggArtSlice) — are flagged (RegionLayout::streams_db_rowid) so the reader wraps those reads in a single WAL snapshot with a content_version recheck. A concurrent retag (delete + reinsert reusing a freed rowid) cannot interleave bytes from two generations of a tag or splice the wrong image. Both the per-handle fast path and the stateless no-fh fallback apply the guard, and the fallback re-validates its freshly opened backing fd against the resolved stamp.

Backing read-ahead

Every backing read — BackingAudio splices and the serve_ogg_window page walk alike — flows through a single BackingReader::read_exact_at (musefs-core/src/readahead.rs). It caches raw backing-file bytes keyed by absolute backing offset in a per-handle adaptive window: a sequential miss reads one large pread (geometric growth up to a per-stream cap) instead of the ≤256 KiB FUSE chunk, so a high-latency backing client (NFS, remote) can pipeline the RPCs behind one syscall; a seek resets the window to the floor. All handles draw from one process-wide RAM budget (--read-ahead-budget-mib, default 64) with deadlock-free try_lock LRU eviction. Keying on the absolute backing offset (not the synthesized output) makes the cache retag-immune, and serving still flows through the per-read validate_opened_backing re-stat, so the cardinal audio-bytes invariant and freshness semantics are untouched. An optional Phase-2 background-prefetch layer (--read-ahead-prefetch) exists and is off by default: amplification alone carries the win on local and low-latency backing, while the threads add a measured ~30 % on top of it only once per-read latency is high enough for the overlap to hide something (see the backing read-ahead benchmarks).

How each format builds its layout differs enough to warrant its own document: FLAC, MP3, M4A, Ogg, WAV.

Mount modes

musefs_core::Mode selects one of two behaviors at mount time:

• Synthesis (default) — the metadata region is generated from the DB and spliced ahead of the backing audio, as above. Resolve-time validation guards the stored audio bounds: if audio_offset + audio_length runs past the backing file's current length, the row no longer matches the file and the resolve fails with a controlled BackingChanged error.
• StructureOnly — pure passthrough: the layout is a single whole-file BackingAudio segment, so the original bytes are served verbatim under the templated tree. Stored audio bounds are irrelevant (the whole file is served) and are not validated in this mode.

In StructureOnly mode, on kernels with FUSE passthrough (6.9+) and a daemon holding CAP_SYS_ADMIN (kernel-gated: run as root or setcap cap_sys_admin=ep the binary), each open registers the backing fd with the kernel and reads bypass the daemon entirely. The capability check is performed at mount time and its absence pre-announced; if registration fails at runtime anyway, passthrough is disabled for the rest of the session (later opens skip the doomed ioctl) and reads fall back to the daemon silently. Freshness for a passthrough handle is open-time-only — it is a plain POSIX fd onto the backing file. In Synthesis mode no single fd represents the spliced bytes, so passthrough never applies.

Directory listings

opendir builds a directory's entries once and holds them, so a paginated readdir is a map lookup rather than a fresh tree walk per call — enumeration stays O(n) in the directory, not O(n²). Handles are capped at 1024 concurrent so a client that opens directories without closing them cannot pin unbounded memory.

Handles do not each pay for their own copy. A listing is keyed by the directory and the virtual-tree generation it was built from, and every handle that agrees on both shares it: a thousand opens of one directory cost one listing and a thousand refcounts, not a thousand copies of a listing whose size scales with the directory's width. All but the first also skip the tree walk outright, which is what an over-cap readdir consults before rebuilding. opendir pins the generation it read, so the address that identifies it cannot be reused while any handle still names it, and a refresh simply means the next opendir builds against the new generation while open handles keep serving the view they were opened on. musefs_dir_listings is the distinct-listing count behind musefs_dir_handles: the gap between them is the sharing, and equality means every open handle is on a different directory.

Over that cap, opendir degrades rather than failing: it returns the stateless handle, and readdir falls back to rebuilding the listing on each call (on the worker pool, like every other blocking operation, not on the single fuser dispatch thread). Listings stay complete — parallel walkers routinely exceed 1024 concurrent directory handles on a large mount — at the cost of that rebuild. musefs_dir_handle_rejections_total counts the opens that took the fallback; the musefs_dir_handles gauge cannot show this, because saturation is bursty enough to read healthy in every sample while thousands of opens are degraded between them.

readdirplus answers the same listing with each entry's attributes inline, so a client that stats what it lists — ls -l, and every media scanner — spends one round trip on the directory rather than one more per entry. The mount requests FUSE_READDIRPLUS_AUTO alongside it, which lets the kernel fall back to plain readdir when the caller is not statting: an attribute-laden entry is several times the size of a bare one, so fewer fit in a reply page, and for a bare ls that is a pessimization rather than a saving.

Directories cost nothing to answer — getattr returns immediately for one without touching the store — and the synthetic entries have static attributes, so only the file entries need work. Those fan out across the worker pool, in rounds of 64, and the reply is assembled by whichever resolution finishes last. The fan-out is the point: concurrent lookups already spread across the pool, so resolving a page serially would be slower for a threaded scanner than the round trips it removes. It is also why nothing here waits — a worker blocking on tasks it queued to its own bounded pool could deadlock behind them, so a round is a countdown rather than a join. Rounds exist because the reply buffer's size is not visible to the handler: resolving a whole wide directory to fill one page would front-load enormous latency onto the first call, and anything over-resolved lands in the size cache for the page that does ask for it.

An entry whose attributes cannot be resolved is still listed, with placeholder attributes and a zero TTL: the kernel caches neither, so the client's next access goes back to lookup and gets the real error — the same thing it sees today, rather than the file silently vanishing from the listing. musefs_readdirplus_total counts the calls; zero means the kernel is not using the op, which is otherwise invisible from the daemon since the capability is negotiated at mount.

Synthetic telemetry namespace

When --expose-metrics is on, the root directory gains a synthetic .musefs-metrics/ entry backed by reserved inodes at u64::MAX - 1 (dir) and u64::MAX - 2 (file) — the same "top of the u64 space" trick the Spotlight marker uses, since InodeAllocator starts at 2 and only increments. The directory and file are disjoint from the macOS Spotlight marker at u64::MAX.

The metrics file is /proc-style: it advertises st_size == 0 and is served via FOPEN_DIRECT_IO, so readers must read to EOF rather than trusting the stated size. Content is rendered at open time from a snapshot of CoreTelemetry (header/size caches, read-ahead budget/charge, virtual-tree footprint, refresh health, suppressed serve-path warnings), FuseTelemetry (uptime, read/dir-handle gates, worker pool, passthrough state), and optional jemalloc/syscall counters (including read-ahead hit/miss) — see musefs-core/src/telemetry.rs for the full metric list. This namespace deliberately bypasses the virtual tree (VirtualTree) and the RegionLayout / segment model: it is injected into root-directory readdir and resolved by direct inode checks, so the cardinal audio path is untouched.

Injecting into readdir while intercepting lookup is only coherent because the tree cannot supply a root child of the same name: .musefs-metrics and the Spotlight marker are reserved in the virtual-tree namespace, and a track that renders to one is ranked to (2) at build time — see the virtual tree (#681). The injected entry therefore needs no dedup, and the two surfaces cannot disagree.

The store & external-writer contract

The SQLite store

musefs-db/src/schema.rs defines the schema as an ordered list of migrations (MIGRATIONS: the MIGRATION_V1 baseline, MIGRATION_V2, which adds the scanner-owned fingerprint/content_hash columns, and MIGRATION_V3, which widens the tags.value and track_art.description caps); user_version records the schema version (3). The store is the interface external tools write to — the beets and Picard plugins under contrib/ write tags and art here out-of-band.

• The baseline schema (MIGRATION_V1): the core tables — tracks (one row per backing file: path, format, audio byte range, size/nanosecond-mtime/ctime stamps, content_version), tags (multi-value key/value rows ordered by ordinal, with an optional value_blob for binary tags), art (content-addressed, deduplicated image blobs), track_art (per-track art links with picture type and ordering), and structural_blocks (read-only, derived-from-file FLAC STREAMINFO/SEEKTABLE metadata, not part of the editable contract). Deleting a track cascades to its tags and track_art rows. Triggers bump the owning track's content_version/updated_at on any tags/track_art edit; CHECK constraints enforce the contract invariants below at commit time. A bounded, self-pruning track_changes ring (capacity 8192, CHANGELOG_CAP) fed by triggers on tracks gives O(changed) refresh — every metadata edit funnels through an UPDATE on the tracks row, relying on SQLite's nested trigger activation (on by default). Freshness-superset triggers make content_version cover every DB-knowable input to synthesized bytes: art_reject_content_update (art is content-addressed and immutable), art_ad (a deleted art row bumps referencing tracks so an orphan rebuilds to a clean serve-time error), tracks_geometry_au (scanner-owned geometry changes), and structural_blocks_ai/_ad.

The external-writer contract

Ownership. External tools get full read/write on tags, art, and track_art. The scanner owns the structural columns of tracks (id, backing_path, format, audio_offset, audio_length, backing_size, backing_mtime_ns, backing_ctime_ns, content_version, updated_at) and all of structural_blocks: those are derived from probing the file, and external tools must run musefs scan rather than compute them.

tracks.fingerprint and tracks.content_hash are also scanner-owned, read-only-derived columns — like structural_blocks, they are never part of the editable tag contract and external tools never write them. fingerprint is a SHA-256 over the probe's parsed output (deterministic per file, excludes filesystem stamps such as mtime/ctime), computed in the parallel probe worker at zero extra I/O. content_hash is a full-file SHA-256, stored as 64-char hex; it is computed only at the full checksum tier (--checksum=full), which requires an eager whole-file read. Neither column is UNIQUE by design — duplicate-content tracks legitimately share both values. On a normal scan, when a probed file's path is not yet in the store and its fingerprint matches exactly one orphaned row (a row whose backing_path no longer exists on disk), the scanner retargets that row to the new path in place, preserving its id, tags, and art rather than orphaning them. This is how musefs recovers from a backing-library move or reorganization: run musefs scan after moving files, and existing store rows follow their backing files to the new locations.

What the store enforces. SQLite CHECK constraints reject the malformed shapes at commit, so an external writer cannot persist them:

• an unknown format string, or a negative length/offset/size/version;
• an audio_offset + audio_length running past the stored backing_size;
• a binary tag row whose value is non-empty;
• an art.byte_len that disagrees with its blob, or a sha256 of the wrong length;
• a picture_type outside 0..=20;
• a tags.key over 256 chars or tags.value over 16 MiB − 1 bytes (FLAC's 24-bit metadata-block ceiling — the largest tag synthesis could serve, so the store never refuses a tag the format could carry);
• tags.key must be non-empty and contain no ASCII control characters (a DB CHECK enforces this, rejecting violating writes — with one blind spot: an embedded NUL terminates SQLite's length()/GLOB, so a key like a\0b slips the CHECK. The scanner's own floor drops it before insert, and the Vorbis path rejects it on synthesis). Additionally, only keys within the Vorbis field-name grammar (ASCII 0x20–0x7D, excluding =) survive FLAC/Ogg synthesis — others are dropped and logged. MP3/M4A custom keys may use the wider set (e.g. =, :, spaces, non-ASCII).
• a value_blob over MAX_BINARY_TAG_BYTES;
• an art.mime over 255 chars or byte_len over MAX_ART_BYTES;
• a track_art.description over 8 KiB;
• a structural_blocks row with an unknown kind, negative ordinal, or body over the FLAC 24-bit block limit.

One ordinal space per key. tags' primary key is (track_id, key, ordinal), which does not discriminate on value_blob: a track's text rows and its binary rows are numbered in the same space per key. A writer that holds a key in both classes must not restart at 0 for the binary rows, or the insert fails with UNIQUE constraint failed: tags.track_id, tags.key, tags.ordinal. The scanner numbers text rows first and continues the same counters for the binary rows (#659), so a track's binary rows for a key begin above however many text values the scan seeded under it.

The rule this leaves for an external writer: a rewrite of the text rows alone — which is what musefs_common.store's replace_tags / merge_tags do, scoping their DELETE to value_blob IS NULL so scanner-written payloads survive a sync — must not grow a key past the lowest ordinal its binary rows already hold. In practice the two key namespaces barely meet: binary keys are APPLICATION / CUESHEET (FLAC), uppercase four-character ID3 frame ids such as PRIV, GEOB, MCDI, SYLT, UFID (MP3/WAV), or ----:<mean>:<name> (MP4, while the text path keys the same atom on its bare name). The primary key compares byte-exactly under the default BINARY collation, so a lowercase cuesheet row can never collide with the FLAC block's CUESHEET row, and the beets plugin — which lowercases every key it emits — cannot produce a colliding row at all.

Case-folding cuts the other way for the delete half, and the difference is worth holding onto: merge_tags clears by lower(key) = lower(?), so that same lowercase cuesheet does remove the scan-seeded CUESHEET text row (#407 — Vorbis keys render case-insensitively, and an exact-case delete would leave the scan row behind as a visible duplicate). The binary row is untouched, being scoped out by value_blob IS NULL, and keeps whatever ordinal it was given. Nothing breaks — ordinals need not be dense — but a writer reasoning about these keys should expect the case-insensitive match when clearing text rows and the byte-exact one when the constraint is checked. Splitting the two classes into independent ordinal spaces would take a schema migration (the primary key replaced by two partial unique indexes on value_blob IS NULL); it was judged not worth a store older builds refuse to open, and #663 records that decision.

Schema identity. On open, musefs also validates schema identity: a sqlite_master comparison against a freshly-migrated reference plus PRAGMA foreign_key_check, rejecting anything that is not the canonical latest schema with a message telling the user to run musefs scan. A store whose user_version is newer than this binary's latest migration (a future or third-party tool bumped the schema) is refused up front with a distinct "store is newer than this binary" error rather than silently treated as already-migrated — an older binary must not risk misreading a newer contract.

Migrations announce themselves. The opposite direction — an open that finds an older store and upgrades it in place — is irreversible (the store stops opening with the previous musefs build), so it is logged at warn, the default filter level: the store path, the version found and the version reached, followed by a completion line at info. Creating a store from scratch is not a one-way step for existing data and logs at info only, and the common case — a store already at the latest version — stays silent, since that path runs on every open and every mount.

Art is immutable once written. art rows are content-addressed by sha256; a trigger rejects any in-place UPDATE of an art row's content columns (data, sha256, mime, byte_len, width, height) with RAISE(ABORT) — a multi-row UPDATE art touching any content column aborts the whole statement. To change a track's art, insert a new content-addressed row and relink it via track_art (which bumps content_version); do not mutate an existing row. Deleting an art row still referenced by track_art (possible only with foreign_keys OFF) bumps every referencing track so the mount serves a clean EIO on the now-orphaned reference instead of stale bytes.

What musefs defends at serve time. CHECKs cannot catch a scanner-owned field mutated to a well-formed value that no longer matches the real file on disk: backing_size or backing_mtime_ns/backing_ctime_ns that drift from the actual file's stat, or audio bounds that fit the stored backing_size but overrun the file once it has shrunk. musefs re-stats the backing file on every resolve and treats such rows as untrusted input, degrading to a controlled BackingChanged/layout error, never undefined behavior. The store's CHECK rejects art over MAX_ART_BYTES (16 MiB − 64 KiB) at write time; resolve also re-checks it (ArtTooLarge, all formats) to backstop a writer that disables check enforcement, and the scanner refuses to ingest a file carrying such art at all (#644). Referential gaps are treated the same way: a track_art row whose art_id has no matching art row (an orphan an external writer can produce with FK enforcement disabled) fails the serve with EIO rather than silently dropping the art.

Merge vs. replace. An external writer may merge rather than fully replace text tags — overwriting only the keys it manages and leaving the rest of the scan-seeded set in place — provided it tracks its own managed-key set out of band (the beets plugin uses a beets flexattr; the store is not the place for plugin state). musefs renders tags outside its native VOCAB (musefs-format/src/tagmap.rs) by passthrough (Vorbis uppercased, mp3 TXXX, mp4 freeform), so such tags appear but are not guaranteed byte-identical to a given tagger's own per-format encoding. A merge matches the keys it manages case-insensitively, so a writer's canonical (lowercase) key replaces a scan-seeded row stored under the backing file's native case (e.g. Vorbis LABEL) instead of coexisting with it — Vorbis keys render case-insensitively, so two such rows would otherwise duplicate.

Path layout offload. External tools can also offload path layout entirely: a plugin evaluates its own (arbitrarily complex) path logic, writes the resulting relative path into a custom text tag — e.g. INSERT INTO tags (track_id, key, value, ordinal) VALUES (?, 'beets_path', 'Pink Floyd/Animals/01 Pigs', 0) — and the user mounts with --template '$!{beets_path}'. Because the field map is just the (lowercased) tag keys, any number of such tags (beets_path, lidarr_path, …) can back different concurrent mounts. The path field keeps embedded / as directory separators but sanitizes each segment and drops empty/./.. segments, so a misbehaving writer cannot inject traversal or empty components into the tree.

The shared Python library. contrib/python-musefs/ encodes this contract for plugin authors, including a generated copy of the schema (musefs_common/schema.py, regenerated from schema.rs by a drift-guarded test — see CONTRIBUTING). Its tag/art replace operations each wrap their DELETE+INSERT in a SQLite savepoint, so they are individually atomic and the "caller owns the transaction" guarantee holds even on an autocommit connection. The Lidarr integration uses the same shared library from a Custom Script workflow. Its Lidarr destination tree is only a tracking aid, made of symlinks by default; musefs remains the consumer-facing filesystem.

CI proves this contract end to end in the contract job (see CONTRIBUTING): a Python writer's tags/art, layered on a scanned track, are synthesized by the Rust serve path and read back by an independent reader.

External writers prune in one of two ways depending on how they own files. For in-place writers (e.g. the beets plugin), existence-based pruning — dropping the row of a removed backing file — is a deliberate act owned by musefs revalidate --prune; the plugin never prunes on its own (it exposes the revalidate pass via beet musefs --revalidate). The prune_missing helper in musefs_common implements the same by-existence delete for writers that prefer to own pruning themselves; like revalidate --prune it deletes only on a confirmed "not found" and keeps any row whose path it merely failed to stat. Link-tree writers (e.g. the Lidarr integration) never delete the backing files they point at, so they prune by identity instead: a source-reported album/artist deletion removes the rows carrying the matching MusicBrainz id.

Connections are mode-typed (Db<ReadWrite> / Db<ReadOnly>), opened in WAL mode with a busy timeout. The serve path uses a DbPool whose per-thread variant hands each reader thread its own connection — WAL reads never contend.

Freshness, tree & scanning

Freshness: two version counters

Two distinct counters drive correctness; they answer different questions.

content_version (per-track column) answers "did this track's served bytes change?". The DB triggers increment it on any input the database can see that changes synthesized bytes: tag and track_art edits, art-row deletes that orphan a reference, scanner-owned geometry changes (format, audio bounds, backing size/nanosecond-mtime), and FLAC structural-block changes. It is therefore a superset key — the one input it cannot cover is an on-disk backing change with no DB write, which resolve (and, since #279, a size-cache getattr hit) catches by re-statting the backing file and degrading to BackingChanged. The scanner stamps the backing file's (size, mtime_ns, ctime_ns) tuple from the probed file descriptor using a pre/post fstat sandwich: if the file's metadata changes between the two stats, the entry is dropped. ctime defeats an mtime-forging writer (e.g. touch -m). The HeaderCache (reader.rs) — a byte-budgeted concurrent cache (64 MiB default) of resolved layouts — keys each entry on it and on the backing-source identity the entry was built from: the row's backing_path and its stamp. Both axes are load-bearing. content_version answers a question about content, so a scan that retargets a row to a moved file — same bytes, new locator — deliberately leaves it alone, while the cached entry still carries the pre-move path that open_handle opens and the pre-move stamp that every serve validates against. A hit that mismatches either axis rebuilds. The getattr size cache compares content_version and the stamp for the same reason; it needs no path comparison of its own, because it re-stats the live path and holds no locator anything opens (#679). Independently of the cache, every resolve re-stats the backing file and errors with BackingChanged if its size, mtime, or ctime drifted from the scanned values, so a silently replaced backing file is never spliced at stale offsets. The per-handle read path re-stats the held descriptor on every read too, so this guarantee holds on the hot path and not only through resolve().

--trust-backing-mtime opts out of the getattr half of that, and of nothing else (#668). On a size-cache hit the mount then serves the cached size and mtime without the re-stat, because on NFS, SMB, or a spun-down array that stat is a network round trip or a head seek rather than a microsecond — one per track per traversal, on every traversal after the first. Resolve, open, and the per-handle read path keep validating unconditionally, so a silently replaced backing is still caught before a single byte is served, and the cold traversal that populates the cache stats regardless. What the flag trades away is the freshness of the one metadata surface that can outrun a backing change: between such a change and the next open, a stat reports the pre-change size and mtime. Off by default. musefs_trust_backing_mtime in .musefs-metrics reports the flag state, which is what tells a quiet musefs_backing_stats_total from a disabled counter.

data_version (PRAGMA data_version, whole-DB) answers "did anyone commit anything?". Musefs::poll_refresh compares it to the last seen value; on a change it consults the track_changes ring and applies an incremental, O(changed) rebuild: only the affected tracks' tree entries are re-rendered, exactly the removed tracks' cache entries are dropped, and the inodes whose content_version rose are reported to the FUSE layer. Any poll whose changelog names a track advances the refresh generation — not only one that changed a render key — because an open handle caches its resolved layout, backing path and stamp included, until that generation moves. If the mount slept past the ring's capacity (or the ring was truncated), it falls back to a full tree rebuild — correct by construction, and a bulk change wants one anyway. The new version stamp is committed only after a successful rebuild; failures arm a retry backoff.

The FUSE layer fires poll_refresh on metadata ops (lookup, readdir, …) off the dispatch thread, so external edits appear without remounting. Polling is debounced (--poll-interval-ms) and rebuilds are single-flighted: a metadata-op storm costs at most one rebuild per interval. When mounted with --keep-cache, the changed-inode notifications drive kernel page-cache invalidation (inval_inode), so a re-tagged file never serves stale cached bytes.

Virtual tree

VirtualTree::build (musefs-core/src/tree.rs) materializes an inode → node mapping from rendered paths. Paths come from beets-style templates (template.rs): $field / ${field} substitutions (with ${a|b} fallback chains) over the track's tag fields, each resolving through per-field fallbacks and then a global default_fallback; [...] conditional sections suppress their literals when every field they reference is empty. With skip_on_missing set (CLI --skip-on-missing), an unresolved top-level field instead drops the track from the mount: render_one returns None, so the track enters neither the snapshot nor the tree, and the incremental refresh path reclassifies a track that loses (or regains) such a field as a removal (or addition). Plain values are sanitized to a single path component ('/' and control characters become '_', components equal to . or .. are dropped, and any component is truncated to 255 bytes on a UTF-8 boundary so it stays within NAME_MAX), while a $!{field} path field keeps '/' as directory separators (sanitizing each segment and dropping empty/./.. segments) so a precomputed multi-level path expands into real directories. Path collisions are resolved deterministically by appending (k) before the extension (disambiguate). A node therefore carries two names: the rendered_name it came from and the possibly-ranked name it is served under. Which directory a path component belongs to is decided by the rendered name (dir_child_named), so every track under one rendered directory lands in one directory even when that directory was ranked away from its base name, and a directory whose rendered name is literally someone else's rank stays a separate directory.

The names the FUSE layer injects at the mount root — .musefs-metrics and .metadata_never_index, listed in RESERVED_ROOT_NAMES — are reserved in that namespace. The synthetic entry holds the base key the way a sibling node would (taken), so a rendered root component landing on one is ranked to (2) by the ordinary collision path and readdir can never emit a root name lookup will not return (#681). The reservation is unconditional — independent of --expose-metrics and of the host OS — so a library mounts to the same paths and inodes however the flag is set and wherever it runs.

mapping.rs bridges DB tag rows to the format layer's inputs and to template fields — ordering and multi-value semantics live there.

Inodes are stable across rebuilds: a persistent path→inode allocator (InodeAllocator) reuses an unchanged rendered path's inode and never recycles a retired one, so a descriptor held open across a refresh keeps resolving to the same node and a stale FUSE handle can never alias a different file. On case-insensitive mounts the key is case-folded, so a survivor keeps its inode even when an unrelated deletion flips a merged directory's display casing (#305). A path that vanished degrades to ENOENT, bounded by the entry/attr TTL. (Retired paths are pruned once they outnumber live ones, bounding the allocator at twice the live tree; a path that returns after a prune gets a fresh inode.)

Scanning

scan_directory (musefs-core/src/scan.rs) ingests a backing directory: collect supported audio files, probe each (format detection → audio offset/length, tags, pictures, structural blocks) on a parallel probe pipeline feeding a single DB writer, committing in batches. Probing reads are bounded — the scanner never slurps whole files — and ingestion caps per-item sizes (MAX_ART_BYTES, MAX_BINARY_TAG_BYTES, and the store's tags.key/tags.value/art.mime/track_art.description limits) so a crafted file cannot balloon the store.

check_storable applies every one of those caps in a single place, before anything is written. A file that exceeds any of them fails — that file, not the scan. It is logged with its path, what was too big, its size, and the limit (RUST_LOG=warn), and counted failed; the rest of the directory scans normally. Nothing partial is stored for it, so the mount never contains a track that is quietly missing a tag or its cover art. FLAC gets one extra check: tags merged from a leading ID3v2 tag (#602) can push the total past what a VORBIS_COMMENT block can hold, which would scan clean and then serve EIO on every read, so that total is checked at scan time too. A supported-extension file that fails to parse, or errors mid-probe, is likewise logged with the reason and counted failed.

check_storable covers the caps the scanner knows to look for, and no set of pre-checks can cover the rest: CHECK, UNIQUE and primary-key constraints are enforced by SQLite inside the ingest transaction, which is where #659 was discovered and where the next unanticipated shape will be. So the outcome is classified at the ingest boundary instead of enumerated ahead of time. A constraint violation (SQLITE_CONSTRAINT, any extended code) is attributable to the rows one file wrote: it fails that file, which is named in the log with the constraint text and counted in the rejected bucket of failed. Anything else — a corrupt, full, read-only or I/O-failing store, and any code this build does not recognise — still aborts the run, because carrying on would produce one identical failure per remaining file. SQLITE_BUSY is neither, and belongs to the writer's locking policy.

The production path commits through BulkWriter, whose transaction holds a whole batch, so failing one file means undoing only its rows. Each file is ingested inside a SAVEPOINT (BulkWriter::item) for exactly that: a statement-level ABORT rolls back only the statement that hit the constraint, which would otherwise leave the batch committing a half-ingested track — a tracks row whose tags or art never landed. The savepoint rolls the file back whole, and the rest of the batch stays committable (#662).

Before v1.4 an over-cap picture or binary tag was instead dropped with a warning and the rest of the track stored. That left users with a mount quietly missing data behind a warn that is easy to lose in a scan of ten thousand files, and it did not cover text tags at all — an over-cap tags.value reached the DB CHECK and aborted the entire scan with an error naming neither the file nor the limit (#644).

Symlinks are not followed by default: a symlinked file or directory is logged (RUST_LOG=info/warn) and skipped, which keeps the walk immune to directory-symlink cycles. Passing --follow-symlinks resolves them — symlinked audio files and directories are scanned — guarded by a visited (dev, ino) set so symlink cycles terminate, and by a second file-level (dev, ino) set so a file reached via both a real path and a symlink is ingested once rather than upserting its canonical track row twice. Because that set keys on (dev, ino), multiple hardlinks to the same inode are likewise collapsed to a single track under --follow-symlinks. Broken symlinks are logged and skipped without aborting the scan. The root argument is always followed regardless of the flag; only links encountered during recursion are gated.

revalidate is the maintenance pass: it re-probes only files whose (size, mtime_ns, ctime_ns) freshness stamp changed — a ctime-only move (e.g. a forged-mtime in-place rewrite) is still re-probed — and it preserves any external tag edits in the DB by refreshing only Layer A. New files are ignored: revalidate only touches rows that already exist in the store. Deletion is opt-in via --prune, which removes tracks under the scanned root whose backing file is gone and garbage-collects now-unreferenced art. Pruning is scoped to the scanned root, so revalidating one library root never removes tracks belonging to another. Because a track is keyed by its canonical backing path, a file scanned via --follow-symlinks whose real target lives outside the scanned root falls outside the prune scope: if that target later disappears, its stale row is not pruned by revalidating this root.

The contrib ecosystem

External writers live under contrib/: python-musefs is the shared store-contract library (schema-version check, tag/art writes, sha256 art content-addressing, the musefs scan shell-out); the beets plugin, the Picard plugin, and the Lidarr integration (a Custom Script workflow) build host-specific tag mapping on top of it. Each one's README covers its own setup and behavior; CONTRIBUTING covers their test suites and the generated-schema/vendoring mechanics.

Getting set up

The working manual for building, testing, and landing a change. For what the pieces are, read the architecture overview first; for per-format behavior, the format docs.

Map of this document:

• Getting set up — prerequisites and the pre-commit hook.
• Build & test — everyday commands, the FUSE e2e suite, the FreeBSD VM harness.
• Test tiers beyond cargo test — property tests, fuzzing, interop, the contract round trip, fault injection, mutation testing, sanitizers, coverage.
• Code conventions — errors, integer casts, lints, unsafe, layering.
• Adding a format — the four-step recipe.
• Python plugins (contrib) — per-suite commands and the gotchas.
• Releasing — the Python (py-v*) and Rust (v*) release flows.
• PRs & commits — conventions and the before-you-push checklist.

Getting set up

Prerequisites:

• Rust — stable (edition 2024) with rustfmt and clippy.
• FUSE (to mount, or to run the FUSE end-to-end tests) — Linux with /dev/fuse and libfuse (libfuse3-dev / libfuse3 plus pkg-config), or FreeBSD with /dev/fuse and the fusefs kernel module (no libfuse — see FreeBSD e2e for the in-tree VM harness).
• Python 3 with ruff and pytest — only for the Python plugin suites.
• shellcheck and yamllint — optional; the pre-commit hook's shell and YAML lint legs each skip with a notice if not installed.

Enable the repo's pre-commit hook once per clone:

git config core.hooksPath .githooks

The hook (.githooks/pre-commit) runs, in order: cargo fmt --all --check, cargo clippy --all-targets -- -D warnings, the full workspace test suite (cargo test --workspace), a conditional cargo-mutants anchor-drift guard (only when .cargo/mutants.toml, scripts/check_mutant_anchors.py, or a musefs-core/musefs-format source file is staged), shellcheck over every tracked shell script, yamllint (relaxed .yamllint) over every tracked YAML file, and ruff check + ruff format --check over contrib/beets/, contrib/picard/, contrib/lidarr/, contrib/python-musefs/, scripts/, and tests/interop/. A few consequences worth internalizing:

• A commit with red tests is always rejected — there is no "commit-now-fix-later" workflow here.
• Python-only changes hit the hook too: the ruff gate lints exactly the union of paths the CI jobs lint, so a commit can't pass the hook yet fail CI lint.
• The cargo gate (fmt/clippy/test) is skipped when every staged path is under docs/ or is a Markdown file, so a docs-only commit stays fast.
• The shellcheck/yamllint legs fire only when a shell or YAML file is staged, and skip with a notice when the tool is absent; when they do run they lint all tracked files of that type, so a sibling file can't drift unnoticed.
• The mutant-anchor guard fires only when the mutants config, its check script, or a musefs-core/musefs-format source file is staged, and skips with a notice when cargo-mutants is absent (CI re-checks it regardless). It re-validates that the .cargo/mutants.toml exclude_re anchors still point at their intended file:line:col after a line-shifting edit.

Build & test

cargo build                              # build the workspace
cargo test                               # all crates (excludes FUSE e2e)
cargo test -p musefs-core                # one crate
cargo test -p musefs-core read_at        # tests matching a substring
cargo clippy --all-targets               # lint (policy: see below)
cargo fmt                                # format

The musefs binary enables the default-on jemalloc feature (jemalloc global allocator + background purge thread). Build the system-allocator variant with cargo build -p musefs --no-default-features — used for the RSS comparison (scripts/rss-churn-bench.sh) and by packagers that forbid vendored C libs.

The FUSE end-to-end tests perform real mounts and are #[ignore]d:

cargo test -p musefs-fuse -- --ignored   # needs /dev/fuse + libfuse

The kernel-passthrough e2e additionally needs CAP_SYS_ADMIN. Don't run cargo under sudo — build first, then run the prebuilt test binary with sudo (find it in target/debug/deps/):

cargo test -p musefs-fuse --no-run
sudo target/debug/deps/<e2e_test_binary> --ignored <passthrough_test_name>

• Read-consistency harness (musefs-fuse/tests/read_consistency.rs): a seeded, reproducible randomized pread/mmap sweep compares live-mount reads against an in-memory oracle (the seed is printed on failure to reproduce). The hermetic FLAC tests — whole-file mmap fidelity and the read-only write-refusal matrix — always run; the multi-format breadth sweep generates fixtures with ffmpeg and skips any format whose codec is unavailable.

FreeBSD e2e

The FUSE e2e suite also runs on FreeBSD, via the scripts in scripts/freebsd-vm/. They are the single source of truth — CI and local runs invoke the same scripts, so they can't drift:

• run-local.sh — host-side orchestrator: creates and boots a FreeBSD VM under qemu/KVM and runs the suite in it. All artifacts go under the gitignored .scratch/freebsd/.
• provision.sh — in-guest: installs git, ffmpeg, and the current stable Rust toolchain via rustup (FreeBSD's packaged rust lags and is too old for some deps), and loads the fusefs kernel module. Run by run-local.sh and CI.
• run-e2e.sh — in-guest: cargo test --workspace then the --ignored FUSE e2e suite (guards that ffmpeg is present so the decode/encode tests don't silently skip).
• serial-run.py — drives the VM over its serial console (the console driver used by run-local.sh).

CI. The freebsd job in .github/workflows/ci.yml runs these in a vmactions/freebsd-vm VM. It is expensive (a full in-VM build), so it does not run on every PR — only when the FUSE/mount surface or its harness changed (musefs/, musefs-fuse/, scripts/freebsd-vm/, Cargo.lock, ci.yml) or on a release tag (v*).

Local run (one command):

sh scripts/freebsd-vm/run-local.sh

Host prerequisites (Debian/Ubuntu packages in parens): qemu-system-x86_64 + qemu-img (qemu-system-x86, qemu-utils), xorriso (xorriso), curl + xz (curl, xz-utils), python3. /dev/kvm for acceleration (it runs without it, just far slower); ~6 GB free under .scratch/.

What it does, end to end:

1. Downloads the official FreeBSD-<rel>-amd64-BASIC-CLOUDINIT-ufs.qcow2 image into .scratch/freebsd/ (cached; downloaded once). That image directs its console to the serial line, which is what lets the harness drive it.
2. Creates a fresh overlay disk from the cached base each run (cheap reset).
3. Boots the VM headless and logs in as root over the serial console (the image has an empty root password — no SSH, no keys, no cloud-init).
4. Serves this repo over a throwaway HTTP server on qemu's user-net gateway (10.0.2.2); the guest fetches and unpacks it.
5. Runs provision.sh + run-e2e.sh over the console and propagates the exit code, then powers the VM off.

Tunable via env: FREEBSD_REL (default 14.3-RELEASE), VM_MEM, VM_SMP, VM_DISK, HTTP_PORT, RUN_TIMEOUT.

To drive your own VM instead, boot any FreeBSD image and, from the repo root inside it as root, run sh scripts/freebsd-vm/provision.sh then sh scripts/freebsd-vm/run-e2e.sh.

Notes:

• FreeBSD uses fuser's pure-rust /dev/fuse backend — no libfuse package; only the fusefs kernel module and base-system mount_fusefs(8) are needed.
• Kernel FUSE passthrough (StructureOnly) is Linux-only; on FreeBSD it falls back to daemon serving.

macOS support is best-effort: CI builds there with fuser's macos-no-mount feature, and the platform-specific logic is unit-tested. Mounted e2e on macOS/FUSE-T is not yet validated.

Test tiers

Test tiers beyond cargo test

Property tests

proptest invariants — panic-freedom, the byte-identical-audio guarantee, tag round-trip stability — live in musefs-format/tests/proptest_*.rs and musefs-core/tests/proptest_read_fidelity.rs. The format-layer suites are gated on the fuzzing feature, which musefs-format's self-dev-dependency enables for all of its own test builds — so a plain cargo test -p musefs-format runs them.

Coverage-guided fuzzing

The fuzz/ crate is excluded from the workspace: workspace-wide build, test, and clippy do not compile it, so a format-layer signature change can break fuzz targets without anything failing locally — CI's fuzz smoke job (cargo +nightly fuzz build) is what catches it. Check locally before pushing a format-layer API change:

cargo install cargo-fuzz                          # one-time; needs nightly
cargo +nightly fuzz build                         # what the CI smoke job runs
cargo +nightly fuzz run <target>                  # flac|mp3|mp4|ogg|wav|ogg_page|b64|vorbiscomment|serve
cargo +nightly fuzz coverage <target>             # confirm coverage reaches the parser
cargo run --manifest-path fuzz/Cargo.toml --bin generate_seeds   # (re)build seeds

Fuzz crash regressions

When you fix a fuzz-found crash:

1. Drop the reproducer bytes into fuzz/regressions/<target>/ (one file per reproducer). The per-PR fuzz smoke job's replay step runs every committed reproducer with cargo +nightly fuzz run <target> <files> -- -runs=0 — a deterministic single pass that fails the build if any known input panics again. This is separate from fuzz/corpus/, which cargo fuzz cmin minimizes (and would prune reproducers from).
2. Where the crash exposed a real logic/behavior defect, also add a focused behavioral test for that logic in the owning crate's suite (the pre-commit hook gates it). The byte replay proves the exact input no longer panics; the behavioral test documents and locks in the fix. They are not interchangeable.

Coverage notes: the per-format targets also drive the bounded/ceiling probers (*_bounded, locate_audio_at_ceiling, read_structure_from) and assert a differential oracle against the full-buffer parse. The serve target fuzzes the read-time serve path (read_at_with_file over adversarial layouts, including serve_ogg_window/OggArtSlice) and is scheduled-only (built per-PR, not smoke-run) because it builds a DB + temp backing file per input. The serve target also exercises hostile DB rows (negative/oversized geometry, invalid formats, orphaned/oversized art, stale binary-tag handles, content-version mismatch) via the musefs-db fuzzing-gated with_raw_conn, plus binary-tag streaming and distinct Opus/Vorbis/OggFLAC fixtures.

Independent-reader interop (mutagen)

Asserts that an independent ecosystem reader sees the tags musefs synthesizes, across all five formats:

pip install -r tests/interop/requirements.txt
MUSEFS_INTEROP_DIR=/tmp/i cargo test -p musefs-core --test interop_emit -- --ignored emit_interop_fixtures
MUSEFS_INTEROP_DIR=/tmp/i python -m pytest tests/interop

External-writer contract round trip

CI's contract job mandatorily proves the Python -> Rust DB contract: it builds the binary, runs each binary-only plugin's musefs_bin tier with MUSEFS_REQUIRE_BIN=1 (a missing binary fails instead of skipping), and runs the round-trip harness. The harness is the single source of truth, run locally with:

pip install -r tests/contract/requirements.txt pytest && pip install -e contrib/python-musefs
bash scripts/contract-roundtrip.sh

It scans real ffmpeg-generated audio (so musefs scan owns the track geometry), writes tags/art through musefs_common.store, synthesizes the served bytes via cargo test --test contract_emit, and asserts with mutagen that the Python tags and art survived. Picard's musefs_bin tier runs in the picard job (it needs the system-Picard environment).

Failure-path fault injection

The reader and DB error paths are exercised under simulated runtime faults. musefs_core::metrics::set_backing_fault(BackingFault::{Eio,ShortRead}) (behind the metrics feature) installs a process-global fault at the positioned backing-read site, cleared by the returned RAII guard. Because it is global, the tests run in their own metrics-gated binaries.

cargo test -p musefs-core --features metrics --test reader_faults
cargo test -p musefs-core --test backing_changed_fault   # real file mutation
cargo test -p musefs-core --test db_corruption_fault      # byte-corrupt DB
cargo test -p musefs-fuse --features metrics -- --ignored # EIO through the mount (needs /dev/fuse)

BackingChanged (re-validated in HeaderCache::resolve) and DB corruption are driven by real conditions, not the seam. ENOSPC/read-only faults are write-path concerns and are out of scope for the read-time suite.

Mutation testing

scripts/mutants.sh wraps cargo-mutants for the logic-bearing crates; .cargo/mutants.toml permanently excludes the thin glue crates (musefs-fuse, musefs-cli, musefs) and feature-gated instrumentation. musefs-latencyfs carries real logic and has its own leg (it needs /dev/fuse to kill its mutants).

The CI parity check for a branch is the in-diff gate — mutate only the lines your branch changed:

git diff "$(git merge-base main HEAD)...HEAD" -- '*.rs' > mutants.diff
grep -q '^@@ ' mutants.diff   # IMPORTANT: an empty diff mutates nothing and exits 0 — a silent false pass
cargo mutants --in-diff mutants.diff -j2 --exclude 'musefs-latencyfs/**' --output /tmp/mutants-out/in-diff

Sharp edges:

• Check the exit status directly. Don't pipe the run through tail/grep — that masks the exit code.

• Scratch space and memory. cargo-mutants copies the source tree into a scratch dir under TMPDIR/MUTANTS_TMP (which must be outside the repo). For a small in-diff mutant set, the default tmpfs /tmp is fine — and faster. For large sets (a full-crate campaign), some mutants are allocation bombs (e.g. a constant-return on a parser position helper spins a collect-loop) that can OOM the host before the test timeout fires: put TMPDIR on real disk and run inside a memory-capped cgroup, e.g.

  mkdir -p ~/.cache/musefs-mutants-tmp
  TMPDIR="$HOME/.cache/musefs-mutants-tmp" systemd-run --user --scope --collect \
      -p MemoryMax=10G -p MemorySwapMax=0 \
      cargo mutants --in-diff mutants.diff -j2 --exclude 'musefs-latencyfs/**' --output /tmp/mutants-out/in-diff
  

  scripts/mutants.sh also supports sharding (MUTANTS_SHARD=i/n, used by CI to split the long musefs-format leg), though a sharded local workflow hasn't been built out.

• Known-unkillable mutant classes get a documented exclude_re in .cargo/mutants.toml, not test contortions. Note that cargo-mutants mutates const initializer expressions too — a constant is not a hiding place for arithmetic the gate flags.

• exclude_re entries are guarded against drift. A few exclusions must pin a specific file:line:col: (the operator+function alone isn't unique in the function); those coordinates rot silently when cargo fmt shifts the code, and a stale anchor can re-point onto a killable mutant — a silent false pass. scripts/check_mutant_anchors.py prevents that: it lists the full unfiltered mutant set (cargo mutants --no-config --list --json) and re-validates every exclude_re entry. It runs in the per-PR in-diff job (.github/workflows/mutants.yml) and its unit tests run in CI's python-musefs job. Run it locally with:

  cargo mutants --no-config --list --json > /tmp/mutants-list.json
  python3 scripts/check_mutant_anchors.py --mutants-json /tmp/mutants-list.json
  

  Each entry carries a machine-checked # guard: comment on the line directly above it:

  • file:line:col anchors — # guard: op="<" fn="probe_file" rows=3. The guard asserts the matched mutants all share that operator and function, occupy one site, and number exactly rows (use fn="" for a const-level site with no enclosing function). A narrowing entry (one that embeds a replacement to leave same-site siblings killable) sets rows to that subset's size.
  • description anchors — # guard: count=N (default 1) asserts the entry matches mutants spanning exactly N distinct sites; this is what catches a newly-added killable sibling silently joining the match set. A bare single-site description entry needs no tag.

  When the guard fails: a found none message means a line:col anchor drifted — re-anchor it to the current coordinates from the listing and re-confirm the mutant there is still genuinely equivalent (a reformat can change surrounding logic, not just line numbers). A count/rows mismatch means a sibling appeared or disappeared — investigate before bumping the number. Pure cargo fmt/line-shift drift can often be repaired automatically with python3 scripts/check_mutant_anchors.py --fix, which re-points an anchor to its current coordinates by operator+function. It only does so when the mapping is unambiguous — every same-operator site in the function is anchored, so the positional match is exact. An anchor that pins one of several same-operator sites (the usual reason it is a file:line:col anchor rather than a description) cannot be derived from the tag alone, so --fix leaves it for manual re-anchoring and reports can't auto-derive the coordinate; it also declines when a site was added or removed. Always eyeball the resulting diff before committing. Every new file:line:col exclusion needs a # guard: tag (the guard rejects an untagged one), and exclude_re patterns must stay within the Rust-regex/Python-re shared subset the guard allows (\. \d + | ^ ( ) *, no inline (?...) groups).

Performance regression gating

cargo test -p musefs-core --features metrics includes tests/perf_counters.rs: golden assertions on deterministic work counters (preads, pread_bytes, scan_bytes_read, art/binary-tag chunks) for the read/serve and ingest paths, plus a tree.rs unit test pinning the refresh rebuild count as size-invariant. These are a hard gate — a legitimate change to read/ingest/refresh work must update the golden numbers in the same PR. They run on every non-doc PR via CI's check job. Constant-factor (wall-clock) changes are surfaced separately by the warn-only perf-ab job (below).

The A/B benchmark runs only when musefs-core/src/** or musefs-format/src/** change. The perf-bench matrix job benches the base and PR commits in parallel on separate runners (one ref each), then the perf-ab job downloads both exported baselines and posts a critcmp delta as a sticky PR comment. It is warn-only and not a required check — GH runner noise (now including cross-runner variance) makes wall-clock unfit for hard gating. Reproduce locally on one machine with scripts/perf-ab.sh <base-sha> out.md.

Concurrency + sanitizers

Concurrent-reader coverage exists at two levels:

cargo test -p musefs-core --test concurrent_reads          # core: HeaderCache + WAL reads (default suite)
cargo test -p musefs-fuse --test concurrent_reads -- --ignored  # mount: DbPool::PerThread (needs /dev/fuse)

The core binary also carries the read-ahead pool's budget-accounting stress (readahead_*, #628): 16 threads register/read/deregister streams against a 4 MiB budget so eviction fires constantly, then assert charged == Σ(registered buffers' bytes.len()) at quiescence — the invariant #536 restored. It lives in this binary precisely so the sanitizer legs below reach it; the serve-path tests around it run with read-ahead disabled. Rounds default to a sanitizer-friendly 24 per thread; MUSEFS_READAHEAD_STRESS_ROUNDS=200 soaks it locally.

CI runs the core test under AddressSanitizer as a required gate (asan job) and both tests under ThreadSanitizer as a non-required best-effort signal (tsan job, continue-on-error). TSan cannot instrument the system C libraries (libfuse, libsqlite3), so it is a signal, not a gate. ASan is ABI-compatible with an uninstrumented std, but TSan is not — so the TSan command needs -Zbuild-std (and the rust-src component) to rebuild std with the sanitizer. Reproduce locally with:

rustup toolchain install nightly
rustup component add rust-src --toolchain nightly   # for TSan's -Zbuild-std
RUSTFLAGS="-Zsanitizer=address" ASAN_OPTIONS="detect_leaks=0" \
  cargo +nightly test -p musefs-core --test concurrent_reads --target x86_64-unknown-linux-gnu
RUSTFLAGS="-Zsanitizer=thread" TSAN_OPTIONS="halt_on_error=0" \
  cargo +nightly test -p musefs-core -Zbuild-std --test concurrent_reads --target x86_64-unknown-linux-gnu

The uninstrumented-C caveat is not theoretical, and it reaches the core test — not just the mounted one. The tsan leg intermittently reports a data race on a memcpy inside walIndexWriteHdr / walIndexTryHdr (sqlite3.c), reached through Db::open_readonly from the serve-path tests. That is SQLite's WAL-index header, guarded by barriers of its own that TSan cannot see — a false positive, not a musefs defect. Every test still passes when it fires; the leg reddens only because TSan exits 66 whenever it emits a warning at all. It is probabilistic, so it shows up on some runs and not others.

So before treating a red tsan leg as a finding, check whether any frame in the reported stacks names musefs code. Pure sqlite3.c stacks are this known noise; a frame in musefs_core is worth investigating.

Dependency advisories & licenses

Two supply-chain gates run in CI and are worth reproducing locally before a dependency bump:

cargo install cargo-audit
# Same 0.19 pin both CI jobs use — the config schema shifts across minor releases.
cargo install cargo-deny --locked --version '^0.19'
cargo audit                                                    # root Cargo.lock, RUSTSEC advisories
cargo deny check                                               # advisories + licenses + bans + sources
cargo deny --manifest-path fuzz/Cargo.toml check --config deny.toml advisories

audit.yml runs rustsec/audit-check (root lockfile only) plus the third command above, because fuzz/ is a separate workspace with its own lockfile that neither the action nor the ci.yml cargo-deny job would otherwise see. cargo-deny adds the license allow-list, which cargo-audit does not check.

Advisories with no upgrade path are allow-listed with a written rationale at the ignore site — .cargo/audit.toml for cargo-audit/the action, deny.toml for cargo-deny (both lists must agree). An unmaintained advisory on a compile-time-only or unreachable dependency is a candidate; a vulnerability is not — those get fixed. Note that cargo-deny warns advisory-not-detected for an ignore whose crate is absent from the lockfile being scanned, which is expected for entries that only apply to one of the two workspaces.

Coverage

cargo install cargo-llvm-cov
cargo llvm-cov --workspace --exclude musefs-fuse --exclude musefs-latencyfs --open
cargo llvm-cov --workspace --exclude musefs-fuse --exclude musefs-latencyfs --lcov --output-path lcov.info

musefs-fuse and musefs-latencyfs are excluded because these FUSE crates' tests need a real mount; their behavior is covered by the separate e2e CI job rather than llvm-cov. The CI e2e job also runs the binary-level cargo test -p musefs -- --ignored and cargo test -p musefs-latencyfs -- --ignored suites so they cannot silently rot (they require /dev/fuse + fusermount3). CI (coverage.yml) runs this on every push/PR and uploads to Codecov (CODECOV_TOKEN repo secret).

Conventions & adding a format

Code conventions

• Errors. Each crate has its own error.rs with a thiserror enum; musefs-core wraps lower layers in CoreError; the CLI is the only anyhow consumer. Internal error paths never discard diagnostics: no Result<_, ()>, no .map_err(|_| …) that drops a source — each variant carries its source (#[from]) or a static reason naming the broken invariant.
• Integer conversions. The four clippy cast lints are deny-via-CI. Widenings use From; u64 -> usize only via the sanctioned usize_from helpers (musefs_db::convert, re-exported by core; musefs-format and musefs-latencyfs carry crate-local siblings — the workspace is declared 64-bit-only); genuine narrowings use try_from (? for input-dependent values, .expect for structurally bounded ones, .unwrap in tests); deliberate bit-truncation keeps as under a reasoned #[expect]. Non-negative DB row fields are unsigned; rusqlite's checked conversions (feature fallible_uint) validate at the row boundary.
• Lint policy. clippy::pedantic minus a few intentional/noisy groups, defined in the root Cargo.toml under [workspace.lints]. The hook and CI deny all warnings.
• Unsafe code. unsafe_code = "deny" is set for the workspace members in the root Cargo.toml ([workspace.lints.rust]); the standalone fuzz/ crate is outside the workspace and is not covered. A genuinely-necessary unsafe is opted in per-site with #[expect(unsafe_code, reason = "...")] — never a bare unsafe block and never by relaxing the workspace lint, so every unsafe is greppable and review-visible. Prefer a safe crate (e.g. rustix for syscalls) over hand-rolled FFI.
• Layering. Keep musefs-fuse, musefs-cli, and the musefs binary thin; cross-cutting logic belongs in musefs-core (see the crate layout).
• Hidden API consumers. benches/ directories and each crate's tests/ are compiled only by --all-targets: after an API change, compile-check with cargo clippy --all-targets, not cargo build.

Adding a format

1. Implement probe + synthesize_layout in musefs-format (mirror an existing module — flac.rs, mp3.rs, mp4.rs, ogg/, wav.rs), returning a RegionLayout.
2. Add the variant to musefs-db's Format enum, then wire it into the match track.format arms in reader::HeaderCache::resolve (musefs-core/src/reader.rs) and into scan.rs (extension list, probe dispatch).
3. Extend the test surface: a fuzz_check::fixtures::<fmt>() minimal file, a fuzz/fuzz_targets/<fmt>.rs target with a seed in generate_seeds, a musefs-format/tests/proptest_<fmt>.rs, and a manifest row in musefs-core/tests/interop_emit.rs.
4. Write docs/<FMT>.md (follow the shape of the existing five).

Python plugins

Python plugins (contrib)

The four packages share one drift-guarded contract; see the contrib ecosystem for the layout and the integration pages for plugin-specific setup.

# python-musefs: self-contained
cd contrib/python-musefs && python -m pytest && ruff check . && ruff format --check .

# beets: install the local python-musefs first so the suite tests the working
# tree, not the PyPI release (see the beets integration page for the venv flow)
cd contrib/beets && pip install -e ../python-musefs && pip install -e ".[test]" && python -m pytest tests

# picard: no install needed (vendored + pythonpath=".")
cd contrib/picard && python -m pytest tests

# lidarr: install the local python-musefs first so the suite tests the working
# tree, not the PyPI release (see the lidarr integration page for the env flow)
cd contrib/lidarr && pip install -e ../python-musefs && pip install -e ".[test]" && python -m pytest tests

Gotchas that have bitten before:

• On PEP 668 "externally managed" systems, bare pip install fails — use a venv for the beets suite.
• The real-Picard tests importorskip Picard and Qt: without an importable Picard (e.g. the system package on PYTHONPATH), they silently skip. When touching the Picard plugin, make sure they actually ran.
• The Lidarr integration is gated by two automated tiers, both deterministic and network-free (Lidarr's metadata server is mocked too):
  • PR check — .github/workflows/lidarr-smoke.yml (scripts/lidarr-smoke.sh): a fast smoke that proves the Custom Script exec path on a real Lidarr (its Test event) and runs the content leg (musefs-lidarr-sync tag-writes, musefs-lidarr-import symlink, served-mount tags, unchanged bytes) against a local mock Lidarr API. Runs on PRs touching the Lidarr surface.
  • Release gate — .github/workflows/lidarr-e2e.yml (scripts/lidarr-e2e/run-e2e.sh): the full real-instance e2e. A real Lidarr, driven by local metadata/indexer/qBittorrent mocks, performs a genuine download-client import of a real CC0 album as a NewDownload, firing OnReleaseImport, which execs the real musefs scripts; the served mount is then asserted to carry Lidarr-supplied metadata the backing file lacked, bytes unchanged. This gates the Python py-v* publish and closes what used to be the manual download-client gap. The vendored CC0 fixture is scripts/lidarr-e2e/fixtures/.
• musefs_common/schema.py is generated from musefs-db/src/schema.rs. After a schema change: MUSEFS_REGEN_SCHEMA_PY=1 cargo test -p musefs-db schema_py, then re-vendor Picard's copy with python contrib/python-musefs/vendor_to_picard.py. Drift is enforced by a musefs-db unit test and the Picard vendor-sync test.
• MAX_ART_BYTES in contrib/python-musefs/src/musefs_common/constants.py is hand-mirrored from musefs-core/src/scan.rs — update both sides together.

Releasing

Releasing the Python packages

The contrib/ Python packages (python-musefs, beets-musefs, lidarr-musefs, and the unpublished musefs-picard) share a single version, decoupled from the Rust crates and released on a py-v* tag. musefs-picard tracks the version but is not uploaded to PyPI (Picard has its own plugin registry; the shared library is vendored into it).

One-time setup (before the first release). Trusted Publishing fails until the publisher exists on PyPI. For each of python-musefs, beets-musefs, and lidarr-musefs:

1. Create/reserve the project on PyPI.
2. Add a GitHub Actions trusted publisher pointing at: owner/repo Sohex/musefs, workflow release-python.yml, environment pypi.

Also create a GitHub environment named pypi in the repo settings (it gates the publish job).

Cutting a release:

1. Choose the new version X.Y.Z and run python scripts/bump_python_version.py X.Y.Z. This rewrites every contrib/*/pyproject.toml version, the __version__ strings, the python-musefs>= dependency floors, and re-vendors python-musefs into the Picard plugin.
2. Review git diff — it should touch only the version/floor lines and the Picard vendored _common/ copy.
3. Promote the ## [Unreleased] section of contrib/CHANGELOG.md to ## [X.Y.Z] - <date>.
4. Commit, then tag and push:

   git commit -am "release: python packages X.Y.Z"
   git tag py-vX.Y.Z
   git push origin HEAD --tags
   

5. release-python.yml runs the version gate, the four Python test suites, then publishes python-musefs, beets-musefs, and lidarr-musefs to PyPI (in that order).

Releasing the Rust crates and binaries

The Rust workspace publishes to crates.io and ships prebuilt cross-compiled binaries on a v* tag, decoupled from the Python py-v* flow. release.yml runs one ordered graph — gate → build → smoke → publish → release-assets — and is the source of truth; this checklist is the human side.

Pre-flight.

1. Working tree clean, on the commit you intend to release.

2. Confirm main is green (CI + coverage). The tag push triggers a fresh ci.yml and coverage.yml run, and the release gate job waits for ci-ok and coverage-ok to be green on the tagged commit before anything builds or publishes — a red tree blocks the release automatically.

3. CARGO_REGISTRY_TOKEN is present in repo secrets.

4. Smoke-build every cross target so jemalloc-sys is known to compile under zig before tagging (the release matrix builds with the jemalloc feature on):

   for t in x86_64-unknown-linux-gnu.2.17 aarch64-unknown-linux-gnu.2.17 \
            x86_64-unknown-linux-musl aarch64-unknown-linux-musl; do
     cargo zigbuild --release -p musefs --target "$t"
   done
   

   If a target cannot build jemalloc-sys, add --no-default-features to that matrix entry's cargo zigbuild in release.yml, rather than blocking the release. The Docker images COPY the binary this step produces (they don't run cargo), so the matching container inherits the opt-out automatically.

Version bump (do this in one commit before tagging).

1. Pick the new version X.Y.Z.
2. Bump the workspace version in Cargo.toml.
3. Bump every internal musefs-* path-dependency constraint that pins the old version (e.g. musefs-db = { version = "X.Y.Z", path = "..." }) — a stale internal floor fails the publish.
4. Promote the ## [Unreleased] section of CHANGELOG.md to ## [X.Y.Z] - <date>.
5. Dry-run package each crate: cargo package -p <crate> --locked for each of musefs-db musefs-format musefs-core musefs-fuse musefs-cli musefs. This catches packaging errors but not the cross-crate index-propagation problem (it resolves siblings via path deps); that is handled in-workflow (next section).
6. Commit, e.g. git commit -am "release: vX.Y.Z".

Tag and push.

git tag vX.Y.Z
git push origin HEAD --tags

The tag push starts both CI and release.yml. The gate job blocks publishing until ci-ok + coverage-ok are green on the tagged tree (45-minute timeout, covering the full matrix including the FreeBSD VM e2e).

What release.yml does.

1. gate — verifies the tag matches the workspace version and waits for the required CI checks to pass on the tagged commit (fails closed on a failed check or timeout).
2. build — cross-compiles the four target binaries.
3. smoke — runs the binary smoke on each target (host + Alpine).
4. publish — publishes crates in dependency order. For each crate it skips the publish if name@version already resolves from the crates.io index, then waits for that version to appear before publishing the next dependent crate (index-propagation; #163). The skip makes a whole-workflow re-run after a partial failure safe.
5. release-assets — creates/updates the GitHub Release and uploads the binary tarballs + checksums (only after crates publishing succeeds).

Retry / rollback.

• crates.io is yank-only — a published version cannot be un-published.
• A partial failure (e.g. crate 3 of 6 published, then a transient error) is recovered by re-running the workflow: the publish loop skips the crates already in the index and resumes, then runs release-assets. No manual cleanup of the published crates is needed.
• GitHub asset upload is idempotent (gh release upload --clobber), so re-runs re-upload safely.

Post-release verification.

1. cargo install musefs (or cargo install musefs --version X.Y.Z) from a clean machine/container.
2. Download a release tarball and verify its checksum: sha256sum -c musefs-X.Y.Z-<triple>.tar.gz.sha256.
3. Confirm all four target tarballs + .sha256 files are attached to the GitHub Release.

Lidarr gate at a v1.0.0 milestone. The Lidarr real-instance e2e (lidarr-e2e.yml) gates the Python py-v* release, not this Rust flow. When a v1.0.0 milestone bundles both, ensure the Python release (and therefore its Lidarr e2e gate) is also run.

PRs & commits

• Conventional-style subjects (fix(format): …, docs: …, ci: …), scoped and imperative.
• main is protected by required status checks: the ci-ok and coverage-ok aggregator jobs must pass. CI also runs the fuzz smoke build, the in-diff mutation gate, and a security audit on PRs. Docs-only changes skip the expensive jobs at the job level — the aggregators still report.
• Benchmark results, when a change warrants them, are recorded in Benchmarks.

Before you push

The pre-commit hook already gates fmt, clippy, the workspace tests, and the Python/shell/YAML lints on every commit. What it does not run — check the ones your change triggers:

• Logic changes → the in-diff mutation gate. It is CI parity, not optional polish.
• Format-layer API changes → cargo +nightly fuzz build; the fuzz/ crate is outside the workspace, so nothing else compiles it (coverage-guided fuzzing).
• musefs-db schema changes → regenerate and re-vendor the Python schema mirror (Python plugins).
• Picard plugin changes → make sure the real-Picard tests actually ran rather than silently skipped (gotchas).
• FUSE/mount-surface changes → run the --ignored e2e suite locally (Build & test); the FreeBSD CI leg only runs on PRs that touch that surface.

Benchmarks

Every optimization pass re-measured apples-to-apples on one box as a PR-isolated before/after pair, plus a cumulative 16caba4→main summary. This file is performance only — correctness gates (byte-identical proptests, FUSE e2e, in-diff mutation) live in CI and the contributor guide, not here.

Read it in three layers:

1. Results at a glance — the cumulative per-subsystem delta and a one-line headline per pass.
2. Methodology — machine, before/after definition, the overlay rule, run conventions, storage placement. Written once; every detail section assumes it.
3. Per-pass detail — one section per pass: what changed, the before/after table, the reproduce command, and the "why" where it matters.

Results at a glance

Cumulative — 16caba4 → current main (e02223e)

Composed from the per-pass isolated deltas below, anchored to current-main absolutes. Non-isolating: a same-harness run at both ends is infeasible (API drift means neither the 16caba4-era harness nor the main harness compiles at the other commit), so these compose the chain of passes that touched each subsystem rather than a single end-to-end measurement. See Cumulative detail for the absolutes and the per-pass composition.

Per-pass headlines

Each headline is the pass's single largest statistically-significant delta on its deployment-representative tier.

One direction inverted vs the historical file: SP1 §4 (compute-isolated, on RAM) is now faster after the change, not slower. The old file recorded SP1 as ~1.9× slower on RAM-backed tempfs (the "honest cost" of the pipeline); on this 8-core box the parallel pipeline wins even on RAM (~1.4×), at higher peak RSS. See SP1 §4.

CI regression gating

BENCHMARKS.md records hand-run absolute numbers; CI guards against regressions in three lanes:

1. Counter gate (every non-doc PR, hard). perf_counters.rs + tree.rs golden work-counter assertions under --features metrics. Catches algorithmic regressions (extra copy, whole-file slurp, O(N) tree rebuild).
2. A/B wall-clock (warn-only, core src PRs). The perf-bench matrix job benches the base and PR commits in parallel on separate runners; the perf-ab job then diffs the two exported baselines and posts a critcmp delta as a PR comment. Never blocks.
3. Release record. The benchmarks job runs the full bench suite at the ci tier on a tag and uploads the numbers as an artifact for curation here.

The fsync-storm (403→0) signal needs a real FUSE mount and lives only in the release lane / the #[ignore] bench_scan_under_latency, not the per-PR gate.

The release artifact is named benchmark-snapshot-<tag>; download it from the tag's workflow run. The job runs on a GitHub-hosted ubuntu-latest runner, not the dedicated box the rest of this file uses, so its wall-times are runner-relative and are not folded into the per-pass tables — only the portable signals (bytes_read, pread/fsync counts, refresh flatness) are cross-comparable. Each release's snapshot is recorded verbatim under Release CI snapshots.

Release CI snapshots

Per-tag records from the benchmarks release job (CI regression gating §3), run on a GitHub-hosted ubuntu-latest runner at the ci tier. This is not the dedicated box the per-pass tables use, so the wall-times here are runner-relative and are a point-in-time record per release — not comparable to those sections. The portable signals (scan_bytes_read, pread/fsync counts, refresh flatness) are comparable, and are the no-regression check.

v1.1.0 — f865afc, single run

No regression vs the curated tables: scan_bytes_read is unchanged from PR2 (flac/ogg/wav = 845 000 / 847 400 / 828 000 B — the −128 B/file ID3v1 gating still holds; mp3 847 200 B unchanged; m4a uses the seek-reader, 0 B), and single-track refresh stays flat with library size.

read_throughput (Criterion, median estimate, µs):

concurrent_read_walk/m8_plus_walker: 931 µs.

† sequential_read/ogg collected only 10k iterations with 19% outliers (Criterion low-sample warning) — treat as noisy.

bench_ingest — ci tier (200 tracks × 4 KiB), runner tmpfs:

bench_refresh — ci tier, single-track re-tag:

refresh-1 vs refresh-N (200-track ci, same instance): refresh-1 0 ms, refresh-N (100 touched) 6 ms. The 5000 > 20000 inversion is single-run noise on the shared runner; the dedicated-box #69 sweep is the clean flat signal.

Methodology

Machine

Before / after definition

History is squash-merged (linear), so each pass is one commit:

• after = the pass's own squash-merge commit.
• before = its parent, <after>^ — PR-isolated, not current main. This preserves attribution (each delta is exactly what that PR changed) and avoids harness drift from later passes.

The overlay rule

Two passes (SP2, SP4) report a bench that did not yet exist at their before commit. For those, the after-commit's harness file is checked out onto the before checkout (git checkout <after> -- <bench_file>) so the old code is measured with the new harness. Overlay use is called out in each affected section.

Run conventions

• bench_ingest / bench_refresh (ignored tests, cargo test --release … -- --ignored): 3 runs, median reported (spread noted where it matters). bench_ingest needs --features metrics.
• read_throughput (Criterion bench): Criterion's own sampling; before side saved with --save-baseline, after side compared with --baseline. Reported Δ is Criterion's change estimate.
• Wall times on /data are box-relative (rotational disk); where a portable signal exists (fsync count, bytes_read, pread count) it is the primary number.

Storage placement

• Durable rows run on /data (rotational btrfs). bench_ingest honors MUSEFS_BENCH_DIR.
• RAM rows run on /dev/shm (tmpfs). bench_ingest honors MUSEFS_BENCH_DIR=/dev/shm/…; bench_refresh and read_throughput ignore it and follow TMPDIR=/dev/shm.

Per-pass detail

SP1 — Ingestion scalability

ccbbfaa^ → ccbbfaa. bench_ingest, --features metrics. No overlay.

What changed: whole-file fs::read slurp + per-file commits at synchronous=FULL → bounded probing reads + parallel-probe/single-writer pipeline + per-batch transactions at synchronous=NORMAL (WAL retained).

1. Durable small files — the fsync/batching win

ci tier (200 tracks × 4 KiB, no embedded art), corpus + DB on /data. Not compute-bound — the before path is dominated by per-file fsync latency.

2. Durable large files — bounded reads + batching

bandwidth tier (1000 tracks × 30 MiB FLAC + art ≈ 30 GiB), on /data, 1 run.

The after path reads only a ~1 MiB metadata window per file instead of slurping each 30 MiB file in full.

3. fsync count — the mechanism

ci tier (200 FLAC) scanned through the passthrough latency-FS (ssd profile), which counts fsyncs at the FUSE layer. Wall is box-relative (rotational /data); the fsync count is the portable signal.

The 403→0 collapse is the root cause of §1's durable speedups.

4. Compute-isolated (RAM) — the trade, now a win on this box

large-compute tier (100k tracks × ~38 KiB FLAC) on /dev/shm (RAM), where fsync is free — so the §1/§3 batching win is neutralized and only raw compute remains. bytes_read ≈ 3.92 GiB both sides (the 38 KiB files are below the 1 MiB window, so bounded reads don't help).

Finding — direction inverted vs the historical file. The old file (6-core EPYC) recorded SP1 as ~1.9× slower on RAM — the deliberate "honest cost" of the pipeline where there is no fsync win to amortize. On this 8-core box the parallel pipeline is ~1.4× faster even on RAM (the extra cores outweigh the per-file coordination), at the cost of ~3.4× peak RSS (96 MB vs 28 MB). The trade has shifted from "small RAM loss" to "RAM win for more memory" on wider hardware.

# durable §1/§2: MUSEFS_BENCH_DIR on /data ; RAM §4: MUSEFS_BENCH_DIR on /dev/shm
MUSEFS_BENCH_TIER=ci MUSEFS_BENCH_DIR=/data/bench \
  cargo test --release -p musefs-core --features metrics --test bench_ingest \
  -- --ignored --nocapture bench_cold_scan_and_revalidate
# §3 fsync count:
MUSEFS_BENCH_LATENCY_PROFILE=ssd MUSEFS_BENCH_TIER=ci MUSEFS_BENCH_FORMAT_MIX=flac \
  cargo test --release -p musefs-core --features metrics --test bench_ingest \
  bench_scan_under_latency -- --ignored --nocapture

SP2 — Incremental tree refresh

ed5f380^ → ed5f380. bench_refresh, RAM (TMPDIR=/dev/shm). Overlay: the bench_refresh_one_across_library_sizes sweep didn't exist at ed5f380^, so the after-commit harness is overlaid on the before checkout.

What changed: replace the O(N) VirtualTree::build_with full reconstruction with apply_changes (in-place im-backed tree mutation) — only nodes whose id appears in the changed/added/removed sets are touched.

ci tier, FLAC, single-track re-tag, 3 runs (median):

Why (Stage A → Stage B): at Stage A the rebuild already rendered incrementally (only the changed track re-rendered, O(changed)), but the subsequent VirtualTree::build_with reconstructed the whole tree from scratch (O(N)) — the remaining linear cost. Stage B's apply_changes removes that full reconstruction; the residual slope (still ~23 ms at 5000) is the lighter O(N) render-key scan + HashMap rebuild that feeds apply_changes, not a full tree rebuild. The speedup grows with library size because diff cost is proportional to changes, not total entries. (Corpus is single-album, so build_with time is slightly optimistic vs a real multi-album library.)

cargo test -p musefs-core --release --test bench_refresh \
  bench_refresh_one_across_library_sizes -- --ignored --nocapture

SP3 — Read/serve residuals

e8d56bd^ → e8d56bd. Criterion read_throughput, RAM. No overlay.

What changed: (1) read_segments writes each BackingAudio run directly into the output buffer's reserved tail (no throwaway vec![0u8; n] + copy); (2) handles: Mutex<HashMap> → lock-free sharded_slab::Slab; (3) size_cache: Mutex<HashMap> → dashmap::DashMap.

sequential_read — per-format (4 MiB files, 128 KiB reads)

The metadata-light formats improve 8–13% from dropping the per-splice alloc+copy. ogg +6.3% is a low-iteration sampling anomaly (Criterion warned "Unable to complete 100 samples in 5.0s" — only 5050 iterations vs 10k for other formats).

concurrent_read_walk/m16_plus_walker

16 reader threads + one metadata walker sharing one Arc<Musefs> (includes thread spawn/join):

This high-variance burst metric regressed on this run — attributable to thread spawn/join overhead in the contention path rather than the read path itself; it is not a sequential-read regression. (The old file recorded this bench as parity/improved; it swings run-to-run.)

cargo bench -p musefs-core --bench read_throughput -- sequential_read concurrent_read_walk

SP4 — Storage-aware Ogg serving

a62453b^ → a62453b. Criterion read_throughput + latency-injected read. Overlay: cold_first_read/seek_read were added by SP4, so the after-commit bench is overlaid on the before checkout.

What changed: replace the eager whole-region Ogg page index with a stateless per-request backwards-scan: find_page_start locates the containing page from a ~65 KB window (CRC-validated entry guard), serve_ogg_window patches each page header algebraically (crc_shift_zeros, no payload I/O), and a one-entry last_page memo short-circuits the scan + CRC guard when the next request lands inside the already-located page.

sequential_read — warm repeat-read (no page-index amortization to win)

cold_first_read / seek_read — the Ogg win

Non-ogg cold/seek stay within ±7% (no page index involved). The wins come from never building the whole-file index up front — the old code reads the entire prefix to serve even one chunk near EOF; SP4 scans ~65 KB backward, then the memo carries the validated page forward. sequential_read/ogg is flat (+1.0%) because it reads the full file linearly regardless — the win is cold-start and seek.

Latency-injected reads (bench_read_under_latency, nfs-hdd) — AFTER only

This bench was introduced by SP4; no before baseline exists.

preads=0: the backwards-scan reads are served from the layout's inline/generated segments without reaching the backing file. Near-equal whole/seek wall time indicates per-file open+resolve latency dominates under nfs-hdd; the local cold/seek benches above are the clean signal.

Why crc_shift_zeros shipped as a hybrid (superseded by #666)

patch_page_header_algebraic advances the CRC past a page's payload via crc_shift_zeros. The per-step loop is O(n) and dominated linear sequential_read on max-size 65 KB pages; the GF(2) matrix-power method as SP4 wrote it is O(log n) but rebuilt and re-squared the base matrix on every call, so it was slower for the small pages real Opus/Vorbis streams carry. The evolution across implementations (ogg benches):

SP4 shipped the hybrid: per-step loop below n=16384, matrix at/above, with a differential test over both paths and the boundary. #666 later removed the threshold by hoisting the squarings into a const, which makes the matrix win at every n.

cargo bench -p musefs-core --bench read_throughput -- cold_first_read seek_read sequential_read
MUSEFS_BENCH_LATENCY_PROFILE=nfs-hdd cargo test --release -p musefs-core \
  --features metrics --test bench_ingest bench_read_under_latency -- --ignored --nocapture

#69 — Refresh O(changed)

e7ae912^ → e7ae912. bench_refresh, RAM. No overlay.

What changed: changelog-driven change detection (changelog_since + render_keys_for on just the changed ids) replaces the O(N) render-key scan, and collision-gated apply_changes dirtying stops the old parent chain from being rebuilt unconditionally. Refresh-1 cost becomes O(changed).

Single-track refresh vs library size (3 runs, median)

A single-track re-tag moves the track out of its shared album dir — the structural worst case for a flat corpus (one artist / one album, N siblings).

The after sweep is flat: refresh-1 @ 20 000 is within 1 ms of @ 100, against a linear ~170 ms slope before.

One-vs-many (same Musefs instance, 200-track ci tier)

refresh-N scales with the touched set, not the library.

# before (apply the 4-point sweep edit first):
sed -i 's/\[100usize, 1000, 5000\]/[100usize, 1000, 5000, 20000]/' musefs-core/tests/bench_refresh.rs
cargo test -p musefs-core --release --test bench_refresh \
  bench_refresh_one_across_library_sizes -- --ignored --nocapture
cargo test -p musefs-core --release --test bench_refresh \
  bench_refresh_one_vs_many -- --ignored --nocapture

#114 — Rendered child lookup (root fan-out)

0881b31^ → 0881b31. bench_refresh, RAM. Overlay: the bench_refresh_root_fanout_one_across_library_sizes bench was added by #114, so its harness is overlaid on the before checkout.

What changed: a rendered-name child index turns the root sibling scan in deepest_existing_ancestor into an indexed miss. The corpus uses N top-level artist directories; the timed update retags one track to fallback Unknown/…, exercising an absent rendered-name lookup at root.

~5× at the 20 000-artist fan-out; ≤5 k is already ≤2 ms on both sides.

cargo test -p musefs-core --release --test bench_refresh \
  bench_refresh_root_fanout_one_across_library_sizes -- --ignored --nocapture

PR2 — Scan pair (#67/#68)

2d4faf3^ → 2d4faf3. bench_ingest, --features metrics, RAM, 3 runs. No overlay.

What changed: (#67) gate the 128-byte ID3v1 tail read to .mp3 files — only MP3 consumes the frame; (#68) ingest_bulk drains the owned Unit batch by value, moving picture payloads into the DB structs instead of cloning.

Wall time — ci tier (200 tracks × 4 KiB, no art), median of 3

Wall time is within run-to-run noise — at ci tier (4 KiB files, no embedded art) there is no picture payload to move, so #68's win doesn't show here. It appears on art-bearing corpora (the bandwidth tier / real libraries) where the clone was O(art-size) per file.

Scan I/O — the #67 signal (scan_bytes_read)

Non-MP3 formats drop exactly the 128-byte ID3v1 tail per file (−25 600 B over the 200-track corpus). MP3 is unchanged; M4A uses the seek-reader, not the front-anchored probe path.

MUSEFS_BENCH_TIER=ci MUSEFS_BENCH_DIR=/dev/shm/bench \
  cargo test -p musefs-core --release --features metrics --test bench_ingest \
  -- --ignored --nocapture bench_cold_scan_and_revalidate

PR3 — Serve-path copies (#70)

32be8f0^ → 32be8f0. Criterion read_throughput, RAM. No overlay.

What changed: four stacked serve-path copy eliminations — DB chunk readers fill the caller's &mut [u8]; read_segments writes ArtImage/BinaryTag/raw OggArtSlice arms into the output buffer's resized tail; Musefs::read_into serves into a caller buffer; and the FUSE layer reuses a per-worker thread-local scratch buffer. None touches synthesis or layout (served audio stays byte-identical).

sequential_read

cold_first_read / seek_read / concurrent

No format breaches the >10% rise gate. The concurrent burst metric improves 19% here (it is high-variance and swings run-to-run; see SP3).

cargo bench -p musefs-core --bench read_throughput -- \
  sequential_read concurrent_read_walk cold_first_read seek_read

#136 — HeaderCache → quick_cache

2e6674e^ → 2e6674e. Criterion read_throughput, RAM. No overlay.

What changed: an S3-FIFO byte-weighted quick_cache replaces the hand-rolled 16-shard Mutex LRU — the serve path's last shared std lock is gone.

At a glance: within noise. No workload regresses outside noise; the only movers are marginal sequential_read improvements on the metadata-light formats.

cargo bench -p musefs-core --bench read_throughput

#112 — StructureOnly kernel passthrough

0881b31 → faec017. Bespoke dd harness (committed: benches/passthrough_dd.sh), sudo (passthrough needs CAP_SYS_ADMIN).

What changed: the backing fd is registered at open (FUSE passthrough, kernel ≥6.9); the kernel serves StructureOnly reads directly from the backing inode, bypassing the daemon round-trip.

512 MiB WAV backing on /dev/shm (RAM-cached, isolates FUSE-path overhead), dd bs=1M sequential read, fresh mount per binary, 3 runs each:

3.36× on this RAM-cached sequential workload: the before path round-trips every ~128 KiB chunk through the daemon (wakeup + positioned read + copy back via /dev/fuse); the after path reads straight from the backing inode's page cache.

sudo benches/passthrough_dd.sh target/release/musefs /dev/shm/pt 512

Cumulative detail

16caba4 → current main (e02223e). Derived, non-isolating — composed from the per-pass isolated deltas above, anchored to current-main absolutes. A same-harness end-to-end run is infeasible: MountConfig.case_insensitive and scan_directory_with/ScanOptions/revalidate_with don't exist at 16caba4 (so main's harnesses can't compile there), and the 16caba4-era harness omits the now-required case_insensitive field (so it can't compile on main either). The deltas below name the contributing passes and the dominant one; unrelated speedups are not multiplied into a single headline.

Current-main absolutes (1 run, native harness)

Ingest — ci tier, /data, bench_ingest:

Refresh — RAM, bench_refresh_one_across_library_sizes: refresh-1 @ 100 / 1000 / 5000 = 0 ms; @ 20 000 = 1 ms.

Serve — RAM, read_throughput (Criterion median): sequential_read flac 569 µs · mp3 563 µs · m4a 566 µs · m4a-last 568 µs · ogg 737 µs · wav 598 µs; cold_first_read ogg 1.507 ms; seek_read ogg 806 µs; concurrent m16+walker 4.15 ms.

Composed per-subsystem deltas

Ingest = SP1 ∘ PR2. Dominated by SP1's durable-fsync elimination; PR2 is the −128 B/file + move-not-clone refinement.

† Bandwidth tier not re-measured at main; figure is SP1's after number.

Refresh = SP2 ∘ #69 ∘ #114. The O(N)→flat journey; dominant pass is #69 (changelog-driven O(changed) rebuild), with #114 shaving the 20 k root fan-out on top.

Serve = SP3 ∘ SP4 ∘ PR3 ∘ #136. SP3 + PR3 drive the cross-format sequential/cold/seek wins (alloc elimination + copy reduction); SP4 owns the ogg cold/seek collapse.

Criterion's own change: lines compare against the previous on-machine baseline (itself already optimized); the absolutes above are the reliable end-to-end signal.

Storage tunables

A proposed --storage-profile {ssd,hdd,nfs} preset would have bumped --max-readahead-kib and --max-background (and enabled --keep-cache) per medium, on the premise that "larger read-ahead hides HDD/NFS latency." Measured against real storage, that premise does not hold — only --keep-cache shows a benefit — so the preset was dropped and these flags keep their defaults. This section records the evidence.

Methodology

Unlike the optimization passes above (tmpfs, in-process Criterion), these run through a real kernel mount with a real reader, because the tunables are kernel↔FUSE negotiation parameters invisible to an in-process driver:

• Backing: real RAID-1 HDD (/home, /dev/md127) and a btrfs HDD span (/data, /dev/sda3); for NFS, a loopback NFSv4.2 export (exportfs + mount -t nfs localhost:…) whose backing is tmpfs (isolates the RPC tax) or HDD (RPC + seeks).
• Latency: tc qdisc add dev lo root netem delay <X>ms adds X per packet → ≈2X RTT per NFS RPC. Tested at 8 ms, 50 ms, and 200 ms RTT (the last ≈ a trans-Pacific server).
• Cold reads: sync; echo 3 > /proc/sys/vm/drop_caches before each measured read — without it the page cache serves repeats and hides all backing latency.
• Mode: synthesis, not structure-only. Structure-only triggers kernel FUSE passthrough when the process is privileged (these run as root), which serves the backing fd directly and bypasses the daemon read path — and with it every tunable that acts on that path. Synthesis splices BackingAudio reads through the daemon, the real serving path.
• Why not the injected MUSEFS_FAULT_*_US model: it cannot show a read-ahead effect. FUSE delivers reads to the daemon in fixed ≤256 KiB chunks (max_pages, already pinned at the kernel's 1 MiB ceiling by fuser's 16 MiB default max_write), so the per-pread count — and thus any per-pread injected latency total — is independent of max_readahead.

Reproduce: benches/storage_tunables_bench.sh (needs /dev/fuse, root, and for the NFS rows nfs-kernel-server + tc). HDD numbers are noisy (±10–15%); the trends, not the digits, are the signal.

--max-readahead-kib — no benefit anywhere; hurts on HDD

Cold single-stream sequential throughput (MB/s), synthesis:

(File sizes differ per column — 512 MiB local, 96 MiB at 50 ms, 48 MiB at 200 ms — so compare within a column, not across. The 200 ms column ≈ a trans-Pacific server: flat to the last digit.)

The window size barely moves throughput, and on HDD values ≥2048 KiB are among the slowest (peak is ~128–512 KiB). The reason is visible on NFS: 512 MiB ÷ 256 KiB × 8 ms ≈ 16 s ≈ the observed 31 MB/s — a single stream is served serially, one ≤256 KiB read at a time, each paying the full RTT, with no prefetch overlap that a larger window could exploit.

--max-background — no effect on read throughput

Wall time (s) for N concurrent cold streams over distinct tracks:

64 ≈ 128 even with 80 > 64 streams. Expected: musefs's FuseConfig notes max_background caps background work and that "foreground reads are bounded only by client concurrency, not by this." The concurrent reads here are foreground. (Concurrency does hide latency — 16 NFS streams reach ~10× single-stream aggregate — but that is client parallelism, which max_background does not gate.)

--keep-cache — the one real win (~3×)

Cold read then immediate reopen (no cache drop between); reopen_s is the signal:

With --keep-cache the kernel retains the page cache across opens, so a re-opened file is served from RAM instead of re-fetched over slow storage — ~3× faster reopen, consistent across HDD and NFS. This is the only tunable worth changing for slow backing (relevant for players/scanners that re-open files), and it needs no preset. It is on by default as of #432 (inode invalidation on retag keeps it consistent); disable with --keep-cache false on memory-constrained hosts.

Conclusions

• Drop the --storage-profile preset. Of the four knobs it would have set, three (max_readahead, max_background, and by extension a per-medium combination of them) show no benefit; max_readahead ≥2048 KiB actively hurts on HDD. The only justified change — enable --keep-cache on HDD/NFS — does not need an abstraction.
• Single-stream latency hiding — addressed in #255 (next section). The serialized read path measured above (512 MiB ÷ 256 KiB × RTT) is exactly what backing read-ahead now fixes.

Backing read-ahead (#255)

Each --max-readahead-kib row above exposed the real bottleneck: a single stream is served one ≤256 KiB FUSE chunk at a time, each paying the full backing RTT, so a 200 ms-RTT NFS mount tops out at ~1.3 MB/s regardless of the kernel read-ahead window. The fix is read amplification in the daemon — BackingReader coalesces a stream's small reads into one large positioned pread (geometric window growth, global RAM budget with LRU eviction), so the backing client can pipeline/parallelize the RPCs behind one syscall. A background-prefetch-threads layer ("Phase 2") was also built and is off by default — originally because it measured as pure overhead, which #671 later showed was a bug in the prefetcher rather than a property of the design.

Methodology

Two harnesses. Real kernel mount (benches/storage_tunables_bench.sh): a real reader (dd) over a real FUSE mount, cold (drop_caches) each sample, median of 3. Local backing on a btrfs HDD; NFS via a loopback NFSv4.2 export plus tc netem for RTT. The corpus is real FLAC (MUSEFS_BENCH_CORPUS_SRC) — a /dev/zero corpus on a compressing fs (btrfs compress=zstd) collapses to a cached extent and never touches the platter, which silently inverts the HDD numbers; real already-compressed audio is incompressible. Loopback NFS needs LOCALIO off on Linux 6.12+. The client and server are the same host, so the kernel negotiates NFS LOCALIO and does local I/O instead of RPC — tc netem on lo then never touches the data path and every "NFS" row silently measures local disk. The tell is cold reads at GB/s while ping 127.0.0.1 shows the injected RTT. The bench now writes N to /sys/module/nfs/parameters/localio_enabled before mounting (restored on exit) and prints localio= in its header line; numbers taken on an older kernel predate the feature. In-process (musefs-core/tests/bench_ingest.rs::bench_read_under_latency): the core read path over musefs-latencyfs (per-op injected latency), isolating the daemon from the kernel FUSE layer. off = --read-ahead-budget-mib 0; phase1 = the default (amplification only); phase1+2 = --read-ahead-prefetch.

Single-stream cold throughput (MB/s)

On NFS read-ahead is a ~6× single-stream win (1.2 → 7.4 MB/s, 75 % of the kernel-passthrough ceiling). On a real local HDD all four configs sit within run-to-run noise (~±15 %) — read-ahead is neutral, not a regression. (An earlier /dev/zero corpus showed a spurious −35 %; it was the zstd-compression artifact above, not read-ahead.)

Concurrent streams (8 × distinct tracks, aggregate MB/s, NFS 200 ms RTT)

In-process, per-op latency (16 MiB Ogg whole read; wall ms / backing preads)

Amplification collapses 774 backing round-trips to 32; the win scales with per-op latency and is already material at SSD speeds (1.7×).

Phase 2 is off by default

Superseded in part by #671. The numbers below stand as measured; the cause attributed to them was a bug, and Phase 2 is a win on high-RTT backing once it is fixed. It remains off by default — see that section for when to turn it on, and this one for what was originally measured.

As measured before #671, background prefetch threads (Phase 2) never beat amplification alone and cost a consistent ~10 %: single-stream NFS 6.8 vs 7.4, concurrent NFS 12.1 vs 13.6, neutral on HDD. The reading at the time was that a single large pread already lets the NFS client pipeline its RPCs, leaving the threads coordination overhead and no overlap to exploit. That explanation was wrong: the prefetcher was evicting the window the reader was inside and re-dispatching windows already in flight, and the ~10 % was the cost of that. Re-measured on the same shape of backing, Phase 2 is now ~30 % faster than amplification alone on a single stream. It stays opt-in (--read-ahead-prefetch) either way — for the reasons in #671, not the ones in this paragraph.

Defaults: read-ahead on at --read-ahead-budget-mib 64, Phase-1 amplification only. Set 0 to disable on local-disk-only setups (no benefit there, though no harm either).

Phase 2 was also amplifying, not just overhead (#671)

The "~10 % overhead" above understated it. Two defects, both reachable only with --read-ahead-prefetch, turned the prefetcher into read amplification:

1. Eviction dropped the window the reader was inside. The ring trimmed to the first window lying fully behind the read frontier and fell back to index 0 — the lowest start, which is the window serving the reader — when nothing was fully consumed yet. next_expected is the end of the last served read, not of the window it came from, so a 512 KiB window serving 128 KiB reads is never fully behind until the reader has consumed all of it. Every completed prefetch therefore evicted the window in use.
2. A legitimate watermark overshoot was read as a seek. A dispatch stops at the first window boundary at or past the horizon, so prefetched_upto normally sits beyond it. plan_prefetch treated that as a backward seek and re-dispatched the whole horizon from the reader's position on the very next read.

Defect 2 was masked by defect 1 (the ring never held the windows that trigger it), so fixing the eviction order alone made real-mount prefetch worse: 9.9× the backing bytes. Both are fixed together.

Measured, in-process over musefs-latencyfs (32 MiB FLAC, whole-file read in 128 KiB chunks, --read-ahead-budget-mib 64; serve-path backing bytes, i.e. foreground refills):

Measured on a real kernel mount (866 MiB FLAC from a real 4290-track library, daemon bytes from /proc/<pid>/io so the prefetch threads are counted):

A synthetic drive loop over the real ReadAhead (400 sequential 128 KiB reads, 256 MiB backing, 8 MiB cap) puts a number on the eviction defect alone: 398 of 400 reads missed and refilled synchronously, 3159 MiB read for 50 MiB asked. After the fix: 1 miss, 0.5 MiB.

Measured on a real NFS mount (loopback NFSv4.2, tc netem 100 ms/way = 200 ms RTT, LOCALIO disabled, real FLAC corpus, cold each sample, median of 3 — benches/storage_tunables_bench.sh prefetch):

This changes the Phase-2 story on high-RTT backends. The earlier finding that the threads cost a consistent ~10 % was measuring the amplification: with it fixed, prefetch is a ~30 % single-stream win at 200 ms RTT (6.3 → 8.2 MB/s) and a ~5 % win on four concurrent streams, where before it was a regression on both.

Phase 2 nevertheless stays off by default for now. The win is one backend and one run, and the other harnesses still show it doing redundant work: over musefs-latencyfs and on a local-disk kernel mount it reads the stream a second time speculatively (≈2× total backing bytes) for wall time within noise of Phase 1 alone. Enable it on a high-latency network backend, where the overlap it buys has something to hide. musefs_readahead_prefetch_reads_total / _bytes_total report that speculative volume, which is what made the defect measurable — the serve-path musefs_backing_pread_* counters never saw the prefetch threads.

Reproduce (needs root for exportfs/mount/tc/drop_caches; build a binary from before the fix to fill the "before" rows):

MUSEFS_BENCH_CORPUS_SRC=<music-tree> MUSEFS_BENCH_CORPUS_MAX_MIB=60 \
MUSEFS_PREFETCH_BINS="baseline=<old-musefs> fixed=<new-musefs>" \
  benches/storage_tunables_bench.sh prefetch <export-dir> 100 96 4

Internal window cap on HDD (#433)

The amplification window doubles per sequential read up to WINDOW_ABS_CAP (8 MiB, musefs-core/src/readahead.rs). The #256 sweep above measured the kernel max_readahead knob — where ≥2048 KiB hurts on HDD — but never this daemon-internal cap, so #433 asked whether 8 MiB is too large for spinning media.

Methodology. WINDOW_ABS_CAP is a compile-time const, so the sweep builds one release binary per value (benches/storage_tunables_bench.sh window-cap, which patches the const in place and restores it after each build). Cold (drop_caches) single-stream reads of the same ~270 MiB real FLAC, synthesis mount, default flags (amplification on, prefetch off), real backing on a btrfs HDD (/data, 4389-track corpus). Reproduce:

WINDOW_CAP_MIB="1 2 4 8 16" MUSEFS_BENCH_CORPUS_SRC=<music-tree> MUSEFS_BENCH_CORPUS_MAX_MIB=300 \
  benches/storage_tunables_bench.sh window-cap <hdd-backing-dir>

Result: no measurable cap effect — the medium's noise dominates. Median MB/s (and the within-cap min–max over 7 cold samples) overlap across every cap, and the apparent ordering is an artifact of measurement order, not the cap: throughput drifts down through each run, so whatever runs first looks fastest. Sweeping the caps in the reverse order reverses the "trend".

The current default (8 MiB) lands at ~61 MB/s in both orderings; the first-measured cap is fastest in both (104 for cap 1 ascending, 86 for cap 16 descending). The within-cap spread (≈50–140 MB/s) dwarfs every between-cap median gap. This corroborates the #256 finding that backing read-ahead is neutral on local HDD.

Decision: keep 8 MiB, no runtime knob. There is no HDD gain to capture, and the cap exists for the proven case — the ~6× single-stream amplification win on high-RTT NFS/remote, where coalescing into one large pread lets the client pipeline RPCs. Lowering the cap to chase an unmeasurable HDD effect would regress that win.

Global allocator — steady-state RSS (#360)

Long-lived high-churn FUSE load fragments glibc malloc, growing daemon RSS over days without a true leak. The musefs binary now defaults to the jemalloc global allocator with a background purge thread. Measured with scripts/rss-churn-bench.sh (Linux; median VmRSS over the flattened tail — steady state, not peak).

Parameters: WORKERS=8 (nproc), FILES=500, CYCLES=200, WARMUP=20, no REFRESH_CMD. DB = a freshly-scanned 4427-track store on tmpfs (/tmp); backing audio on /data (HDD). Concurrent cat-to-/dev/null churn drives the open/read/release handle-table and read-synthesis allocation path.

Decision: SHIP jemalloc. Steady-state RSS is ~62% lower (jemalloc ≤ system malloc, the §4 ship rule). Under identical churn glibc retained ~46 MiB of dirty pages that jemalloc's decay + background purge return to the OS — the #360 fragmentation failure mode, reproduced and fixed. The gap is far outside run-to-run noise, so no within-noise tie-break was needed.

Scan fingerprint overhead (#464)

Bench: cargo bench -p musefs-core --bench fingerprint_overhead Corpus: 200 minimal FLAC files (~200 B metadata + 4 KiB audio) in tempfile::tempdir() (TMPDIR = tmpfs/RAM). Single-threaded scan (jobs: 1). Criterion, 20 samples.

Delta: +60.6 ms / 200 files = +303 µs/file overhead (+129%).

Interpretation: The 129% overhead on this synthetic RAM-backed bench exceeds the plan's ≤15% threshold. The overhead is dominated by the extra UPDATE tracks SET fingerprint = … SQLite execution per file inside the batch transaction — not by SHA-256 hashing cost (SHA-256 of a few hundred bytes is sub-microsecond). On a real HDD-backed library the probe I/O (tens-of-ms per file) is the bottleneck, making both the hash and the DB write negligible. See plan Task E2 step 3 decision note: the decision to keep SHA-256 and add the length CHECK was escalated to the controller because the raw percentage exceeded the stated threshold, even though the absolute overhead (303 µs/file) is operationally negligible at disk I/O rates.

Scan fingerprint overhead — SSD latency profile (#464)

Bench: bench_scan_under_latency in musefs-core/tests/bench_ingest.rs (MUSEFS_BENCH_LATENCY_PROFILE=ssd). Corpus: 200 minimal FLAC files (~200 B metadata + 4 KiB audio) on a musefs-latencyfs SSD-latency FUSE mount. Default thread count (jobs: 0). 3 runs each, median reported.

Delta: +19 ms / 200 files = +95 µs/file overhead (+8.6%).

Interpretation: Under an SSD latency profile the I/O dominates and the fingerprint overhead drops to +8.6% (+95 µs/file), well within the plan's ≤15% threshold. The RAM bench's +129% (+303 µs/file) was an artefact of RAM eliminating the I/O that would normally dwarf the extra SHA-256 hash and DB write. At real SSD rates the fingerprint cost is operationally negligible.

#666 — Precomputed CRC matrix powers

Box: not the reference 8-core machine of Methodology — a 16-core/32-thread desktop, rustc 1.97, release. Before/after run back to back on that one box, so the deltas are internally comparable; the absolutes are not comparable with the per-pass sections above.

What changed: crc_shift_zeros (musefs-format/src/ogg/crc.rs) dropped its MATRIX_THRESHOLD hybrid. The power-of-two GF(2) transition matrices are now a compile-time const, so a shift by n costs n.count_ones() matrix applies at every n — no per-byte loop below the threshold and no per-call re-squaring above it.

The function itself (standalone harness, 20 000 calls per point)

The discontinuity between 8 192 and 16 384 on the before side is the old threshold: below it the per-step loop, at/above it the re-squared matrix. 1 890 B is the mean page size of ffmpeg/libvorbis output; 65 025 B is a max-size page.

read_throughput — the new ogg-vorbis fixture

Overlay run: the fixture landed with this change, so the before side is the old crc_shift_zeros measured with the new harness.

All three at p < 0.05.

Why the pre-existing ogg (Opus) fixture shows nothing

Those deltas are contended-box noise, not the change. Instrumenting patch_page_header_algebraic shows every call against that fixture passes crc32(DELTA) == 0 and takes the zero-state early return, so the shift costs nothing on either side. Two properties of write_ogg cause it, and write_ogg_vorbis was added to cover both:

• Page size. write_ogg laces a whole track as one packet, so every page is max-size (trailing = 65 285). The CRC advance is per-page and scales with payload length, so that measures a different regime from the ~1 890 B pages real encoders emit.
• Renumbering. RFC 7845 puts each Opus header packet on its own page, which is also how synthesis lays them out — so the served header matches the original, no audio page is renumbered, and every DELTA is all-zero. A real Vorbis encoder instead packs the comment and setup headers together, so synthesis always lengthens the header and every audio page shifts. That is the case the algebraic patch exists for, and it is now a fixture.

The two properties are pinned by write_ogg_vorbis_uses_encoder_realistic_page_sizes and write_ogg_vorbis_renumbers_every_audio_page_on_serve, and the format carries its own row in the perf_counters golden gate.

cargo bench -p musefs-core --bench read_throughput -- ogg

Release notes

Curated, upgrade-focused notes for each release. For the exhaustive, per-change list see the Changelog; for the external-writer contrib/ packages (which version independently) see the contrib changelog.

v1.3.0

A compatibility and maintenance release. The headline change makes musefs ingest FLAC files that carry a leading ID3 tag instead of rejecting them, and the dependency tree is clear of open RUSTSEC advisories. The on-disk schema is unchanged (still version 2) and no CLI flag or default moved, so upgrading is a drop-in — but a re-scan is worth running, see Upgrading from v1.2.0.

Highlights

• FLAC with a leading ID3 tag now scans (#602). Some FLAC files put one or more ID3v2 tags in front of the fLaC marker — non-standard, but common in the wild, often a blank header left behind by a converter. musefs used to reject these with no parseable audio metadata, counting them failed; they now parse normally.
• Their ID3 tags come with them. Text frames and embedded cover art in the ID3 header are ingested as a fallback: the FLAC's own VORBIS_COMMENT and PICTURE blocks win, and the ID3 tag only supplies what the FLAC itself does not carry. A FLAC whose tags live only in its ID3 header therefore arrives tagged rather than blank. The served file is a stock FLAC with no ID3 tag — the tag is metadata, and musefs regenerates metadata rather than copying it.
• No open security advisories. musefs-core's persistent collections moved from the archived im crate to the maintained imbl fork, clearing two unsoundness advisories along with the unmaintained ones, and a crossbeam-epoch vulnerability was patched. Both lockfiles were refreshed.

See the Changelog for the full list.

Upgrading from v1.2.0

No schema migration. The store stays at user_version 2, and you can roll back to v1.2.0 without touching it.

Re-scan to pick up previously-rejected files. FLACs with a leading ID3 tag were never added to the store, so a bare musefs scan <dir> ingests them now — no --force needed, and files already tracked are left alone. If a scripted pipeline was exiting 2 because of these files (failed Y with Y > 0), it will now succeed.

musefs-core API. VirtualTree::children returns an opaque impl ExactSizeIterator<Item = (&str, u64)> instead of borrowing the backing OrdMap. This only affects code depending on the musefs-core crate directly; callers that iterated are unaffected apart from the item type, and by-name lookups have always had VirtualTree::lookup. The change takes the persistent-collection crate out of the public API, so swapping it again cannot break a downstream consumer.

External writers. The contrib/ packages are unchanged this cycle; no update is needed alongside musefs.

v1.2.0

A behavior-focused release: the headline change makes scan non-destructive by default. The on-disk schema is unchanged (still version 2), so there is no migration — but a default that previously overwrote curated metadata has been removed, so read Upgrading from v1.1.0 before you update any automation that calls scan.

Highlights

• scan is now additive. A bare musefs scan ingests only files not already in the store and never overwrites curated tags or art. Re-running it to pick up newly-added files is safe; the old full-reimport-from-disk behavior is now opt-in via scan --force. This closes a footgun where a routine re-scan silently clobbered tags written by Picard / beets / the store.
• revalidate is its own subcommand. Promoted from the scan --revalidate flag, it refreshes the structural/serving facts (audio bounds, checksums, FLAC structural blocks) of in-store files whose backing bytes changed, while preserving curated tags, art, and binary tags. Files not yet in the store are ignored — that is scan's job.
• Deletion is opt-in. revalidate no longer prunes by default; pass revalidate --prune to drop rows whose backing file is gone and garbage-collect orphaned art. No bare command deletes or overwrites store data.
• Data-loss fix. Revalidating a changed file no longer clobbers curated tags, art, or binary tags — previously the structural/checksum backfill routed through a full re-seed from disk.

See the Changelog for the full list.

Upgrading from v1.1.0

No schema migration. The store stays at user_version 2; nothing about the on-disk format changes, and you can roll back to v1.1.0 without touching it.

Behavior changes to check:

• Bare scan no longer re-imports existing tracks. Automation that runs musefs scan <dir> expecting it to refresh tags on files already in the store now skips them (reported as already present). To re-seed curated metadata from disk, use scan --force. The common case — re-running scan to pick up newly-added files — is unchanged.
• scan --revalidate is deprecated. It still works but prints a warning and forwards to the non-pruning revalidate path; it will be removed next release. Switch to the musefs revalidate subcommand — and note it no longer prunes, so add --prune for the old delete-gone-rows-and-GC behavior.
• revalidate does not prune by default. Scripted maintenance that relied on scan --revalidate to remove rows for deleted files must now pass revalidate --prune explicitly.

External writers. The contrib/ packages are bumped to 1.2.0; the beets and Picard plugins are updated for the new CLI — the beets plugin's autoscan now resets the store with scan --force, and beet musefs --revalidate maps to revalidate --prune (both transparent to you). Update these packages alongside the binary; older copies invoke the deprecated scan --revalidate alias, which still works for this release.

v1.1.0

A feature-and-hardening release on top of the v1.0.0 stable line. No CLI flags or store columns were removed, but the on-disk schema steps to version 2 and a few defaults change observable behavior — read Upgrading from v1.0.0 before you update an existing store.

Highlights

• Runtime telemetry. An opt-in --expose-metrics (env MUSEFS_EXPOSE_METRICS) surfaces a synthetic .musefs-metrics/ directory at the mount root whose metrics file renders Prometheus-format counters for getattr/read/open activity, backing read-ahead behavior, and (with the jemalloc build) allocator stats. Off by default. See Tuning & metrics.
• Scan progress indicator. scan and scan --revalidate render a live progress bar on an interactive terminal and fall back to periodic ingested N/M (P%) lines when output is redirected. A new --quiet/-q suppresses it.
• --skip-on-missing template flag. Opt-in (env MUSEFS_SKIP_ON_MISSING): drops a track from the mount when a top-level template field stays unresolved, instead of substituting --default-fallback. The motivating case is --template '$!{beets_path}' --skip-on-missing, hiding tracks beets left without a beets_path rather than collapsing them into an Unknown bucket.
• --read-ahead-prefetch flag. Opt-in background prefetch threads layered on read amplification, default off — benchmarks found amplification alone delivers the read-ahead win, so enable this only when profiling a backend where a single large read does not self-pipeline.
• riscv64 release platform. Prebuilt riscv64gc-unknown-linux-{gnu,musl} binaries and linux/riscv64 Docker images now ship with each tagged release. Container bases moved to current stable (Debian trixie, Alpine 3.23).
• statfs reply. The mount now reports a synthetic non-zero capacity with ample free space, so df no longer shows a 0-byte filesystem and capacity-checking importers (Lidarr et al.) no longer balk.
• Per-extension skip breakdown. End-of-scan summary breaks the skipped count down by lowercased extension (e.g. skipped 42: jpg=20, cue=10, log=8) so a large skip count is diagnosable. Log-only; the counters are unchanged.
• musefs vacuum. A maintenance command that compacts the SQLite store — reclaiming the free pages that prunes, orphan-art GC, and the migration leave behind — and reports the space reclaimed. Run it while unmounted. See Maintenance.

Plus a substantial round of correctness and robustness fixes across the read fast path (rowid-reuse consistency for art segments), the MP4/QuickTime metadata walk, ID3 synthesis, and the prune/delete paths — see the Changelog for the full list.

Upgrading from v1.0.0

1. Back up your store. The schema migration below is one-way. While no scan or external writer is touching the database, copy musefs.db (and its -wal / -shm sidecars if present). A v1.0.0 binary has no guard against a newer store and may misread one that has been migrated, so keep the backup if you might roll back. From v1.1.0 onward a binary instead refuses to open a store whose schema is newer than it understands, with a clear error.

2. Automatic schema migration (user_version 1 → 2). The first time a v1.1.0 binary opens the store — for example musefs scan — it migrates in a single transaction. The migration:

• Adds scanner-owned tracks.fingerprint and tracks.content_hash columns (nullable SHA-256 hex, non-unique by design) plus a fingerprint index. They start NULL and are populated on the next scan; external writers do not set them.
• Rebuilds the tags table so the 256 KiB value cap counts bytes rather than characters (the v1 CHECK was up to ~4× looser for multibyte text). Any row that was already over the byte cap is dropped in the rebuild (this only reaches genuinely pathological data — a single tag value larger than 256 KiB of bytes, which a real library never has, and such rows were already unreadable under the byte-counting read guard anyway; in practice no store is affected).

The migration applies automatically the first time a v1.1.0 binary opens the store, but you should still run musefs scan --db <store> once after upgrading: that is what populates the new fingerprint / content_hash columns, which the scanner's content-identity refind logic relies on. Then remount. See The SQLite store for the full schema contract.

3. Behavior changes to check.

• scan exit code. scan/scan --revalidate now exit 2 when any file fails to parse or ingest (previously always 0 on a non-fatal run). A clean scan still exits 0; a hard error still exits 1. Pipelines that key off the exit status — e.g. musefs scan … && musefs mount … — will now correctly stop on a partial-ingest failure; update any script that assumed 0.
• --fallback keys are case-insensitive. A per-field --fallback AlbumArtist=… (or any non-lowercase key) is now matched against the template field instead of silently never applying. If you worked around the old bug by lowercasing keys, no change is needed; uppercase keys now take effect.
• df on the mount now shows a synthetic capacity instead of zeros.
• Extended attributes (getxattr/setxattr/…) now return ENOTSUP explicitly on the read-only mount; the caller-visible result is unchanged, but the per-probe [Not Implemented] warning is gone.

4. External writers (beets, Picard, Lidarr, python-musefs) version independently and need no change for this upgrade: the new fingerprint / content_hash columns are scanner-owned and nullable, so the external-writer contract is unchanged. Update those packages on their own cadence.

Earlier releases

For v1.0.0 and earlier, see the Changelog.

Changelog

All notable changes to this project are documented here. The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

The contrib/ Python packages have their own decoupled version and changelog: see the contrib changelog.

For curated, upgrade-focused notes (highlights and per-version migration steps), see the Release notes.

Unreleased

Added

• readdirplus is implemented, folding the per-entry lookup into the directory read: a client that stats what it lists — ls -l, every media scanner — spends one round trip on the directory instead of one more per entry. Expect a few tens of percent off a repeat traversal rather than a multiple; a cold one is dominated by synthesis, where the round trip is a few percent. Directories and the synthetic entries are answered inline, and the file entries fan out across the worker pool in rounds, because concurrent lookups already spread across that pool and a serially resolved page would be slower for a threaded scanner than what it replaces. FUSE_READDIRPLUS_AUTO is requested too, so the kernel keeps using plain readdir for a listing nobody stats, where the larger entries would only cost reply pages. musefs_readdirplus_total reports whether the kernel is sending the op at all (#667).

• --trust-backing-mtime skips the backing re-stat that getattr performs on a metadata-cache hit, serving the cached size and mtime instead. Off by default, and scoped to getattr alone. The re-stat exists to catch an on-disk change that left content_version untouched (#279), which is the right default and stays the default; it has no escape hatch for backings where a stat is not roughly a microsecond. On NFS, SMB, or a spun-down array it is a network round trip or a head seek, and a warm cache does not help: a scanner walking ten thousand tracks pays ten thousand synchronous stats on every pass, and --attr-ttl-ms cannot debounce them because each track is stated once per traversal and a traversal outlives any TTL worth setting. open and the read paths validate unconditionally either way, so a replaced backing file is still caught before a byte is served, and the cold traversal that populates the cache stats regardless — the hit-path stat is the cost of every pass after the first, not of the first. musefs_trust_backing_mtime reports the flag state, which is what tells a quiet musefs_backing_stats_total from a disabled counter (#668).

• Chaptered .m4b files are supported. A moov may now hold chapter tracks (text, sbtl) alongside its single audio (soun) track, and every track's stco/co64 chunk offsets are relocated when the moov is regenerated, not just the first track's. A Nero chapter list (moov/udta/chpl) is copied verbatim into the rebuilt udta. Previously every chaptered file — which is to say most of an audiobook library, and chapters are why the .m4b extension exists — was counted as unparseable at scan time (#672).

• Musefs::drain_prefetch waits for the Phase-2 prefetch pool to finish every job it accepted, or a timeout to elapse. Serving never needs it — prefetch is fire-and-forget there — but sampling the prefetch counters without it misses reads still in flight, and a caller that owns the backing filesystem itself (the latency-injecting mount the read benches use) can otherwise tear it down under a worker mid-read and park that thread in uninterruptible sleep (#671).

• musefs_readahead_prefetch_reads_total and musefs_readahead_prefetch_bytes_total (#671) report the positioned backing reads issued by the Phase-2 prefetch workers. The serve-path musefs_backing_pread_* counters never saw those threads, so a prefetcher re-reading the stream several times over was invisible to every counter the daemon exposes and showed up only as backing I/O nothing accounted for. That is exactly the shape of the amplification bug fixed in this release, which is why the counters exist.

• musefs_dir_handle_rejections_total (#626), a monotonic counter of opendir calls that could not be given a cached directory snapshot. The existing musefs_dir_handles gauge cannot stand in for it: saturation is bursty, and a walk that produced 7,525 rejections never showed a gauge sample above 593. It is also the signal that the stateless path from #616 is in use and directories are being rebuilt on every readdir.

• musefs_serve_warns_suppressed_total (#653), a monotonic counter of serve-path failure warnings the rate limiter downgraded to debug. The count previously escaped only as a parenthetical inside the next admitted warning, which is both unscrapeable and carried by the admitted lines alone. The failure mode matches #626's: suppression is bursty by construction — 10 admitted per 30-second window, the rest dropped — so a scrape landing between bursts sees nothing, and "quiet" and "failing faster than it can log" look identical. Since the limiter moved into musefs-core (#650) the counter covers the synthesis warns too, not just the FUSE errno path.

Changed

• Directory handles on the same directory share one listing instead of copying it each. opendir took a private snapshot per handle, so the table's memory was the directory's width times the handle count: on a template that collapses a library into one directory, a client opening the 1,024-handle cap on it — which needs no privilege — pinned tens of gigabytes. A listing is now keyed by directory and virtual-tree generation, and handles that agree on both share it. All but the first also skip the tree walk that builds one, which an over-cap readdir consults before rebuilding as well. musefs_dir_listings reports the distinct-listing count behind musefs_dir_handles (#675).

• An MP4 file skipped for its track layout now reports the handler types found (unsupported MP4 track layout: expected one audio (soun) track, optionally with text/sbtl chapter tracks; found [soun, vide]) instead of a bare "not a supported MP4/M4A file", so the skip explains itself (#672).

• benches/storage_tunables_bench.sh gains a prefetch mode that A/Bs two or more musefs binaries (MUSEFS_PREFETCH_BINS="label=path ...") over one NFS+netem corpus, and its real-corpus filter now picks up .opus files. Its NFS modes also disable NFS LOCALIO for the run: on Linux 6.12+ a loopback mount negotiates local I/O and bypasses the RPC transport, so tc netem on lo had no effect on the data path and every "NFS" row measured local disk at GB/s (#671).

• Behavior change. A scan that hits a DB constraint violation on one file now runs to completion instead of stopping there (#662). Three observable consequences. It exits 2 — scan completed, at least one file failed — where it previously exited 1 as a hard error, so a pipeline keying on the exit code sees a different value for the same library. The store ends up holding every file in the library except the rejected one, rather than only the batches that committed before the abort, so a rescan after the fix no longer has an unknown amount of the walk left to redo. And the rejected file is reported rather than fatal: it is named in the log with the constraint text, counted in the new rejected bucket of the failed N: … summary, and missing from the mount while everything else is served. Anything that treated a constraint violation as a signal to stop the scan no longer gets one; the exit code and that summary are the signals to key on. The engineering is in Fixed below. See Scanning and Exit codes.

• The tags.value cap rises from 256 KiB to 16 MiB − 1, and track_art.description from 1 KiB to 8 KiB (schema MIGRATION_V3). The new tag cap is FLAC's 24-bit metadata-block ceiling — the largest tag synthesis could ever serve — so the store no longer refuses a tag the format itself can carry. Existing stores upgrade in place, and automatically, on the next musefs scan or musefs mount: both open the store read-write and run the migration, which only widens the constraints and so carries every existing row across. No rescan of audio is needed and nothing has to be regenerated.

• A backing file whose metadata exceeds a store limit now fails that file instead of being stored with the offending part quietly dropped. Oversize embedded art and binary tags were previously omitted from an otherwise-stored track with only a warn to show for it, which is easy to lose in a scan of ten thousand files and leaves a mount silently missing data. Such a file is now logged with its path, what was too big, its size and the limit, counted failed, and skipped; the rest of the directory scans normally.

• The virtual tree interns each node name into one shared Arc<str> rather than storing it separately in Node.name, Node.rendered_name, the children key and both rendered_children keys (#617). Measured over 200,000 tracks / 222,001 nodes: ~1713 to ~1284 bytes per track (~84 MiB, -25%), with tree build 10-15% faster from the removed allocations. A case-insensitive mount saves more, since folded_children held a sixth copy. The tuning guide gained a "Memory footprint" section. The full rendered paths were still stored twice at this point; the next entry shares those as well.

• Each entry's rendered path is stored once and shared between the inode allocator's map key and TrackRenderState.path, rather than allocated independently by each (#629). Measured over 200,000 tracks with 41-byte paths: 1150 to 1078 bytes per track (-6%); with 137-byte paths the saving is -14%. What it removes is exactly one whole path per track, so the gain tracks path length and template depth rather than track count — a flat --template gains little. Sharing is conditional: a disambiguated leaf keeps its own allocation, since keying it on the path its bare-named sibling already interned would collapse two nodes onto one inode.

• Full tree rebuilds and the head of every scan read projected columns instead of materializing a whole Track per row (#621) — roughly 40 MB of transient allocation on a 200,000-track store, on a path already holding a pool connection.

• The serve-path warn rate limiter is process-wide instead of FUSE-local (#650). It bounds failure warns to a burst of 10 per 30-second window, but it lived in musefs-fuse and was reachable only from the errno-reply path, so the warns synthesis itself emits bypassed it: a dropped Vorbis tag key (once per header-cache miss, and that cache is byte-budgeted, so an evicting library re-warns for the same track indefinitely), art over the byte cap, and a failed art-blob read — the last fires per art window, so one bad blob produced many lines for one file. The limiter now lives in musefs-core next to telemetry.rs and both crates share one budget, which is the right unit: the operator's concern is total serve-path log volume, not per-crate volume. Only the budget is shared, not the attribution: the emit side is the musefs_core::serve_warn! macro, so each record still takes its target from the call site's own module and per-crate RUST_LOG filtering (RUST_LOG=warn,musefs_fuse=debug) reaches exactly what it did before.

• readdir's unknown-fh fallback runs on the worker pool instead of inline on the fuser dispatch thread (#623), matching the offload every other blocking operation already used. This matters more now that over-cap opendir makes that fallback the normal path for large directories.

• Scan failures are now broken down by reason and their per-file warnings capped (#651). A scan that ends failed 37 also logs failed 37: unparseable=30, io=5, oversize=2 (and walk errors N: … for directories the walk could not read), so the number that drives the exit-2 partial-failure signal explains itself instead of having to be reconstructed from N individual lines. The per-file messages themselves are capped at ten per reason per scan, the rest dropping to debug — an unreadable subtree or a share that went away mid-scan no longer emits one warning per file. The existing per-extension skip breakdown moves from warn to info (so it now needs -v / RUST_LOG=info): cover art and .cue sidecars are the normal contents of a music library, and a warning on every healthy scan only teaches operators to tune warnings out. The skipped count itself is unchanged and still printed in the per-target summary.

Fixed

• A track that renders to .musefs-metrics or .metadata_never_index at the mount root no longer collides with the synthetic entry the FUSE layer injects there (#681). readdir appended the synthetic name to the root listing while lookup intercepted it before consulting the tree, so the name appeared twice with different inodes, ls reported the synthetic inode for both rows, and the user's own subtree was unreachable anywhere under the mount. Both names are now reserved in the virtual-tree namespace (musefs_core::RESERVED_ROOT_NAMES): a rendered root component that lands on one is pushed to its (2) rank, exactly as a colliding rendered name already is, so the synthetic entry keeps the base name and the user's data keeps a reachable one. The reservation is applied once, when the tree is built, and does not depend on --expose-metrics or on the platform — the same library therefore mounts to the same paths and inodes whichever way the flag is set and wherever it runs.

• A directory that had been ranked away from its base name by a collision is no longer re-created once per track. Which directory a path component belongs to is now decided by the rendered name rather than the stored one, so a library where a file renders to the same name as a directory — say a track that renders to Live next to an album directory Live — serves that directory's tracks from one Live (2) instead of scattering them across Live (2), Live (3), Live (4), one directory per track. By the same correction, a directory whose rendered name is literally another's rank keeps its own contents instead of absorbing them: Live (ranked to Live (2)) and a real Live (2) are two directories again. Surfaced while reserving the injected root names, which reach the same collision path (#681).

• Moving a backing file no longer wedges its track for the life of the mount (#679). A scan that retargets a row to a relocated file rewrites the path and the freshness stamp and correctly leaves content_version alone — the served bytes did not change — but the getattr size cache, the layout cache and every open file handle accepted their cached entry on content_version alone while holding the pre-move path and stamp. Each then validated the live file against the old stamp and failed, permanently: the file listed as -????????? ? and every read returned EIO until a remount. Both caches now compare the row's backing-source identity as well as its content identity, and a poll whose changelog names any track advances the refresh generation, so open handles re-resolve too. A move is not the only way in: any in-place re-stamp that leaves the content unchanged reached the same wedge.

• One unparseable METADATA_BLOCK_PICTURE no longer discards every other embedded picture in the same Ogg file, and the drop is logged instead of being swallowed by the scan path. Base64 decoding also tolerates ASCII whitespace, so a value wrapped in the older 76-column MIME style decodes rather than failing at the first line break (#673).

• A panicking worker-pool task permanently leaked a SQLite read connection and up to three file descriptors (#669). threadpool retires a worker that unwinds and spawns a replacement, and the replacement gets a fresh ThreadId — the key DbPool::PerThread stores connections under, and never evicts. The dead worker's connection therefore stayed in the map for the life of the mount while musefs_pool_workers kept reading healthy. read, lookup, getattr and open already ran their synthesis inside a catch_unwind, but opendir, the readdir stateless fallback and both poll_refresh tasks did not. Every pool submission now goes through one outer panic boundary, so no task can unwind out of a worker, and the two directory paths additionally guard their listing build so a panic there is answered with EIO instead of dropping the reply and hanging the syscall. Per-worker connection counts are now genuinely bounded by --workers.

• Phase-2 read-ahead prefetch (--read-ahead-prefetch) amplified reads instead of merely adding overhead (#671). ReadAhead::insert_window trimmed the ring to the first window lying fully behind the read frontier and fell back to index 0 when it found none. Windows are sorted by start, so index 0 is the lowest offset — the window the reader is currently inside. The frontier (next_expected) is the end of the last served read, not the end of the window that served it, so a 512 KiB window feeding 128 KiB reads is never "fully behind" until the reader has consumed all of it, and under ring pressure the fallback dropped exactly that window. Driving the real ReadAhead with the real plan_prefetch/prefetch_depth logic over 400 sequential 128 KiB reads: 398 of 400 reads missed and refilled synchronously, 3159 MiB of foreground backing reads for 50 MiB asked. The window keeps doubling across those refills because off == next_expected still holds, so it is pure amplification rather than seek thrash. The eviction order now prefers a window already fully consumed, then the furthest-future window the reader is not inside, and read_into advances the frontier before inserting so the trim sees where the reader actually is.

  Fixing that exposed a second defect it had been masking. A prefetch dispatch stops at the first window boundary at or past the horizon, so prefetched_upto legitimately ends up beyond the horizon; plan_prefetch read that overshoot as a backward seek and re-dispatched the whole horizon from the reader's position on the very next read, once the adaptive window outgrew the FUSE read size. With the eviction fix alone, a real 866 MiB FLAC read through a real kernel mount cost 8.5 GiB of backing reads (9.9x). plan_prefetch now tolerates an overshoot of up to one window before treating the watermark as a seek, and the same read costs 1.01x the file.

  With both fixed, the Phase-2 story changes on high-latency backends. On a real loopback NFS mount at 200 ms RTT (tc netem, LOCALIO disabled) prefetch is now a ~30% single-stream win, 6.3 → 8.2 MB/s, and a ~5% win on four concurrent streams; before the fix it was a regression on both, which is what the earlier "~10% overhead" finding was actually measuring. It stays opt-in: on local disk and over musefs-latencyfs it still reads the stream a second time speculatively (≈2x the backing bytes) for wall time within noise of amplification alone, so the win is one backend and one run. Measurements and method are in Benchmarks.

• A DB constraint violation raised while ingesting one file no longer aborts the whole scan (#662). This changes observable behavior — the exit code and what the store holds afterwards — and Changed above states that part; what follows is why and how. Cap violations were pre-checked in check_storable (#644), but that covers only the caps the scanner knows to look for; every other constraint the schema enforces — the CHECKs and the UNIQUE/primary-key constraints — was discovered by SQLite inside the ingest transaction, where the per-file context is gone, and propagated out as fatal. The reported case (#659) killed a scan 41% into an 891k-file library, about an hour in, leaving whatever the earlier batches had committed and no record of where the walk stopped. Pre-checking each newly discovered constraint does not converge, so the error is now classified at the ingest boundary instead: a constraint violation (SQLITE_CONSTRAINT, any extended code) is attributable to the rows one file wrote, and becomes one failed file in a new rejected bucket, named in the log with the constraint text and reported in the end-of-scan breakdown. Errors that say the run itself cannot proceed — SQLITE_CORRUPT, SQLITE_FULL, SQLITE_IOERR, SQLITE_READONLY, SQLITE_NOTADB, and any code this build does not recognise — still abort with the message they always did; SQLITE_BUSY is neither and stays with the writer's locking policy. Because the production path commits through BulkWriter, whose transaction holds a whole batch, each file is now ingested inside a SAVEPOINT (BulkWriter::item): a statement-level ABORT undoes only the statement that hit the constraint, so without one the batch would commit a half-ingested track — a tracks row whose tags never landed. The savepoint rolls the rejected file back whole and leaves the rest of the batch committable.

• A scan no longer aborts on a backing file that carries the same tag key as both a text value and a binary payload (#659). tags' primary key is (track_id, key, ordinal); it does not discriminate on value_blob, so a track's text rows and binary rows occupy one ordinal space per key. ingest numbered them independently — text rows from a per-key counter, binary rows from a single running index across the track — so a key present in both classes produced two rows at ordinal 0 and the ingest transaction failed with UNIQUE constraint failed: tags.track_id, tags.key, tags.ordinal. Unlike a cap violation (#644) this was not routed to a per-file failure, so it killed the whole scan — the reported case died 41% into a 891k-file library after an hour. Generalising that containment to any constraint violation is tracked separately in #662. The two classes now draw from one shared per-key counter, text first, which also makes binary ordinals per-key rather than track-wide. Reachable shapes: a FLAC CUESHEET Vorbis comment beside a CUESHEET metadata block, an MP3 TXXX frame whose description names a binary frame the same tag carries (PRIV, GEOB, MCDI, a non-MusicBrainz UFID), and an MP4 freeform atom written with both a text and a binary data box.

• Scan log records and the progress bar no longer clobber each other on an interactive terminal (#648). ScanReporter renders an indicatif bar on stderr and the log facade writes to the same stderr, with nothing coordinating the two: a record was emitted at whatever column the last bar frame left the cursor on, and the next 120 ms tick issued a clear-line that ate part of it. Both the warning and the bar came out mangled. This was reachable on essentially the first interactive scan of any real library, because the end-of-walk skip tally (#341) warns about cover.jpg / .cue / .log / .nfo sidecars, and every unparseable file warns from inside the pipeline. The CLI now owns a single process-wide stderr draw target: the scan bar draws through it, and the binary's env_logger is installed wrapped in a sink that emits each record while that target is suspended — bar cleared, record written, bar redrawn below it. The per-target summary line (scanned N: …, on stdout) is suspended the same way; it used to be glued onto a bar frame whenever no log record happened to precede it. Off a terminal the draw target is hidden and suspending is a no-op, so this change left the --quiet and piped milestone paths byte-for-byte alone, as it did the verbosity policy (-v/-vv/-vvv, RUST_LOG taking precedence), which stays in the binary. (The milestone line is renamed by the progress-bar convergence fix below, in this same release.)

• The scan progress indicator now reaches 100% when files fail (#655). The bar's length is the walked file count, but its position only advanced on a committed file, so any failure left it permanently short — a run with 12 unparseable files out of 42 finished at 30/42 (71%) and was then cleared, which reads as an aborted scan rather than a completed one about to report failed 12. On the piped path the final 100% milestone was simply never printed, so a log-scraping consumer waited for a line that could not arrive. A dispatched file that fails or races now advances the same progress sequence as a committed one; the two together always account for the walked total, and a debug assertion pins that. The piped milestone line is renamed ingested N/M (P%) → processed N/M (P%), because it now counts every file the pipeline finished with rather than only the successes — scripts matching the old prefix need updating.

• A tag larger than the store's cap no longer aborts the entire scan (#644). The scanner had no length check on text tags, so an over-cap value reached the DB CHECK inside a batch commit and failed the whole run with CHECK constraint failed: length(CAST(value AS BLOB)) <= 262144 — naming neither the offending file nor what the number meant. Every cap the scanner can trip is now checked in one place before anything is written, and a violation fails only that file, with a message naming it. The same applies to the tags.key, art.mime and track_art.description caps, which had the identical unattributed-abort failure mode. Should a store write still fail fatally, the error now names the file it died on.

• A FLAC whose tags outgrow what a VORBIS_COMMENT block can hold is rejected at scan time rather than stored and then served EIO on every read. This is reachable by merging a leading ID3v2 tag's fields into a FLAC's own comments (#602): ID3v2's tag size is synchsafe 28-bit (256 MiB) while a FLAC metadata block is 24-bit.

• A leading ID3v2 tag on an MP3 (or a FLAC) is stepped over by its declared size even when its major version is not one musefs can parse the frames of, instead of the whole file being rejected. The ID3v2 header has the same shape in every version, so its size is enough to step over the tag, and the spec's rule for a version a reader does not understand is to ignore it. Such a tag's frames are still not read.

• Over-cap opendir degrades to a stateless directory handle instead of replying ENFILE (#616). The 1024-handle cap was assumed to sit well above any real client, but bfs — an ordinary parallel find, and the default find in several distributions — exceeded it immediately: 7,525 rejections over a 200,000-track mount, 17,910 files never enumerated. The rejection surfaced to the operator as "Too many open files in system", which points at the kernel rather than the mount, and an indexer that logs and continues would simply present an incomplete library. Serving over-cap directories through the existing stateless fallback keeps listings complete and preserves the memory bound the cap was added for, at the cost of an O(N) rebuild per readdir.

• Unmount helpers are resolved against /usr/bin, /bin and /usr/local/bin before falling back to a bare-name PATH lookup (#620). The mounting guide steers operators toward running as root for kernel passthrough in StructureOnly mode, and nothing sanitizes PATH there, so a writable PATH entry meant attacker-chosen code executed as root on SIGTERM.

• A failed batch commit during a scan winds the pipeline down before the error propagates (#618). ByteBudget gained a close() that wakes waiters, so a worker parked in acquire on a condvar only flush ever signalled is no longer stranded. This was benign for the CLI, where process exit reaps everything, but scan_directory_with / revalidate_with are public API and an embedder that caught the error accumulated leaked threads and their in-flight art bytes. The two state-mutex unwrap()s adopted the daemon's lock_recover policy at the same time, retiring the open note in lock.rs.

• find_page_start bounds the number of candidate pages it CRC-validates per call (#619). The header pre-filter almost never admits a false OggS on real audio, but a file whose audio region is deliberately packed with OggS\x00\x00 cleared it at every offset, allowing up to ~65,000 CRC validations — each with its own positioned reads — for a single seeking read. Hardening rather than a live vulnerability: the attacker is whoever can place a file in the scanned library.

• access is implemented and replies ok (#624), so fuser's default no longer logs [Not Implemented] per mount. The mount carries RO and, with allow_other, DefaultPermissions, so the kernel already enforces the presented mode bits.

Internal

• The ogg_page fuzz target round-trips the page machinery the serve path actually depends on — verify_page_crc and patch_page_header_algebraic — instead of only decoding a header (#625). Coverage against the committed seed rose from 51 edges / 69 features to 129 / 267.
• The read-ahead budget invariant restored by #536 now has a concurrent regression test that the ASan and TSan CI legs actually reach (#628). The sanitizer legs previously ran a test that builds ReadAheadPool::new(0), leaving the pool disabled throughout. No live bug was found; this closes the coverage gap.
• musefs-core/tests/tree_footprint.rs gates the virtual tree's per-track resident cost (#629), turning #617's throwaway probe into a committed ceiling. It samples VmRSS either side of VirtualTree::build_with with the rendered paths materialized beforehand, so the delta is the tree's own marginal cost. The ceiling is deliberately loose — a gate that catches only a large regression is worth more than one that reddens on a loaded runner — and a floor assertion keeps a broken measurement from passing vacuously.
• deny.toml's allow and ignore lists can no longer rot (#622). A stale RUSTSEC-2025-0167 ignore and an unmatched ISC license allowance both warned and exited 0, so neither surfaced on a PR — which is how an entry ends up silently pre-exempting the next real advisory for the same crate. Both were dropped, and advisory-not-detected / license-not-encountered are now errors in the deny job. The promotion is scoped to the root graph, which is what the lists are authored against; the fuzz-lockfile scan in audit.yml allows the advisory diagnostic explicitly, since off that graph it is noise.
• Issue, pull-request and CODEOWNERS templates (#627). The PR checklist covers the steps that are easy to forget and silently break something: the pre-commit hook, cargo +nightly fuzz build after a format-layer API change, regenerating the Python schema mirror after a musefs-db change, and a changelog entry.

1.3.0 - 2026-08-19

Added

• FLAC files carrying one or more ID3v2 tags in front of the fLaC marker are now scanned instead of being skipped with "no parseable audio metadata" (#602). The tag run is stepped over to reach the FLAC stream, and its text frames and APIC pictures are ingested as a fallback beneath the file's own VORBIS_COMMENT / PICTURE blocks — so a FLAC whose tags live only in the ID3 header lands in the store with its tags. A trailing 128-byte ID3v1 tag is trimmed from the audio length (checked only for files with a leading ID3v2 tag, so a stock FLAC pays no extra read). Neither tag survives into the synthesized file, which is a stock FLAC starting at fLaC.

Changed

• musefs-core now builds its persistent virtual-tree collections on imbl 7 instead of the archived im 15. Clears RUSTSEC-2026-0248 / RUSTSEC-2023-0126 (im) and RUSTSEC-2026-0251 / RUSTSEC-2026-0255 (sized-chunks).
• VirtualTree::children now yields an opaque impl ExactSizeIterator<Item = (&str, u64)> instead of borrowing the backing OrdMap, taking the persistent-collection crate out of musefs-core's public API so swapping it stays an internal detail. In-tree the only caller is readdir, which iterates; name lookups have always had VirtualTree::lookup.

Fixed

• Bumped crossbeam-epoch 0.9.18 -> 0.9.20 (RUSTSEC-2026-0204, invalid pointer dereference in the fmt::Pointer impls) and num-bigint 0.4.7 -> 0.4.8 (0.4.7 was yanked). Both are dev-dependency-only paths (criterion -> rayon, mp4 -> num-rational).

1.2.0 - 2026-06-18

Changed

• Bare scan is now additive: it skips rows already in the DB instead of re-seeding them from disk. Use scan --force when you want the old full-reimport behavior.
• revalidate is now its own subcommand and no longer prunes by default. It refreshes changed rows' structural data while preserving curated tags/art; use revalidate --prune to drop rows whose backing file is gone and garbage-collect orphaned art.

Deprecated

• scan --revalidate is a deprecated, warned alias for revalidate and will be removed next release. It does not prune; use revalidate --prune when you need deletion.

Fixed

• Revalidating a changed file no longer clobbers curated tags, art, or binary tags in the DB.

1.1.0 - 2026-06-17

Added

• Runtime telemetry (.musefs-metrics): an opt-in --expose-metrics flag (env MUSEFS_EXPOSE_METRICS) surfaces a synthetic .musefs-metrics file at the mount root rendering Prometheus-format counters — getattr/read/open activity, backing read-ahead behavior, and (when built with jemalloc) allocator stats. Off by default; the file is absent unless enabled. See the README Metrics section (#394).
• Scan progress indicator: scan and scan --revalidate render a live progress bar (indicatif) with an elapsed-time summary on an interactive terminal, falling back to periodic ingested N/M (P%) log lines when output is non-interactive. A new --quiet/-q flag suppresses it (#406).
• --skip-on-missing template flag: an opt-in --skip-on-missing (env MUSEFS_SKIP_ON_MISSING) drops a track from the mount when a top-level template field stays unresolved, instead of substituting --default-fallback. Per-field --fallback chains and [...] optional sections are unaffected (a field resolved via its fallback counts as present). The motivating case is --template '$!{beets_path}' --skip-on-missing, which hides tracks beets left without a beets_path rather than collapsing them into an Unknown bucket (#408).
• --read-ahead-prefetch flag: opt-in background prefetch threads layered on top of read amplification, default off — benchmarks found amplification alone delivers the entire read-ahead win, while the threads add ~10% overhead with no measured benefit. Enable only when profiling a backend where a single large read does not self-pipeline (#255).
• riscv64 release platform: prebuilt riscv64gc-unknown-linux-{gnu,musl} binaries and linux/riscv64 Docker images now ship with each tagged release. Container bases bumped to current stable: glibc Debian bookworm → trixie (bookworm has no riscv64 image), musl Alpine 3.20 → 3.23 (3.20 is end-of-life).
• statfs reply: the mount now reports a non-zero synthetic capacity with ample free space instead of fuser's all-zero default, so df no longer shows a 0-byte filesystem and capacity-checking importers (Lidarr et al.) don't balk (#368).
• Per-extension skip breakdown: at end of scan, a summary line breaks the skipped count down by lowercased extension (e.g. skipped 42: jpg=20, cue=10, log=8, <none>=4), logged at warn so it shows by default, so a large skip count is diagnosable — expected sidecars versus genuinely unexpected files. Log-only; the ScanStats struct and CLI summary are unchanged (#341).
• musefs vacuum command: compact the SQLite store, reclaiming free pages left by prunes, orphan-art GC, and the schema migration. Runs VACUUM + a WAL checkpoint and reports the space reclaimed; run it while unmounted (#566).

Fixed

• Art/serve rowid-reuse consistency: the read fast path's WAL-snapshot + content_version guard, previously gated only on binary-tag layouts, now covers all DB-rowid segments (art ArtImage/OggArtSlice too) via RegionLayout::streams_db_rowid, and the stateless no-fh read fallback now applies the same snapshot/recheck and re-validates its freshly opened backing fd against the resolved stamp. A concurrent external retag + gc_orphan_art + reinsert can no longer splice a wrong image or stale tag bytes mid-read (the audio-bytes invariant was never affected) (#502, #503).
• Per-field --fallback case-insensitivity: fallback keys are now ASCII lowercased to match template field names, so --fallback AlbumArtist=… (any uppercase) is honored instead of silently never matching (#504).
• Tag value byte cap: both the schema CHECK (rebuilt in the MIGRATION_V2 upgrade) and the read-time tags.value guard now count bytes, not UTF-8 characters, so the 256 KiB materialized-memory bound is exact rather than up to ~4x looser for multibyte text. The upgrade drops any pre-existing over-cap rows (already unreadable under the byte-counting reader guard) (#505).
• Embedded NUL in ID3 metadata: synthesized ID3 frames now reject a DB-sourced tag key, tag value, art mime, or art description containing an embedded NUL instead of emitting a frame a downstream parser would misread (#506).
• Orphan-art GC NULL safety: gc_orphan_art uses NOT EXISTS rather than NOT IN (subquery), so a NULL art_id could not silently turn the GC into a no-op (#507).
• Mount usability: mount now warns when the mountpoint is non-empty (its contents are shadowed for the mount's lifetime), and a permission-denied mount (e.g. an AppArmor-restricted prefix) prints actionable guidance instead of a bare "Permission denied" (#508, #509).
• Silent mp4 oversize drops: oversized embedded covr cover art and binary freeform (----) values in .m4a/.m4b files are skipped in the format layer before materialization (to avoid building a large image out of a large moov), which previously dropped them with nothing in the logs. The scan now emits a warn line for each, matching the logging the other formats already had (#343, follow-up to #284).
• xattr log noise: getxattr/listxattr/setxattr/removexattr now reply ENOTSUP explicitly (read-only filesystem, no extended attributes) instead of falling through to fuser's default, which logged a [Not Implemented] warn on every xattr probe (ls -l, indexers, backup tools). The caller-visible result is unchanged (#364).
• MP4 path-to-ilst leniency: the walk to moov/udta/meta/ilst now uses the same lenient box scan as the metadata extractors, so a single malformed or truncated sibling box anywhere on the path no longer suppresses an otherwise well-formed ilst and silently drops every tag and cover. The audio/structure path stays strict (#542).
• QuickTime bare meta atoms: the meta parser only consumes the 4-byte FullBox version/flags prefix when it is actually present (a zero word), so a QuickTime-style bare meta — which has no such prefix — is read instead of landing mid-header and dropping all tags and art (#543).
• scan exit code on ingest failure: scan/scan --revalidate now exit 2 when any file fails to parse/ingest (failed > 0), instead of always exiting 0. A pipeline such as musefs scan … && musefs mount … can now detect a partial or total ingest failure; a clean scan still exits 0 and a hard error still exits 1 (#554).
• Release smoke audio-bytes check: scripts/smoke-binary.sh (the per-arch release gate) now compares the served file's encoded audio stream against the untouched backing file, asserting the cardinal byte-identical-audio invariant rather than only checking the fLaC magic — so a target-specific positioned-read or offset regression in a cross-compiled binary is caught (#547).

1.0.0 - 2026-06-12

First stable release.

Added

• Lidarr integration: a new contrib/lidarr/ package that drives symlink-based placeholder imports and syncs Lidarr metadata into the musefs SQLite store.
• FUSE mount-access controls: new --allow-other, --owner, and --group flags mount with allow_other + default_permissions so accounts other than the mounting user can reach the view and the presented owner/group/mode bits are enforced; --owner/--group imply --allow-other. A non-root allow_other mount is pre-flight checked against /etc/fuse.conf user_allow_other and fails early with guidance if it is missing. See the README Ownership and permissions section (#293, #294).
• Hardened deployment assets: the container image runs as a dedicated unprivileged user with a build-arg-configurable UID/GID, and the musefs-scan.service systemd unit ships a strong sandbox (the FUSE-mounting musefs.service deliberately cannot be sandboxed). See the systemd hardening notes (#317, #318, #319).
• crates.io distribution: the musefs binary is published to crates.io as of this release and installable with cargo install musefs. A new thin musefs wrapper crate owns the binary (musefs-cli is now a library crate), and a tag-triggered release workflow publishes all crates in dependency order.
• Fuzzing & property tests: coverage-guided cargo-fuzz targets for every format parser (FLAC, MP3, MP4, Ogg, WAV), the byte-level primitives (Ogg page parsing, base64 windowing, VorbisComment), and the serve path — the latter drives the full synthesis pipeline over hostile DB rows and binary tags via a fuzzing-gated Db::with_raw_conn. Plus proptest invariants — panic-freedom, the byte-identical audio guarantee, and tag round-trip — an end-to-end read-fidelity property, and a mutagen interop test asserting an independent reader sees the tags we synthesize.

Changed

• mount --db now requires an existing store. Mounting against a missing database path is rejected before any FUSE setup instead of silently creating and migrating an empty store, so a mistyped --db fails loudly rather than mounting an empty view. scan --db still creates the store if absent (#309).

Fixed

• Scanner no longer drops files and embedded art silently: embedded cover art over MAX_ART_BYTES (and binary tags over MAX_BINARY_TAG_BYTES) were filtered out at ingest with no log line, so a track whose art exceeded the cap appeared to simply have none — indistinguishable from a scan bug. The drop is now logged (RUST_LOG=warn). Likewise, a supported-extension file that fails to parse or errors mid-probe was counted failed with the underlying error discarded; the reason is now logged. Note: oversized art in .m4a/.m4b files is dropped earlier, inside the format layer, and is not yet logged (#284, #343).
• Lidarr custom-script env var casing: Lidarr stores custom-script environment variables in a .NET StringDictionary, which lowercases every key, so a Linux script actually receives lidarr_sourcepath / lidarr_eventtype rather than the PascalCase names Lidarr's docs list. The integration read the PascalCase names, so with a real Lidarr every import failed and every event parsed as unsupported. Lidarr env vars are now resolved case-insensitively. Found by the issue #141 real-instance smoke run.
• VorbisComment parse OOM (DoS): a crafted comment block declaring a huge entry count made Vec::with_capacity attempt a multi-gigabyte allocation; the pre-allocation is now bounded by the readable byte count. Found by the new vorbiscomment fuzz target.
• MP4 box-bounds integer overflow: an untrusted 64-bit extended box size made the box-bounds check (pos + total) overflow usize — a panic in debug and a silent wrap in release that accepted a bogus box length. The addition is now checked. Found by the mp4 fuzz target.
• ID3v2 parsing unbounded allocation (DoS): the id3 crate eagerly allocates a frame's declared size (ID3v2.3 frame sizes are plain 32-bit, up to 4 GiB), so a crafted tag could exhaust memory at scan time — via an MP3 or a WAV embedded id3 chunk. Parsing is now gated on validated ID3v2 frame bounds and an ID3v2 tag at offset 0 (the id3 reader scans forward). Found by the mp3 and wav fuzz targets.
• Scan counters now match their documented contract: musefs scan reports every non-audio file (any unsupported or missing extension — .jpg, .cue, .log, .nfo, cover art, etc.) as skipped, and supported-extension files that fail to parse (e.g. a corrupt .flac) as failed. Previously malformed files were miscounted as skipped and unsupported files were not counted at all, so expect skipped to be larger than before on a real library (#301).
• Symlink scans no longer double-count: with --follow-symlinks, a file reached via both its real path and a symlink is ingested and counted once instead of inflating scanned; multiple hardlinks to the same inode are likewise collapsed to a single track (#302).
• Stable inodes on case-insensitive mounts: the inode allocator is now keyed on the case-folded path in case-insensitive mode, so an unrelated deletion that flips a merged directory's display casing no longer reassigns a survivor's inode (#305).
• Lidarr autoscan now honors the scan timeout: an import/release-triggered autoscan applies the shared 120s scan timeout, matching the beets and Picard integrations, so a wedged musefs scan fails with a controlled timeout instead of blocking the custom-script process indefinitely (#312).

0.2.0 - 2026-05-27

First public release.

Added

• Formats: synthesis for M4A/M4B (MP4), Ogg (Opus, Vorbis, FLAC-in-Ogg), and WAV, alongside the existing FLAC and MP3 — metadata generated on the fly from the SQLite store and spliced in front of byte-identical backing audio.
• Arbitrary tag support: a single canonical tag vocabulary maps common fields to each format's native slot (ID3 frame / MP4 atom / Vorbis field); any other tag round-trips through the format's extension slot (ID3 TXXX, MP4 ---- freeform, raw Vorbis field). User-defined key casing is preserved.
• beets plugin (contrib/beets/): syncs beets' canonical tags and cover art into the store keyed by each file's real path, with no remount and no audio rewrite.
• Performance, concurrency & caching pass: worker-pool offload of blocking reads, lock-free virtual-tree swap, per-handle I/O, a bounded LRU header-layout cache, debounced single-flighted refresh with stable inodes, kernel/mount tuning flags, bounded-memory MP4 resolves, and opt-in --keep-cache with auto-invalidation.

Notes

• Read-only mount; tag edits happen out-of-band against the SQLite store and are picked up automatically (PRAGMA data_version polling). See the README Supported formats section and the per-format docs for round-trip limitations.

0.1.0

• Initial MVP (FLAC and MP3 synthesis, virtual tree with beets-style templates, synthesis / structure-only mount modes, auto-refresh, scan / scan --revalidate). Never published publicly; superseded by 0.2.0.

Security

Security Policy

Supported versions

Security fixes target the latest release (see CHANGELOG.md); there are no maintained backport branches.

Reporting a vulnerability

Please report vulnerabilities privately via GitHub's security advisory form: github.com/Sohex/musefs/security/advisories/new. Do not open a public issue for an undisclosed vulnerability.

You can expect an acknowledgment within a few days. Confirmed issues are fixed as a priority, the fix is noted in the changelog, and you will be credited in the advisory unless you prefer otherwise.

What counts

musefs's primary threat surface is parsing untrusted media files: the scanner probes arbitrary bytes at scan time, and the serve path re-parses file fronts at resolve/read time. Anything a crafted file can do beyond "fail to scan with a controlled error" is in scope — memory unsafety, panics reachable from file contents, unbounded allocation, and hangs. Parser denial-of-service findings are real vulnerabilities here, not mere robustness bugs: several (a VorbisComment pre-allocation OOM, an MP4 box-bounds overflow, an ID3v2 allocation bomb) have been found by the project's fuzz targets and fixed — see CHANGELOG.md. Those fuzz and property suites run continuously (CONTRIBUTING.md); a fuzz reproducer is the ideal report attachment.

Also in scope: anything that lets a crafted database (the mount trusts its --db only as far as the documented contract) or a hostile local writer violate the read-only guarantee on backing files.