Hig: project-aware incremental archiving for developer workflows

Abstract

Developer projects are repeatedly archived during local builds, handoffs, checkpoints, and tool-assisted workflows. Conventional archive utilities optimize an individual invocation, whereas repeated project archiving must also manage change discovery, secure reuse, bounded memory, and variable local storage behavior. This paper presents Hig, a project-aware incremental archiving framework whose current first-major-release line is v1.9.7. Hig combines an HIGV2 archive surface, project snapshots, a watcher-assisted change journal, reusable prepared objects, bounded-memory pipeline staging, and standard cryptographic primitives. It does not introduce a new compression, encryption, hash, or cryptographic algorithm; its contribution is the engineering framework that coordinates established techniques for recurring developer workflows.

The paper reconstructs the evidence-backed evolution from v1.0.1 through v1.9.7, separating controlled comparisons from observations collected under different I/O conditions. In a controlled v1.9.6/v1.9.7 cold-path study, v1.9.7 reduced source-read worker time by 75.8% while preserving archive size and tested cross-version extraction compatibility. A 15,330-file large-project comparison is retained as contextual evidence because the archived record does not preserve all baseline tool settings. Real-project results also reveal non-stationary storage effects that preclude a blanket throughput claim.

Keywords: Archiving; developer tools; incremental systems; project-aware storage; reproducible benchmarking; secure systems.

I. Introduction

Developer projects are archived repeatedly: before experimentation, during local automation, while moving work between machines, and as a compact project-state handoff. In this setting, the cost of a pack operation is not defined solely by compression throughput. It also includes determining what changed, deciding which previously prepared objects may be reused, allocating bounded memory, preserving a stable archive surface, and writing to a storage device whose latency may vary substantially.

Hig is a Rust implementation intended for these recurring workflows. Its release line evolved from a file-level cached archiver in v1.0.1 to the HIGV2 format and, subsequently, project snapshots, daemon-backed coordination, watcher-assisted invalidation, and cold-path reuse refinements in v1.9.7. Hig uses Zstandard compression, BLAKE3 hashing, Argon2id password derivation, and ChaCha20-Poly1305 authenticated encryption; it does not claim a new primitive in any of these categories.

The paper makes five contributions:

1. It defines a public, non-sensitive functional model for project-aware incremental archiving and states the boundary between the published system description and proprietary implementation details.
2. It reconstructs the Hig evolution across v1.0.1-v1.9.7 from release artifacts, including positive, failed, and inconclusive optimization outcomes.
3. It evaluates archive size, warm project workflows, controlled cold-path behavior, memory policy, correctness, and compatibility using recorded benchmark artifacts.
4. It adds an engineering verification path connecting build gates, tests, cross-version extraction, digest checks, and benchmark qualification to the reported claims.
5. It provides a measurement discipline for future releases without conflating distinct corpora, security modes, or storage conditions.

II. Related Work

Incremental transfer and storage systems have long exploited similarities between successive data states. The rsync algorithm uses rolling checksums to identify matching blocks across endpoints. LBFS applies content-defined chunking to exploit similarity across files, while Venti uses content-addressed archival storage. These works establish the value of reuse, but their primary settings differ from a local developer-oriented archive workflow that must integrate filesystem change observation and an archive writer.

Deduplicating backup systems and foundational chunking studies show the trade-off between chunk granularity, metadata cost, and reuse. Hig adopts the general insight that one block treatment is not appropriate for all files. Its public model therefore contains small-object aggregation, large-object chunk handling, and independent payload preparation, while omitting private thresholds and cache schemas.

Archive format and compression design have different priorities. ZIP remains a ubiquitous interoperability baseline, while Zstandard specifies a modern compressed-data format. Hig retains a stable HIGV2 surface during the v1.9.7 cold-path work, keeping performance refinement separable from format migration.

Borg and Restic are repository-oriented backup systems with content reuse, encryption, and retention-oriented workflows. Git represents project history through content-addressed objects, and Bazel remote caching reuses action outputs rather than general archive payloads. Hig is not presented as a replacement for these systems; its scope is a portable archive artifact for a repeatedly changing local project, with a project-state layer intended to reduce redundant preparation before that artifact is emitted. No experiment here claims a head-to-head comparison with Borg, Restic, Git, or Bazel.

Security-sensitive archive tools must make explicit use of established primitives and threat assumptions. Argon2id is specified for password hashing, ChaCha20-Poly1305 provides authenticated encryption, and BLAKE3 supplies a fast cryptographic hash. The framework composes these primitives; it does not modify their security definitions. Performance measurements for encrypted and unencrypted workflows are therefore not directly interchangeable.

III. Framework Design and Disclosure Boundary

A. Design Goals

Hig targets compact archives for mixed developer trees, efficient repeated pack operations, integrity-aware secure defaults for password archives, bounded resource behavior, and stable extraction compatibility during optimization. The implementation offers balanced and fastest modes with explicit security and trust choices rather than silently weakening the default password path.

B. Public Functional Architecture

A client invokes a pack operation directly or through a local coordination service. In project mode, a watcher and change journal maintain a project snapshot. The scan stage validates current file state and produces work descriptions. A planner groups or chunks work at a high level, the bounded-memory stage prepares payloads, and the writer emits the archive and protected metadata or payloads when password encryption is selected.

This structure supports scheduling that can prioritize smaller work while maintaining deterministic output ordering, together with a buffer pool that bounds reusable working storage. These are scheduling and resource-management mechanisms, not novel compression or cryptographic algorithms.

C. Evolution of the Framework

Early v1.0.1 artifacts demonstrate file-level block reuse but also expose poor small-file behavior. v1.2.0 introduces HIGV2 batch and chunk representations. The v1.8 series adds mature daemon/session and release-gate instrumentation. The v1.9 series focuses on watcher-assisted project mode, write-path telemetry, adaptive memory behavior, and reduction of duplicate cold-path reads.

D. Disclosure Boundary

The paper reports externally observable behavior, compatibility, and experimental measurements. It does not disclose internal cache schemas, exact block-selection thresholds, raw key-handling details, storage paths, proprietary scheduling heuristics, or deployment hardening procedures. Commands, corpus characteristics, versions, modes, output metrics, and checksums can still be reported without publishing internal secrets.

IV. Evaluation Methodology

A. Artifact Set and Version Scope

The analysis uses the project release artifact set from v1.0.1 through v1.9.7. It includes synthetic source, repetitive, random, and small-file corpora; release-gated source-tree benchmarks; a 15,330-file large developer project corpus; a 17,583-file real IDE corpus; and controlled v1.9.6/v1.9.7 cold-path corpora. This is a longitudinal suite rather than a single universal benchmark. Numbers are compared directly only when corpus, command mode, version pairing, and environment conditions support the comparison.

<table>
<thead><tr><th>Family</th><th>Primary evidence</th><th>Valid interpretation</th></tr></thead>
<tbody>
<tr><td>v1.0-v1.2 synthetic</td><td>Source, repetitive, random, small files</td><td>Functional evolution and edge cases only</td></tr>
<tr><td>v1.6-v1.8 release gates</td><td>Qualified and unqualified local volumes</td><td>Same-run comparison with qualification retained</td></tr>
<tr><td>v1.9 project watch</td><td>15,330-file developer project</td><td>Warm workflow behavior and stage hotspots</td></tr>
<tr><td>v1.9.7 cold path</td><td>Interleaved v1.9.6/v1.9.7 runs</td><td>Controlled comparison for retained-read optimization</td></tr>
<tr><td>v1.9.7 real IDE</td><td>17,583 files, 505.9 MB</td><td>Memory behavior and I/O sensitivity</td></tr>
</tbody>
</table>

B. Metrics

The study records archive bytes, core duration, CLI wall time where available, stage timings, cache reuse, watcher status, memory staging, and source-to-extracted-tree digest equality. Archive size is a direct byte count. Duration is reported as recorded by the associated artifact, typically a median when repeated samples exist. Archive size ratio is R_s = B_archive / B_input. For paired controlled experiments, relative change is Δ_x = (x_v1.9.7 - x_v1.9.6) / x_v1.9.6 × 100%.

C. Benchmark Provenance and Reproduction Envelope

The retained artifacts preserve corpus counts and bytes, Hig version, cache state, daemon mode, encryption mode, memory policy, repeat count, and selected command shapes. The controlled cold-path study also preserves the input digest and interleaving rule.

The historical v1.9.6 large-project comparison does not consistently preserve host model, operating-system build, filesystem, baseline utility version, compression level, or all baseline flags. Consequently, the controlled v1.9.6/v1.9.7 cold-path pair is the causal performance evidence, while the historical utility comparison is a transparent contextual observation.

Future evaluations should publish a machine-readable host manifest, exact command lines and baseline versions, generated public corpora with deterministic seeds, raw repetitions, and confidence intervals. Until then, the protocol is internally auditable rather than externally fully reproducible.

D. Correctness and Engineering Verification

Correctness requires successful extraction and equality of the restored tree digest with the input digest. The v1.9.7 cold-path experiments additionally use bidirectional cross-version extraction. The engineering verification path covers static quality, core behavior, archive correctness, compatibility, performance telemetry, and environment qualification. Each layer has a boundary: static quality does not prove performance, tests are finite, correctness covers measured corpora, compatibility is not a complete format proof, telemetry is I/O-sensitive, and qualification does not normalize hardware.

E. Multidimensional Evidence Map

The evaluation separates evidence dimensions instead of reducing the system to one speed score. “Ctrl.” means controlled or compatibility-validated, “Meas.” means directly measured telemetry, and “Obs.” means an observation useful for interpretation but insufficient for a causal performance claim. A strong result in source-read reduction must not be used to imply stable end-to-end wall time on every storage device.

V. Results

A. Longitudinal Findings

The earliest v1.0.1 artifacts show why a project-aware framework is needed. On 500 tiny files totaling 18 kB, the first archive was 142,524 B, an expansion of 691.8%. Repetitive data compressed effectively, motivating aggregation in HIGV2. In v1.2.0, HIGV2 batch mode archived a 133,195-B source corpus to 28,250 B, compared with 31,238 B without batching and 32,967 B in the legacy one-file-per-block format. This is a format-level observation on that corpus, not a universal compression ranking.

The v1.8 release artifacts show a mature measurement posture: repeat sampling, volume qualification, cache and stage telemetry, and explicit security-mode labels. In qualified v1.8.5 large-project measurements, the balanced secure daemon pack-core median was 377.56 ms and zip was 3,750.94 ms. Later v1.9 artifacts add project-watch scenarios and retain failed gates when output I/O prevents the target.

B. Large-Project Archive Comparison

The v1.9.6 Project-A experiment uses an anonymized large developer project corpus with 15,330 files and 198,974,616 input bytes. The HIG archive was 57.11 MB; recorded zip and tar.gz outputs were 67.75 MB and 61.33 MB. Recorded first-pack times were 1.18 s for Hig, 4.38 s for zip, 10.89 s for tar.gz, and 8.50 s for tar.zst.

These values are not treated as a fair causal ranking: the volume was unqualified and the artifact does not preserve baseline utility versions, levels, or complete command flags. They document the historical workload and motivate the controlled study rather than establish general superiority.

C. Project-Mode Workflows

For v1.9.6, project snapshot reuse recorded 15,330 hash reuses and no watcher overflows. Project bootstrap pack took 1,430.03 ms; a single-edit pack took 187.42 ms; a five-edit pack took 174.41 ms; and a 1,000-event burst pack took 293.77 ms. The project warm median was 169.99 ms, missing the 150-ms target. Stage telemetry identifies output write at 105.18 ms, output flush at 38.29 ms, and cryptographic work at 20.98 ms as the largest observed contributions.

This miss matters: a project-aware system can eliminate data-read and compression work for unchanged payloads while still being constrained by output and filesystem behavior.

D. Controlled v1.9.7 Cold-Path Comparison

The controlled cold-path experiment uses a 6,154-file corpus of approximately 291 MiB, new caches, balanced compression, daemon disabled, and interleaved runs. Unencrypted median source-read worker time decreased from 810.5 ms in v1.9.6 to 196.2 ms in v1.9.7, a change of -75.8%. Core duration decreased from 1,605.5 to 1,495.5 ms, a change of -6.9%, while the 119.60-MB archive size showed no material change.

In password-encrypted runs, read time decreased by 74.5%, but end-to-end duration was neutral within output-I/O variance. The narrow supported claim is that v1.9.7 reduces duplicate source-read work for its retained window; it is not a universal first-pack speedup claim.

E. Memory Policy and Real IDE Corpus

The 17,583-file real IDE corpus is useful for resource-policy evaluation but not blanket speed ranking. Default v1.9.7 adaptive mode used no payload spool bytes and recorded 289.56 MB peak pipeline memory, compared with 125.83 MB in an explicit fixed low-memory spool configuration. The low-memory configuration moved 180.51 MB through a same-volume spool. Absolute durations were collected under severe non-stationary storage latency, so the apparent 7.24x duration difference cannot be attributed solely to memory policy.

A subsequent hot-raw experiment eliminated block-preparation source rereads for the full 505.9-MB corpus but increased reported peak pipeline memory to 795.47 MB. The appropriate conclusion is a transparent time-memory trade-off, not a default throughput claim.

F. Write-Path Diagnostics and Compatibility

On a 292-MB benchmark corpus with 163.98 MB of archive payload bytes, the median total write stage was 658 ms. Payload writing contributed 641 ms, approximately 97% of the measured write stage. Manifest writing, header writing, temporary-file creation, preallocation, and rename were negligible by comparison. This identifies a concrete next target: reducing the number or cost of buffered memory-payload writes.

Both secure cold-path archives extracted correctly with both v1.9.6 and v1.9.7. The real IDE adaptive archive restored all 17,583 files and all 505,906,599 bytes. These checks support compatibility for the tested cases only, not a formal proof over all possible archives.

VI. Discussion and Threats to Validity

The longitudinal record supports the central framework claim: archive preparation benefits from coordinating change awareness, reuse, scheduling, memory staging, and stable output behavior. It also supports narrow claims about HIGV2 aggregation on the recorded v1.2.0 corpus, unchanged project-state reuse, and reduced duplicate reads in controlled v1.9.7 tests.

The data do not prove a new compression or cryptographic algorithm, a universal throughput advantage, or monotonic speed increase across releases. Synthetic tests, qualified release gates, unqualified volumes, anonymized large-project workloads, and real IDE workloads use different corpora and environmental states. The real IDE report records large storage-latency variance, and the Project-A utility record lacks complete baseline configuration metadata. Collapsing these measurements into one cross-version speed curve would be methodologically incorrect.

The current compatibility result covers two versions and measured corpora only. It does not establish behavior for symbolic links, sparse files, extended attributes, access-control lists, malformed archives, path-normalization edge cases, or every cross-platform filesystem combination. The security discussion identifies standard primitives but is not a third-party audit or a complete threat model.

Version v1.9.7 is the current first-major-release line, not a completed feature set. Future work includes stable-NVMe repetition of real-project benchmarks, broader platform validation, improved output-write telemetry and coalescing, adaptive large-file policies, formalized threat modeling, and wider compatibility matrices. New algorithm research should follow isolation of these systems-level bottlenecks rather than precede it.

VII. Conclusion

This paper presents Hig as a project-aware incremental archiving framework, not a claim of new cryptographic or compression primitives. The v1.0.1-v1.9.7 artifact record shows progression from file-level reuse to HIGV2 block planning, local coordination, watcher-assisted project state, and controlled cold-path reuse.

The strongest quantitative conclusion is deliberately narrow: v1.9.7 reduced controlled source-read worker time by 75.8% without changing measured archive size or tested cross-version extraction behavior. Other measurements demonstrate compact project archives and useful warm workflows while exposing output-I/O and memory trade-offs. By preserving negative results and environment qualification, the paper provides a defensible baseline for future Hig releases and for eventual investigation of genuinely new algorithms.

Source and complete paper

This article body follows the complete paper while omitting acknowledgments, conflict-of-interest statements, and references. Read the complete English paper on Haokir Research; the page also provides the PDF download.