Package model and storage

Dockpipe distinguishes what you author (source trees) from what you ship (tarballs and the local compile store). This doc describes that split and how dockpipe package compile / dockpipe build validate packages.

If you only need the common workflow, start with package-quickstart.md. This file is the detailed package/store reference for users configuring shared stores and authors packaging their own workflows. Paths under a project refer to your project, not a required clone of Dockpipe.

Canonical layout (mental model)

Dockpipe separates project build outputs, user installs, system installs and published artifacts. Source authoring and build validation are described alongside those roots below.

1. Project-local build output lives under <workdir>/bin/.dockpipe/. The default project package root is <workdir>/bin/.dockpipe/internal/packages/, or DOCKPIPE_PACKAGES_ROOT when explicitly overridden. dockpipe package compile materializes tarballs here: core/, resolvers/, workflows/ (dockpipe-workflow-* only). This is a working store for the project, not the global install root.

2. Global installs live under DOCKPIPE_GLOBAL_ROOT when set, otherwise the OS user data directory (for example ~/.local/share/dockpipe on Linux). Global packages use <global-root>/packages/ and global core uses <global-root>/templates/core/. There is no bin/ segment in the global root.

3. System installs may also ship package tarballs under a shared root (override with DOCKPIPE_SYSTEM_ROOT). Linux packages install compiled tarballs under /usr/share/dockpipe/packages/; runtime resolution checks project-local first, then per-user global roots, then system-shared roots.

4. Published / remote packages are versioned tarballs on a static origin or registry-like store. Each installable unit is a tarball (for example dockpipe-workflow-<name>-<ver>.tar.gz, dockpipe-resolver-…, dockpipe-core-…). The engine can load package tarballs directly when appropriate; it does not need every remote package unpacked into an authoring tree.

5. Source-level resolution applies when you point at authoring trees: repo workflows/, templates/, src/core/…, etc. — normal edit/run without packaging.

6. Build (dockpipe build / compile all) is considered successful only if:

   • compile order is core → resolvers → workflows,
   • each workflow and resolver package resolves a valid namespace (from package.yml, config.yml / resolver.yaml, or repo-root dockpipe.config.json packages.namespace as a default),
   • every depends entry in package.yml names a package already present in the compiled store (names from package.yml under core/, resolvers/, workflows/).

Runtime may still resolve workflows from project-local tarballs under bin/.dockpipe/internal/packages/workflows/, explicit local package sources from packages.sources in dockpipe.config.json, global packages under <global-root>/packages/workflows/, or packages.tarball_dir / release/artifacts when no on-disk workflow config wins; optional packages.namespace filters tarball choice when set.

State roots

Use the public dockpipe scope commands or package SDK helpers when scripts need state paths; do not hardcode another user's cache or state location.

Published packages and source projects

Normal installers provide CLI/core. Install optional packages from a published catalog with dockpipe package install or the launcher Marketplace. No Dockpipe source checkout or release workflow is required.

For a reproducible setup, use an immutable manifest URL and the selected package's checksum. When authoring your own packages, keep YAML, scripts and assets together and declare dependencies so another user can run them without your source tree.

Authoring vs execution (two modes, both supported)

Authors are not forced to pick one path: keep editing and running from source for low friction; use compile → package → release when you want a self-contained published artifact.

Authoring: workflow YAML vs resolver / runtime / strategy slices

A workflow is primarily config.yml (plus assets next to it). In that file you declare what to run and which profiles to merge — you do not embed full resolver or strategy trees inside the workflow directory (unless you choose colocated assets for scripts/images).

So one repo can ship workflows, resolvers, and runtimes / strategies / the rest of the spine via compile core ( templates/core or src/core ): list compile.workflows as the entry point. Resolver packaging uses the same roots as workflows plus flat src/core/resolvers and templates/core/resolvers when those exist. Runtimes still live under core; workflows only select them. Compile emits dockpipe-workflow-*, dockpipe-resolver-*, and core tarballs while package.yml records depends, namespace, and metadata. Step-by-step keys: ../workflows/workflow-yaml.md.

Local compile (dockpipe build / dockpipe package compile / dockpipe compile)

Materialize a project-local store under bin/.dockpipe/internal/packages/ without moving authoring trees.

Repo-root dockpipe.config.json (optional) — JSON with a compile object:

• workflows — array of repo-relative (or absolute) directories to scan for named workflow folders (each with config.yml). Same roots drive dockpipe package compile resolvers (nested …/resolvers/<name>/) plus flat src/core/resolvers and templates/core/resolvers when present.
• core_from — optional path override for compile core (same as --from).
• secrets (optional) — not secrets themselves; pointers for humans and tooling:
  • op_inject_template — repo-relative or absolute path to a mapping file for op inject (e.g. .env.op.template with op:// lines). dockpipe doctor reports whether that file exists when dockpipe.config.json is present in the current directory.
  • environment, environments — default selection and named resolver-backed environments, with explicit variable bindings. See Secret environments.
  • notes — free-text reminder (e.g. vault naming, policy).
• packages.sources (optional) — explicit local package sources used for resolution without publishing first:
  • { "kind": "store", "path": "C:\\Source\\dockpipe\\bin\\.dockpipe\\internal\\packages" } for a compiled package store root
  • { "kind": "tarball_dir", "path": "release/artifacts" } for a directory of dockpipe-*.tar.gz artifacts
  • these supplement local/global/system package roots and do not replace packages.tarball_dir, namespace filtering, or self-hosted S3/R2 package mirrors

If dockpipe.config.json is missing, compile uses built-in defaults for each omitted key. dockpipe init seeds a starter JSON. Maintainer package trees (for example packages/ in this repo) are not implied: add them explicitly under compile.workflows when you want them compiled.

Compile steps:

1. compile core — copies src/core (default when src/core/runtimes exists) or templates/core, or compile.core_from / --from, into packages/core/ and writes package.yml (kind: core). Omits top-level resolvers/, bundles/, and workflows/ from that copy so those slices stay separate packages.
2. compile resolvers — repeatable --from; default roots are compile.workflows, src/core/resolvers, and templates/core/resolvers when those directories exist. Pack roots under each --from directory include: resolvers/ (flat); packages/<group>/resolvers/; dockpipe/<group>/resolvers/ (e.g. dockpipe repo: agent, ide, secrets, cloud/storage, …). Every immediate child of each pack root that contains profile/ becomes one tarball (dockpipe-resolver-<name>-…) — the store still lists separate installable resolvers. src/core/resolvers stays a flat list (no nested resolvers/resolvers/). Strategies and runtimes are not packed from the same vendor folder as resolvers; keep lifecycle slices in compile core until a follow-up convention exists.
3. compile workflows — every config.yml under each --from root (recursive walk). With no config, the default is workflows/ when present. List maintainer roots (for example packages/) in compile.workflows when you want them compiled. dockpipe package compile bundles is an alias for compile workflows. Optional compile_hooks: in workflow/resolver config.yml is a list of shell strings run against the staged package copy after validation and before the tarball is written; Dockpipe exports DOCKPIPE_COMPILE_WORKDIR, DOCKPIPE_COMPILE_SOURCE_DIR, DOCKPIPE_COMPILE_STAGING_DIR, and DOCKPIPE_COMPILE_KIND so hooks can read source-only inputs while writing packaged outputs into the staged tree (e.g. go build, codegen, asset bundling).
4. compile all — runs core → resolvers → workflows and emits compiled runtime/security/image manifests, but does not run Docker builds.
5. dockpipe build — runs compile all --force, then prebuilds Dockerfile-backed image artifacts by default and records materialized receipts under bin/.dockpipe/internal/images/by-fingerprint/. dockpipe run can reuse those receipts before deciding whether a Docker build is needed. Use dockpipe build --no-images when you want manifest/package materialization only. dockpipe clean --dry-run previews the exact checkout-local bin/.dockpipe/ tree and logical bytes; ordinary dockpipe clean removes that complete disposable tree. dockpipe rebuild separately resets the resolved compiled store, including DOCKPIPE_PACKAGES_ROOT, through guarded target validation before building; ordinary clean never follows that override outside the checkout.

The runner checks compiled package-store roots before authoring CoreDir so you can opt in to the compiled store per workdir. Declare namespaces, depends, and metadata in the authored package.yml before compiling; do not rely on edits to generated package output.

Remote catalogs and user installation

dockpipe package catalog --remote <HTTPS manifest URL> returns JSON for the current execution platform (OS/architecture and, inside Flatpak, runtime identity). It accepts a latest pointer, release catalog, or explicit platform store manifest. Latest pointers use origin-relative manifest paths; release catalogs use release-relative stores.<platform>.manifest paths. Store manifests retain the existing schema-1 packages.core, packages.workflows, and packages.resolvers layout. The response includes the resolved store manifest, platform, and flattened packages. An explicit store URL must declare a compatible platform; native and Flatpak stores cannot be substituted.

dockpipe package install --remote <URL> --kind workflow --name <name> installs the selected archive under the user package root in the appropriate core, workflows, or resolvers subdirectory. DOCKPIPE_GLOBAL_ROOT selects another user root. The command prints JSON with its installed path. Use the catalog response's resolved manifest URL and --sha256 <digest> to pin a selection; a changed checksum is rejected before downloading the archive.

Installation requires HTTPS, restricts catalog references and redirects to the selected origin, bounds manifest/download/expanded sizes, and verifies SHA-256 and archive metadata. Catalog paths cannot traverse directories. Archives may contain only files and directories inside their declared package; links, duplicate members, conflicting metadata, and unsafe paths are rejected. An omitted package.yml version uses the store's version, matching the existing build contract. The completed archive is renamed into the user store only after verification. Failed verification preserves an existing package; publication atomicity follows the host filesystem's rename guarantees. Installing the same filename replaces that user copy. Other versions remain installed, and existing runtime resolution precedence still applies.

These checks establish integrity against the chosen HTTPS catalog; they are not independent publisher signatures. Configure origins you trust to supply executable packages. Installation does not execute package code, modify the system/Brew store, resolve dependencies automatically, or fetch anything during ordinary workflow resolution. Required resolvers must already be installed or installed separately. Normal installers supply only core; optional packages are selected explicitly from the remote catalog. The complete store is a separate opt-in download.

dockpipe package list --format json returns packages, warnings, and install_root, covering project, configured, user, and system package stores. Entries include name, version, kind, description, source, path, and removable. Only optional user-store archives are removable. This is an inventory of available packages; a displayed entry does not promise it wins every workflow's resolution precedence. The existing text listing remains scoped to the project store.

dockpipe package uninstall --path <absolute archive path> removes the selected optional workflow or resolver archive from the user store. Use a removable: true entry from the JSON inventory. Core, system/installer packages, project/configured stores outside that root, compiled directories, and symbolic links are rejected. Uninstall retains package state, settings, caches, other versions, and dependencies; it never recursively deletes a package tree. The response contains the removed path.

The launcher offers Uninstall on removable Installed entries and explains why managed entries are protected. Marketplace marks a matching name/kind/version Installed and disables duplicate installation; matching versions do not prove identical staging content. Cancel operation is visible only while a request is active and stops both catalog and inventory requests, with a bounded forced stop if the process does not exit.

Network boundary: install, not every run

HTTPS / CDN / registry traffic should be confined to explicit install (and publish) commands — e.g. dockpipe install core, dockpipe package catalog/install, dockpipe release upload. After artifacts are on disk, dockpipe run against local workflows or installed packages should not need network unless the workflow itself does (e.g. docker pull, API calls).

Package metadata may declare an OCI image reference as a hint/reference:

image:
  source: registry
  ref: ghcr.io/acme/tool@sha256:...
  pull_policy: if-missing

The package manifest does not become runtime truth. Compile folds the reference into the effective runtime/image manifests, and run consumes those compiled manifests. If the image is already local and valid, run stays local. If it is missing, run may pull only when the compiled pull policy and compiled network policy allow it.

Lifecycle: compile → package → release

1. compile — Validate workflow YAML and materialize a self-contained tree: copy the workflow and (as the implementation grows) pull in domain-specific assets referenced from source so the compiled directory is the single execution root for that package.
2. package — Archive that compiled tree (plus package.yml, checksums, optional lock metadata). dockpipe package build store turns the compiled store into gzip tarballs (one per core / workflow / resolver) and packages-store-manifest.json. dockpipe run --workflow can stream a workflow from dockpipe-workflow-<name>-*.tar.gz when config.yml in the archive sets namespace: and on-disk paths do not win (see packages.tarball_dir / packages.namespace in dockpipe.config.json). dockpipe package build core builds the templates/core artifact for dockpipe install core.
3. release — Upload tarball + manifest to your static origin; consumers install to pull it.

CI can chain the same steps locally: compile → package → release.

Workflow install and resolver dependencies

When dockpipe install (workflow package) exists end-to-end, installing a workflow should also install declared dependencies, primarily kind: resolver packages (depends / pins in package.yml), including a transitive closure where needed. Domain-specific scripts and assets belong inside the workflow package / compile output — not as a separate CDN hop for every file at run time. Resolver packages remain shared adapters (tool profiles, resolver-owned assets).

Package dependencies should remain package-shaped:

• depends names other package ids.
• requires_resolvers names resolver profile ids that must be available project-locally, globally, or from the install closure. Workflows that use logical resolver script ids such as scripts/dorkpipe/... should declare the owning resolver here instead of relying on workflow-package fallback.
• requires_capabilities names dotted capability ids for catalog/search and dependency checks.
• platforms declares which host platforms the package supports, and dependencies.host names required external executables with OS-specific install commands. Core preflights these before workflow execution. Tools bundled under the selected package’s assets/tooling/bin take precedence over host executables during preflight and in host-script PATH. Flatpak packages declare platforms: [flatpak] and supply runtime-compatible dependencies; native distribution installers are disabled in Flatpak.
• image may point at a normal OCI reference, ideally digest-pinned, but Docker layers remain in Docker/OCI registries rather than Dockpipe package tarballs.

Security metadata in package.yml should stay compatibility-only, for example compatible_security_profiles or requires_network: true if added later. Effective network, filesystem, process, and Docker enforcement settings belong only in compiled runtime manifests.

Distribution split (core and optional packages)

CLI + core provide the base application. Workflows and resolver packages are the optional features you browse, version and install.

1. Packages (installed, store-backed)

Packages are self-contained artifacts you fetch from an object store (e.g. Cloudflare R2 behind HTTPS) or another registry. They are building blocks for YAML workflows: full workflows, slices of templates/core (resolvers, runtimes, strategies, assets), or extra workflow tarballs from package installs.

• Default project compile layout: <workdir>/bin/.dockpipe/internal/packages/ — kept under internal/ so user-created and installable packages stay separate from other bin/.dockpipe/ state (runs, handoffs, CI). Override with DOCKPIPE_PACKAGES_ROOT (absolute path, or relative to workdir), e.g. vendor/dockpipe-packages if you want packages versioned in git without fighting a blanket .dockpipe/ ignore.

Suggested subdirectories (mirror authoring concepts; not all are required):

Metadata: each installable unit should include package.yml next to its payload (see dockpipe package manifest). Core fields: name, version, title, description, author, website, license, kind (workflow | resolver | core | assets | bundle). Optional namespace — same rules as workflow config.yml namespace: (lowercase label; reserved words like dockpipe, core, system are rejected).

Versioning: package.yml version is the independent release identity of that package; it does not need to match the CLI's repo-root VERSION. Bump a package when its published contents change. Generated workflow/resolver manifests inherit the nearest explicitly versioned owning package.yml, searching from the source directory up to --workdir. An explicit child version wins. External sources consult only their own manifest; loose sources without an owner fall back to repo-root VERSION (or 0.0.0). An inherited version change invalidates compiled child output even when the child's files have not changed. Core remains tied to the CLI version. Store assembly preserves package versions; its --version is only a fallback, not a bulk version override. Versions must be semver-shaped so tarball names and CDN paths stay stable.

Rich metadata (authoring & store discovery) — optional but recommended for workflow and resolver packages:

Use dockpipe package manifest to inspect example package metadata.

Compression: store objects are typically .tar.gz (or .tar.zst later) to keep bandwidth and R2 storage small; the CLI unpacks into the layout above. Binary-only packs are possible for asset-only packages if you add a small unpack step later.

CLI today: dockpipe install core pulls templates/core into templates/core (bootstrap path). dockpipe package compile workflow validates YAML and copies a workflow tree into bin/.dockpipe/internal/packages/workflows/<name>/ (see docs/cli-reference.md). Future dockpipe install package … will unpack registry artifacts and extend resolution order (project workflows/ + installed packages + embedded bundle).

Clone, education, and commercial packages

• package.yml may set allow_clone: true so dockpipe clone <name> can copy a compiled workflow package from bin/.dockpipe/internal/packages/workflows/<name>/ into workflows/<name>/ (or --to). This supports education and forking when the author opts in.
• allow_clone: false or omitting allow_clone means dockpipe clone refuses — appropriate for commercial or binary-only releases where the publisher does not grant a recoverable source tree.
• Optional distribution: binary documents that the published artifact is not meant for source recovery; publishers who need strong protection should ship only non-recoverable binaries (obfuscation, native modules, etc.) — Dockpipe cannot cryptographically enforce that; allow_clone is the explicit license bit for dockpipe clone.
• dockpipe package compile workflow writes allow_clone: true and distribution: source into a generated package.yml so local compiles stay cloneable unless you edit the manifest before release.

2. Uncompressed working tree (authoring / clone)

Not “packages” in this sense: the repo you edit every day.

• workflows/ (default project root), src/core/workflows/<name>/ (bundled examples in the dockpipe repo), legacy templates/<name>/ — YAML and assets you build or clone, commit, and run with --workflow as today.
• No package.yml required; this is normal development.

3. Disposable runtime and durable package state

bin/.dockpipe/ is the project-local disposable tree for generated state: host run records (bin/.dockpipe/runs/), workflow artifact roots, collision-safe package runtime directories, handoffs, optional demo stubs, caches, and installed package material under bin/.dockpipe/internal/. Step outputs: files are resolved under the selected workflow output/artifact scope. Every producer in this tree must tolerate its complete removal.

Durable package state is separate. dockpipe scope --package <owner-id> and dockpipe get package_state_dir resolve an owner-only OS state directory keyed by stable project identity plus the exact trimmed, case-preserving owner ID. DOCKPIPE_PACKAGES_ROOT, DOCKPIPE_GLOBAL_ROOT, checkout basename, remote URL, and the legacy sanitized scope do not relocate or identify it. Public suffixes are validated as data and cannot traverse outside that root.

On first public access, an undeclared third-party scope conservatively copies its complete validated legacy bin/.dockpipe/packages/<sanitized-scope> tree into durable state with byte inventory, source identity, owner-only permissions, interruption recovery, and atomic publication. Legacy bytes remain untouched; durable state wins after publication and divergence is reported without automatic merge. Maintained mixed packages instead declare exact package-owned migration aliases:

package_state:
  compatibility_import: package-owned
  owner_ids: [my-package, my-resolver]

That declaration means package code exhaustively selects durable cohorts and leaves caches, build output, scratch, and unresolved cohorts in package runtime. Dockpipe propagates the selected package manifest through package/workflow script context so the generic importer can match exact owner IDs without engine knowledge of maintained packages. It is not a retention label and does not authorize deleting or rewriting legacy state. Unknown scopes have no declaration and therefore import whole. There is no durable-to-legacy symlink or cross-project/name-folding heuristic.

Package scripts should use PackageRuntimeDir or shell SDK dockpipe_sdk path package-runtime <owner-id> ... for disposable products. DOCKPIPE_PACKAGE_STATE_DIR is accepted only when it equals the resolved durable directory or names an independently existing owner-only override outside the checkout; workdir refresh clears inherited state authority.

dockpipe clean --dry-run needs no project declaration. It deterministically reports the exact resolved checkout bin/.dockpipe/ target, logical bytes, file count, and whether removal would occur. Ordinary clean removes that whole tree after rejecting links/reparse points, special files, filesystem substitutions, and unsafe boundaries. A missing tree is a reported no-op. Durable state and external DOCKPIPE_PACKAGES_ROOT stores remain outside ordinary clean authority.

bin/.dockpipe/internal/packages/ is the default store for fetched or compiled package trees (workflows, core slices, assets) — the same conceptual layout whether content arrived as a .tar.gz or from dockpipe package compile workflow. Uncompressed authoring under workflows/ remains normal; compile validates and materializes into bin/.dockpipe/internal/... when you opt into the packaged path. Resolution order for --workflow is implemented in workflow_dirs.go (workflows/ → packages → legacy templates/ paths, etc.).

Image artifact metadata is not a package payload by default. See ../runtime/image-artifacts.md for the user-facing lifecycle. Use:

• bin/.dockpipe/internal/images/ for project-local image artifact indexes.
• bin/.dockpipe/internal/cache/images/ for project-local cached image artifact records.
• <global-root>/images/ for global image artifact indexes.

Compiled workflow tarballs may contain .dockpipe/image-artifact.json and per-step image manifests as inspectable compiled truth, but the local/global image index is separate from packages/ so packages, runtime manifests, image artifacts, and run records do not blur together.

4. Global installs

Execution environment and Marketplace artifacts

Native processes select OS-ARCH Marketplace stores. Inside Flatpak, Dockpipe reads /.flatpak-info and selects linux-ARCH-flatpak-RUNTIME_ID-RUNTIME_BRANCH. Installing Flatpak or using an atomic distribution does not switch a native CLI.

Store builds record this key as platform in packages-store-manifest.json. Release catalogs index stores by the same key. Direct/pinned store URLs must also match; unlabelled legacy stores remain native-only. Missing Flatpak artifacts fail explicitly instead of falling back to native packages.

Optional packages remain Dockpipe archives with dependencies supplied by the runtime or explicitly bundled and tested by authors. See the Flatpak build contract. The base installer continues to contain required core only.

User stores

Global installs are for user-wide Dockpipe extensions. They do not live under bin/.dockpipe because there may be no project checkout involved.

The user package store contains packages/core/, packages/workflows/ and packages/resolvers/. Legacy core-tree installs use templates/core/ instead. Use dockpipe package list --format json to see the user install root, installed archives and warnings for your environment. Do not infer a user-store location from the project compile directory.

For the current CLI remote-install contract, see Remote catalogs and user installation.

Resolution order (source and installed packages)

Project source workflows and configured source roots can override installed workflows of the same name. Package lookup checks the project store, configured package stores, the user store and system stores; explicit tarball sources and legacy/bundled fallbacks are also supported. Use --workflow-file to select an exact source file when name lookup is ambiguous.

This is current install/run behavior, not a future global-install proposal. The selected process platform also controls remote catalog compatibility, including Flatpak runtime identity as described above.

See also ../concepts/architecture-model.md and ../cli-reference.md (dockpipe package, dockpipe install).