Architecture decisions

Why Oren is the way it is. Every decision here has an alternative that was considered and rejected for a specific reason — the reason matters more than the decision, because it is what tells you when to revisit it.

Task and Implementation are separate documents

Decision. A contract (Task) declares what goes in, what comes out and which resources it needs. An implementation (Implementation) declares which image does the work and where each resource is mounted.

Why. v0 had a TaskBuilder that mixed the two sides:

builder.addVolumeMap("source", "/source")

"source" is a contract name; "/source" is an implementation path. Because they lived together, task definitions could only exist inside the CLI — and that is why the project could not have a catalogue.

The exact line. “I need a git repository” belongs to the task. “I mount it at /source” belongs to the implementation. “I need a GCP credential” belongs to the task. “It goes to /keys/gcp-credential.json” belongs to the implementation.

A consequence that pays off. Two implementations of the same contract are interchangeable with one line in the pipeline. Verified with analyze-commits: the shell and Node versions produce byte-for-byte identical output.

Dependencies are a single typed list

Decision. Directories, credentials and engines live in dependencies, each with a type from a closed vocabulary.

Rejected alternative. Separate fields (workspaces, secrets, capabilities), which was the first version of the spec.

Why. As three fields, the question “which implementation asks for less” does not exist. As one typed list, it does — and it is the basis of the ranking in the portal. It was the user who pointed this out, and the original proposal was worse.

Two axes per type.

Semantics — what it is (secret/gcp-service-account). The task author declares this.
Form — how it arrives (file, socket, string, directory). Derived from the type by the registry.

Deriving the form from the semantics avoids declaring it twice, and is what allows the CI generator to translate credentials correctly.

Floor and delta. The Task declares the minimum any implementation will need. The Implementation declares what it adds. Two implementations are only comparable because the floor is identical — what is left is an engineering choice.

Demonstrated in practice: no contract requires engine/docker. The previous build implementation used the host socket (privilege critical); the new one uses kaniko and asks for nothing beyond the floor.

`effects` has three values

Decision. none, reads-external, external.

	cacheable	safe to repeat
`none`	yes	—
`reads-external`	no	yes
`external`	no	no

Why. The execution engine caches aggressively. For a task that publishes an image, serving from the cache means the push did not happen — with the pipeline reporting success. Demonstrated: an npm-publish run twice received 409 Conflict from the registry on the second, proving it really re-ran.

reads-external came out of terraform-plan: it changes nothing, but reads remote state and real infrastructure. Caching would return a plan computed over a world that has already changed. The distinction does not change caching — neither is cacheable — but it changes the risk, and is what lets the portal show that planning is safe to repeat and applying is not.

Criterion. If I run it again with the same inputs and nothing happens, is the pipeline still correct? If not, it is external.

Asymmetry of mistakes. Marking external too often costs cache. Too rarely produces a silently absent deploy. When in doubt, external.

The protocol travels by file, not by argv

Decision. /oren/input.json and /oren/output.json. stdout and stderr are free for logging.

Rejected alternative. v0 passed JSON in argv and read the output from stdout delimited by markers.

Three problems that solved at once.

Escaping. JSON in argv crosses the entrypoint’s shell; $, quotes and newlines required escaping neither side got right.
Collision. stdout was a channel for logs and for data at the same time. A worker printing the delimiter corrupted the output.
Leaking. Tokens in argv show up in ps and in any echo of the command.

Unplanned gain. The protocol became runtime-agnostic, which made CI generation possible without changing anything. A worker reading $OREN_INPUT_PATH runs the same through the CLI or through a GitLab job.

An OCI image is the only runtime

Decision. Every worker runs as an OCI image. The language is a build-time choice, resolved by SDK and template.

Rejected alternative. Per-language runtimes (container, node, python…), which was in the spec at one point.

Why. It was the user who noticed: the “runtimes” I had modelled were not runtimes — they were languages. With a single execution path, npm’s transitive tree disappears from the lockfile, the second supply-chain surface disappears, and adding a language stops being a spec change.

The distinction is temporal.

	Build-time	Execution-time
What it is	SDK / template	OCI image + v1 protocol
Who sees it	whoever writes the worker	CLI, portal, lockfile

The SDK is distributed through the base image, not through a package manager. A “shell SDK” is four functions copied into the image — it needs no npm. That generalises to any language, including those with no convenient registry.

One image can serve several tasks

Decision. Implementations share an image and are distinguished by runtime.entrypoint.

Rejected alternative. A command input in the contract, so one task could do plan or apply.

Why. terraform-plan and terraform-apply have different effects, the plan is an output of one and an input of the other, and the artefact directory is mutable only in the first. A single contract would have to declare the worst case in effects and would lose the information that planning is safe to repeat.

But separate contracts do not require separate images: both point at oren-terraform-hashicorp and pick /worker/plan.sh or /worker/apply.sh.

Formulation. A subcommand is an entrypoint, and the implementation is what chooses it — not an input of the contract.

Nothing may depend on live in-memory state

Decision. Outputs are written to .oren/runs/<id>/ and re-read from there, even in a local run.

Why. It is what makes the local mode a particular case of the generated mode, where each step becomes a separate job and nothing survives between them beyond what was materialised. Without it, the “works on my machine, breaks in CI” class of bug would exist by construction.

What the invariant forbids. Passing live references between steps (container handle, socket, SDK object), non-serialisable outputs, and any CLI cache that is not in the lockfile or in a declared path.

An exception that needed handling. Outputs marked secret are not persisted — writing a credential in plain text into .oren/ would undo the guarantee that sensitive material does not touch the disk. Resume treats steps that depend on a secret value as non-resumable.

The execution engine is dedicated to Oren

Decision. Its own container (oren-engine-<version>) and volume (oren-cache-<version>), with the official Dagger image.

Why. Dagger runs one engine shared per machine. Without isolation, oren cache prune would wipe the cache of any other Dagger use of the user’s, and oren cache limit would rewrite their global configuration. Dagger is an implementation detail of Oren and should have no consequences outside it.

It is not a fork. Same image, same digest. What changes is the --name and the volume, chosen by the SDK via _EXPERIMENTAL_DAGGER_RUNNER_HOST.

Accepted risks. The variable is prefixed _EXPERIMENTAL_ and can change without notice. And the cache is not shared with other Dagger uses — common layers are downloaded twice.

Versioning. The container and volume carry the version because BuildKit’s cache format does not guarantee compatibility across engine versions. Old versions are removed when the new one starts, otherwise every update would leave an orphan volume.

The worker’s identity is immutable

Decision. oren.lock records a reference with a digest (repo@sha256:...) or the content digest of the build.

Why. Consent is anchored to the worker’s identity. Before the lockfile, that identity was the tag — and tags are mutable. Republishing alpine:3.20 with different content kept the fingerprint intact, and the authorisation granted to one image applied to another that nobody reviewed. The mechanism presented as protection against privilege escalation had exactly that hole.

For a local build, the digest is of the directory’s content: moving the worker does not change the identity, changing the content does.

Decision. .oren/consent.json, with a fingerprint over the worker’s identity + the set of dependencies.

Why a file. A prompt on every run becomes an automatic “yes”. Recorded, the team inherits the decision and a privilege escalation shows up as a diff in code review — reviewing privilege stops being a prompt and becomes part of the flow that already exists.

Why the fingerprint covers the dependencies. A new image, a new dependency, or a directory that became mutable produce a different key and require a new authorisation.

Fatigue. A low privilege dependency does not ask. Reading a directory does not deserve ceremony, and too much ceremony is what makes the mechanism stop being read.

In CI, the decision is to fail. With no terminal and no recorded authorisation, the pipeline stops. --yes exists but is an explicit opt-in. New privilege should not be granted on its own.

The registry is open between organisations

Decision. One organisation publishes a contract, any other may implement it.

Why. It is what stops the original problem from coming back at another scale: if every company wrote its own contract, we would have ten incompatible versions of build-docker-image.

Three rules that follow.

A namespace is required in order to publish. Without one, the second analyze-commits already collides.
A published version is immutable. Republishing breaks whoever depends on it, with no signal.
There is no deletion, only deprecation. We already know how the alternative ends.

A consequence about official implementations. They are reference, not recommended. Being “official” must not become weight in the ranking — otherwise the ranking stops measuring privilege and starts measuring provenance, and the incentive for third parties to publish disappears.

The portal stores the document, not a normalisation of it

Decision. The task.yaml is stored intact. Indexes for search and listing are derived from it.

Rejected alternative. Normalising into tables — one for inputs, another for dependencies, and so on.

Why. Normalising is reimplementing the spec inside the database, and then there are two definitions of what a task is. They diverge on the first schema change, and the divergence shows up as a bug for whoever consumes it. With the document intact, the spec remains the single source, the portal and the CLI validate against the same JSON Schema, and the indexes are a recreatable cache.

The interface renders the document — as the catalogue pages already do, generated from the task.yaml.

The portal serves resolution, never execution

Decision. The API answers three questions: resolve namespace/name@range to a contract, list a contract’s implementations with the dependency delta, and return the image and digest of an implementation.

What that forbids. Endpoints that run a pipeline, schedule, or keep run state as the source of truth. After oren install, a build cannot depend on the portal — that is what the lockfile guarantees, and what keeps the promise that the pipeline belongs to no platform.

A consequence about ordering. Design that API before the data model. The reverse path produces an API that mirrors the database instead of serving the client.

The portal trusts what the implementation declares

Decision. Publishing an implementation does not require verifying that the declared dependencies match what the image does.

Why. The defence already exists and is better: the CLI grants exactly what is declared. An implementation that uses the Docker socket without declaring engine/docker fails at run time. Under-declaring yields no advantage in the ranking — it yields a broken worker.

Verifying at the portal would require running the image in a controlled environment, which is expensive and still circumventable (the behaviour need only depend on input). Not worth it before there is real abuse.

Contracts and implementations in a single repository

Decision. workers/ is a single repository.

Why. They change together. Changing a type in the registry or adding a field to the spec requires revising every contract in the same move — separated, it would become eleven coordinated changes.

Exit criterion. When a worker has its own release rhythm, a different maintainer, or becomes a third-party dependency, it moves out. Monorepo → polyrepo is cheap in that direction; the reverse is not.

`implementation` is required in every step

Decision. The pipeline schema requires implementation. There is no default — not even when the contract has a single registered implementation.

Why. Omitting it was a break at a distance. With a single candidate, the step worked; the day someone published a second one — another person, another organisation, another repository — the pipeline started failing without anyone having touched it. It is the same non-determinism oren.lock exists to eliminate, surviving in a place we were not looking.

What runs has to be a function of what is written in the repository, not of what happens to be published at run time.

An alternative considered and refused. Defaulting to the implementation from the same organisation that published the contract. It fails on two counts: it collides with the decision that provenance carries no weight in the ranking (it would be the maximum weight — it does not change the order, it decides), and it does not solve the real case, because the same organisation can have several. techlite/analyze-commits has two, both low, both with zero additional dependencies: no available tie-break means anything. And an arbitrary tie-break is worse than none — it looks intentional.

A consequence for oren add. It started writing the step into oren.yaml rather than printing a snippet to copy. The choice remains human and made once; what changes is that it is recorded in the file instead of depending on someone having read the command’s output. With more than one candidate it asks, showing the cost of each; outside a TTY it requires --impl.

Accepted cost. Verbosity. Every step gains a line that, in the single-implementation case, carries no choice at all — it only makes it explicit.

The portal validates the document at publish time

Decision. ValidadorDeDocumentos refuses any document that does not satisfy the spec schema, before it enters the catalogue.

Why. The portal serves the document intact and every CLI validates it on the other side against the same schema. Accepting an invalid document is publishing a bomb: whoever resolves that version receives something their own tool refuses, and the error shows up on the consumer’s machine — far from whoever published it, and without them finding out.

The Javadoc already claimed that validation existed. It did not: only kind was checked, and the schemas sat in the resources unused.

Accepted cost. The library (networknt/json-schema-validator) brings Jackson 2 along, and the project uses Jackson 3. The two coexist — distinct packages — and the crossing happens at a single point, inside the validator. The alternative was reimplementing JSON Schema.

What this does NOT solve. The inverse case: a document valid for a new portal and invalid for an old CLI, because the spec schemas are strict. See Compatibility.

Once public, always public

Decision. A task or implementation published as public can never become private. The transition exists only in the other direction: private → public.

Why. Making private what was already public breaks every pipeline that references it, on machines nobody touched, and with no signal — tomorrow’s oren install fails resolving something that resolved yesterday. It is the same class of problem that version immutability avoids, with an aggravating factor: the version still exists, it merely stopped being reachable.

This holds even when whoever published it has every right to change their mind. The damage is not theirs.

A consequence for the paid model. Private contracts and implementations are the add-on it makes sense to charge for — an organisation publishes to the registry without exposing it to the world. But privacy is a choice at publish time, not a switch. An account that stops paying cannot have what was public taken down; what is lost is the ability to publish new things privately.

What this imposes on the schema. Visibility belongs to the published document, not to the organisation. If it were an organisation field, changing its plan would change the visibility of everything it has ever published at once — exactly what this decision forbids.

Postponed, not discarded. Billing, plans and invoicing are part of the plan — what does not make sense is facing them now. On their own they make a project the size of the whole registry (payment methods, subscription cycle, dunning, tax), and starting them before the registry is usable would delay what is still missing here.

What this decision guarantees is that postponing is not expensive: the “once public, always public” rule and visibility living in the document are what stop the billing model, when it exists, from needing to touch what has already been published.

The namespace is flat; teams separate permission, not names

Decision. There will be no sub-namespaces (acme/platform/build). An organisation with several teams either has one organisation per team (acme-platform, acme-data), or uses internal teams — which control who publishes what, without appearing in the identifier.

Why. The namespace appears in every reference of every pipeline that uses the task. One more level turns it into a reflection of the org chart — and org charts change. A team that restructures would change the reference of everything it has published, which collides head-on with once public, always public: the version would still exist, under a different name, and whoever referenced it would stop finding it.

A company’s internal structure should not leak into an identifier that third parties write in their own files.

Precedent. npm and GitHub separate the two: teams exist and control permission, but the package name stays flat (@acme/x). GitLab mixed them — subgroups change the path — and it is the recurring source of “where is X?” and of expensive migrations when a team moves.

What remains open, and is the real need. Today the role belongs to the whole organisation: whoever can publish, publishes everything. “Only the platform team publishes acme/build” is per-resource permission, and needs no folder at all to exist. That is what is missing from the model — not one more level in the name.

Grouping to navigate — 200 tasks in a large organisation — is presentation. It is solved with tags and filters in the interface, without touching the identifier.

Accepted cost. An organisation that wants per-team namespaces needs to create separate organisations, and members repeat across them. That is more administration; in exchange, no internal reorganisation breaks anyone’s reference.

Maintainers restrict per task; their absence restricts nothing

Decision. A task can have maintainers. If it does, only they publish new versions of it. If it does not, the organisation role applies.

Why. It is the real need behind the question about teams — “only the platform team publishes acme/build” — solved without touching the identifier. See the namespace is flat.

It restricts, never grants. The organisation role remains the floor: a maintainer who is not a publisher publishes nothing. The per-task restriction takes away from whoever could already, and does not give to whoever could not. Without that, assigning a maintainer would become a side channel for granting access, bypassing the admin.

Having no maintainer means no restriction, not “nobody may”. The opposite would lock, on the migration’s first run, every task that already exists — and nobody would publish anything until a maintainer was assigned to each. For the same reason, removing the last maintainer returns the task to the organisation: a task whose only maintainer left the company cannot stay locked forever.

By task name, not by version. A maintainer belongs to a task over time. Tying it to a version would require reassigning on every publication — right at the moment the permission needs to already exist.

The task does not need to exist. Reserving who looks after build before the first version is the normal case: it is how a team guarantees nobody publishes the v1 in their place.

Accepted cost. Management is person by person. For a team of fifty, it is tiring. Teams as a grouping come later — and they come as sugar on top of this, without changing the semantics: a team expands into its accounts at check time.