Configuration

A dataferry config is a directory of YAML files, conventionally named ferry/. This chapter explains how the directory is read, what each file holds, how environment variables and secrets work, and which settings change what a run writes. Every key is listed in the config reference.

The ferry/ directory

ferry/
  ferry.yaml          # required: version and name, optional include:
  connections.yaml    # required once there is a flow: the databases
  runtime.yaml        # optional: state store, workers, memory, error policy
  flows/              # every *.yaml / *.yml below here, at any depth
    orders.yaml
    sales/
      refunds.yml

Commands read the directory given by --config <dir> (or the FERRY_CONFIG environment variable), and ./ferry by default. The directory must contain ferry.yaml: that file is what marks it as a config. Pointing --config at a file instead is an error that suggests its directory.

How the files are loaded:

• Each file holds only its own top-level key: ferry.yaml holds version:, name: and include:; connections.yaml only connections:; runtime.yaml only runtime:; a flow file only flows:.
• .yaml and .yml are both accepted. Having connections.yaml and connections.yml side by side is an error.
• Every *.yaml and *.yml file under flows/ is a flow file, at any depth. Other files (a README.md) are ignored, and so are files and directories whose names start with . (editor swap files). A flow file must contain flows:; write flows: [] to keep a placeholder. A config with no flows at all is an error.
• Order doesn’t matter. Files are read sorted by path, which only decides which error is reported first. Moving a flow to another file changes nothing.
• One YAML document per file. A second document (---) is an error, which keeps every error’s file and line exact.
• Names of connections and flows are lowercase letters, digits and _, starting with a letter. Flow names are unique across all files, and a duplicate is reported with both places.
• Unknown keys are errors at every level, so a typo fails validate instead of being ignored.

error[F0105]: runtime.wrokers: unknown key `wrokers`
 --> ferry/runtime.yaml:7:3
  |
7 |   wrokers: 4
  |   ^^^^^^^^
  = help: did you mean `workers`?
config is invalid: 1 error

YAML details that matter

The loader reads YAML 1.2, where on, yes and no are strings, not booleans (many editors still assume YAML 1.1). The grammar never uses them as keys: joins say keys:, not on:.

Expressions and other values must be YAML strings. A bare true, null or 5 where an expression is expected is an error, so YAML’s type guessing never silently changes an expression. Quote it ("true"):

error[F0105]: flows[products].to.products.columns.in_stock: expected an expression (a string), `{expr, type, ...}` or `{nest, columns, ...}`, found boolean `true`
  --> ferry/flows/products.yaml:12:21
   |
12 |           in_stock: true
   |                     ^^^^
config is invalid: 1 error

For the same reason, quote any expression that starts with *, [, {, &, !, |, >, ', ", % or @, or that contains : (a colon and a space). status: lower(status) needs no quotes; label: "'id: ' || id" does.

ferry.yaml

# Postgres → Postgres merge: one order summary row per order, with the
# customer looked up and the order's items nested as an ordered array.
version: 1
name: pg_order_summaries

# Escape hatch for files outside the convention (0003 §8). Here, flows kept in
# a legacy directory that is not flows/.
include:
  - legacy/*.yaml

• version: 1 is the version of the config grammar, and the only one.
• name: names the migration in the state store and in reports.
• include: is an escape hatch for files outside the convention.

include:

include: is a list of paths or globs (*, **), relative to the ferry/ directory. They may point outside it (../shared/connections.yaml), so several configs can share one file of connections. A config with flows still needs its own connections.yaml: when all its connections come from an include, keep a local one holding connections: [].

• An included file holds flows:, connections: or both, never runtime: (all runtime settings live in runtime.yaml).
• Its entries merge with the conventional files’ under the same rules: a duplicate flow or connection name is an error naming both files.
• A path that doesn’t exist, or a glob that matches nothing, is an error.
• A file reached twice (by the flows/ convention and a glob, or by two globs) is loaded once.
• Only ferry.yaml may have include:, so includes can’t nest.

connections.yaml

# `name` and `type` are fixed by 0003 (`description` is reserved). The other
# fields are each type's own, per 0004 §2. `schema` defaults to public.
connections:
  - name: shop
    type: mongo
    url: ${FERRY_GOLDEN_MONGO_URL}
    database: shop

  - name: warehouse
    type: postgres
    url: ${FERRY_GOLDEN_PG_URL}

Each entry has a name and a type, and every other key belongs to the type: PostgreSQL has url, password, schema, drop_indexes and foreign_keys; MongoDB has url and database. An unknown type is an error that lists the types the binary has. name, type and description are reserved in every entry (description has no meaning yet and is rejected).

validate also checks that each connection is used in a role its connector has:

error[F0303]: flows[products].sink: flow `products` writes to connection `shop`, but the `mongo` connector in this build has no sink
 --> ferry/flows/products.yaml:4:11
  |
4 |     sink: shop
  |           ^^^^
  = help: `mongo` provides: source
config is invalid: 1 error

Flows refer to a connection in two ways: an entity as <connection>.<entity> (split at the first dot, so app.public.orders is entity public.orders on connection app, which Postgres also resolves as table orders in the connection’s schema:), and a connection on its own in sink: and runtime.state.connection.

Environment variables and secrets

A config never needs to hold a credential. ${NAME} in a string value is replaced with the environment variable NAME:

• Only in connections.yaml entries (including connections from included files) and in runtime.state.path. runtime.state’s backend: and connection: are checked as written, before substitution, so ${...} there is an error too. A ${ anywhere in a flow is an error, because the config hash is taken before substitution: an environment-driven filter could change between a run and its resume, and the resumed run would write different rows.
• NAME is letters, digits and _, not starting with a digit. ${NAME} is the only form: there is no default value syntax and no ${file:...}. To use a secret mounted as a file, export it first: export PG_URL=$(cat /run/secrets/pg_url).
• A variable that isn’t set is an error that names it, without printing any other value.
• A variable set to the empty string is a value, the empty string. There is no way to write a literal ${ in a connection value.
• Substitution works on parsed values, so a variable can’t add YAML structure, and its value is never expanded again.

A ${ in a flow, and a variable that isn’t set:

error[F0108]: flows[products].to.product_dimensions.where: `${` in a flow: environment variables are substituted only in connections and runtime.state
  --> ferry/flows/products.yaml:23:16
   |
23 |         where: dimensions.width_cm > ${MIN_WIDTH}
   |                ^^^^^^^^^^^^^^^^^^^
   = help: write the value into the flow, so it is part of the config hash (0003 §9)
config is invalid: 1 error

error[F0108]: connections[warehouse].url: environment variable not set: CATALOG_PG_URI (connections.yaml:9:10)
 --> ferry/connections.yaml:9:10
  |
9 |     url: ${CATALOG_PG_URI}
  |          ^^^^^^^^^^^^^^^^^
  = help: export it before running dataferry; values are read only from the environment (0003 §9)
config is invalid: 1 error

Every substituted value, and the value of every secret field (Postgres url and password, Mongo url), is replaced with *** in everything dataferry prints and writes: diagnostics, plan output, logs and reports. That is a backstop for driver errors that echo a connection string. A secret field that holds a literal value is accepted with a warning, since config directories get committed:

warning[W0301]: connections[warehouse].url: holds a secret as a literal value, and config directories get committed
 --> ferry/connections.yaml:9:10
  |
9 |     url: ***
  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  = help: write `${NAME}` and export the value in the environment instead
`catalog` is valid: 2 connections, 1 flow, 1 warning

runtime.yaml

Every key is optional; without the file, every default applies.

runtime:
  # The state store is on the flow's sink connection, so data and the
  # partition's done-record commit in one transaction (TxnWithState, 0004 §8).
  # Any other connection here would make the flow fall back to
  # DeletePartitionThenInsert.
  state:
    backend: postgres
    connection: warehouse
  # Tuning: workers and memory_budget are not in the config hash (0003 §10.5).
  workers: 2
  memory_budget: 256MiB
  # …0006 has one bad item: the whole document goes to _ferry_dlq and the
  # run carries on. One dead letter is under max_errors, so `dataferry run`
  # still exits 0.
  on_error: dead_letter
  max_errors: 10

Behavioural settings change what a run writes or how it treats errors, so they are part of the config hash: change one and resume refuses the run. Tuning settings only change how fast, or how much at a time, so they are left out of the hash: a run that ran out of memory can be resumed with a bigger memory_budget, or with more workers. Partitions are stored when a run starts, so a changed partition_rows only applies to new runs.

The state store and TxnWithState

Where the state store is decides each flow’s idempotency strategy. A flow gets exactly-once TxnWithState only when state.connection is the flow’s sink:; every other flow falls back to DeletePartitionThenInsert, and plan and run say why. The catalog example with state: {backend: file} runs as:

  flow `products`: DeletePartitionThenInsert (the state store is a file (`backend: file`), not on "warehouse")

validate checks that state.connection names a connection whose type is backend: and that this type can hold the state store (only postgres in this release).

Per-flow settings

A flow may override a setting only if its effect is limited to that flow:

    # Ten documents in partitions of four gives three partitions, which is
    # enough to exercise resume. Allowed per flow because it only sizes this
    # flow's own work (0003 §6), and left out of the config hash (§10.5).
    runtime:
      partition_rows: 4

• partition_rows is honoured per flow.
• on_error and max_errors are accepted in a flow’s runtime: only with the same value as in runtime.yaml. The runner applies them to the whole run in this release, so plan refuses a flow that sets a different value (exit 4), rather than silently ignoring it. Per-flow budgets are planned for a later release.
• state, workers, memory_budget and the two source limits are process-wide, and setting them on a flow is an error.

Ordering flows

Flows with no relation between them run at the same time, sharing the workers. A flow starts only after every flow it waits for has committed all its partitions:

• the flows it names in depends_on:;
• any flow that writes a table it reads, on the same connection. When flow b reads reporting.customers and flow a writes customers to reporting, b waits for a.

Only the reading waits. Every flow’s entities are discovered, and its partitions planned, when the run starts, so a table one flow reads from another must exist by then, and its partitions are cut on the rows it holds at that moment (the first and last ranges are open-ended, so rows added later are still read, in uneven partitions).

flows:
  - name: order_summaries
    # customer_id points at reporting.customers. FKs are validated at
    # finalize anyway (A6), so this ordering is an optimization.
    depends_on: [customers]

depends_on: names must exist, appear once, not name the flow itself and not form a cycle. validate checks all of this without connecting. If a flow fails, the flows that wait for it don’t start, and the run reports them as blocked. Within one flow, the order of its tables comes from parent: (Transforms), not from depends_on:.

You rarely need depends_on: for foreign keys: a Postgres sink drops the foreign keys of existing target tables during a load and validates them at the end (PostgreSQL), so write order is an optimization, not what keeps the data consistent.

The config hash

A run pins a hash of the config, and resume and retry refuse to continue (exit 8) when it has changed. The hash is a SHA-256 of the config’s meaning:

• It is taken after defaults are filled in, so writing a default out (mode: insert) doesn’t change it.
• It is taken before ${ENV} substitution: url: ${PG_URL} is hashed as that text, so rotating a password doesn’t block a resume, and secret values never reach the hash.
• File layout doesn’t count: moving a flow to another file, renaming files, reordering flows or connections, editing comments or changing include: leaves it unchanged, as long as the model is the same.
• What has an order keeps it: columns: (the DDL column order), merge from: entries, named where: predicates and expect:.
• Tuning settings are left out: workers, memory_budget, partition_rows, source_max_connections and source_rows_per_sec, in runtime.yaml and in flows.

plan prints the hash on its first line, and the report records it.

Editor support

dataferry schema --out schemas/ writes a JSON Schema for each kind of file, including the options of every connector the binary has, so YAML editors can complete keys and flag mistakes as you type. See JSON Schemas.