Xylem

Digital xylem that transports data through workflows with structure and support

Xylem is a workflow runner for composing command-line tools into repeatable, configuration-driven pipelines. A workflow declares ordered modules, inputs, variables, requirements, and execution options in JSON or YAML. Xylem prepares the modules and runs the same ordered pipeline for each input.

The workspace contains two Rust packages:

xylem-cli provides the xylem command.
xylem-lib provides configuration, module, interpolation, and execution types used by the CLI.

Execution model

Read a JSON or YAML workflow configuration.
Check declared host requirements.
Prepare local or remote modules and their environments.
Resolve workflow, module, and input variables.
Execute modules in declaration order for every input.

Inputs may run concurrently. Use the global --threads N option to bound the Rayon work managed by Xylem. Child tools can still create their own threads.

For the stable module-level interface, see the module contract.

Getting started

Install with Homebrew

On macOS or Linux, add the ORNL tap and install Xylem:

brew tap research-enablement/xylem https://code.ornl.gov/research-enablement/homebrew-xylem.git
brew install xylem
xylem --version

Download a Linux release

Download the current Linux binary from the Xylem releases, make it executable, and place it on PATH:

chmod +x ./xylem
sudo mv ./xylem /usr/local/bin/xylem
xylem --version

Build from source

Xylem requires the Rust toolchain declared by the workspace:

git clone https://code.ornl.gov/research-enablement/xylem.git
cd xylem
cargo install --locked --path xylem-cli
xylem --version

Run a workflow

Run a configuration with informational logging:

xylem -vv run --config path/to/config.json

Add one or more inputs at the command line when the configuration does not declare them:

xylem --threads 4 -vv run --config path/to/config.json --input first,second

Use xylem --help and xylem run --help for the exact options supported by the installed version.

Command-line reference

The xylem executable currently exposes the run command:

xylem [OPTIONS] [COMMAND]
xylem [OPTIONS] run [RUN OPTIONS]

Global options include verbosity flags and -t, --threads <N>. The worker count defaults to 10, must be greater than zero, and limits Xylem’s Rayon work rather than threads created by child programs.

The run command accepts:

--config <PATH> for a JSON or YAML configuration;
--input <LIST> for comma-delimited workflow inputs;
--dry-run to describe work without executing module commands;
--offline to prohibit remote acquisition;
--skip-verify-checksum and --skip-verify-requirements for explicit troubleshooting bypasses; and
reporter and output options reserved for structured result presentation.

When --config is omitted, Xylem looks for config.json in the current directory. Run the generated help for version-specific spelling and defaults:

xylem --help
xylem run --help

Workflow configuration

Xylem reads JSON and YAML configurations. A workflow describes requirements, zero or more inputs, shared variables, ordered modules, and execution options.

{
  "version": "1.0.0",
  "description": "Run a local greeting module",
  "requirements": ["sh"],
  "input": ["first", "second"],
  "variables": {
    "GREETING": "Hello"
  },
  "modules": [
    {
      "name": "Greeting",
      "type": "script",
      "programmingLanguage": "generic",
      "uri": "file:./modules/greeting",
      "template": {
        "command": "sh",
        "arguments": ["run.sh", "{{ GREETING }}", "{{ INPUT }}"]
      }
    }
  ]
}

Relative file: module paths resolve from the directory where Xylem is launched. Modules execute in array order for each active input.

Variables

Workflow variables are available to every module. Module variables override workflow values. Xylem then supplies reserved values such as MODULE_LIST, INPUT, and PARENT_DIRECTORY. Placeholders use {{ VARIABLE_NAME }} syntax.

Resolved values are passed to child processes with an _XYLEM_ prefix. For example, GREETING is available as _XYLEM_GREETING.

Options

Configuration options include dry_run, offline, skip_verify_checksum, and skip_verify_requirements. Equivalent CLI flags are combined with the configuration values.

See the repository’s module specification for the normative Xylem 1.x module contract.

Module contract

A module is either a script project or an executable binary. Each declaration has a name, type, URI, and optional variables and command template.

Local modules use file: URIs. Remote Python modules may use supported Git transports, while remote binaries use HTTP or HTTPS and require a SHA-256 checksum unless the user explicitly selects the unsafe troubleshooting bypass.

Command templates declare a command and its argument list. Prefer the array form for arguments because every item remains one process argument after variable interpolation. Templates may also select a Pixi or Conda environment and a supported container runtime.

The authoritative, versioned contract is MODULE_SPEC.md. It defines module fields, URI handling, variable precedence, environment managers, binary verification, and the conformance checklist.

Examples

The repository’s examples/ directory contains single-module and composed workflows.

Python with Pixi

The self-contained Python example demonstrates environment creation, nested variable interpolation, MODULE_LIST, and namespaced child-process variables. Install Pixi, then run it from the repository root:

cargo run --package xylem-cli -- -vvvv run --config examples/python-with-pixi/config.json

With Xylem installed:

xylem -vvvv run --config examples/python-with-pixi/config.json

The first execution creates the declared Pixi environment. Later executions reuse it.

Geospatial pipelines

The atmospheric-correction, pansharpening, and orthorectification examples show how ordered modules exchange paths through variables. Their module URIs are deployment-specific placeholders and must be updated before use.

Architecture

The CLI owns argument parsing, logging setup, top-level orchestration, and the process exit boundary. The library owns configuration parsing, module preparation, variable resolution, command construction, and workflow execution.

configuration (JSON or YAML)
            |
            v
        xylem-cli
            |
            v
        xylem-lib
       /    |     \
      v     v      v
 modules  inputs  variables
       \    |     /
        v   v    v
       child commands

ExecutionLimits and ExecutionContext provide the current worker-budget boundary. Modules execute in declaration order for an input, while multiple inputs and module preparation may use the configured Rayon pool.

See ARCHITECTURE.md for repository-level notes and the planning wiki for proposed contract changes.

Contributing

Clone the repository, install the development tools listed in the Makefile, and use the standard project checks:

make test
make lint
make book

Use make book-write to serve the book locally. The generated output is stored under docs/book and is not committed.

The root CONTRIBUTING.md documents the development environment, cross-platform testing, logging conventions, release tasks, and Homebrew tap maintenance.

Changes to public configuration or CLI behavior should update the relevant book page in the same merge request. CI builds the book for documentation changes, while successful main-branch or version-tag pipelines can publish it to the Xylem documentation site.

Homebrew formula updates are submitted to the organization tap as merge requests. Its required macOS and Linux jobs audit, build, install, test, and exercise upgrades before the formula can merge.

Keyboard shortcuts

Xylem documentation