The Execution Engine

Execution Engine

The execution engine is the core of Flow, responsible for running executables defined in YAML files.

Runner Interface

The execution system uses a runner interface pattern where each executable type implements:

type Runner interface {
	Name() string
	Exec(
		ctx context.Context,
		exec *executable.Executable,
		eng engine.Engine,
		inputEnv map[string]string,
		inputArgs []string,
	) error
	IsCompatible(executable *executable.Executable) bool
}

Current runner implementations include:

• Exec Runner: Shell command execution
• Request Runner: HTTP request handling
• Launch Runner: Application/URI launching
• Render Runner: Markdown rendering
• Serial Runner: Sequential execution of multiple executables
• Parallel Runner: Concurrent execution with resource limits

Workflows (Serial and Parallel)

The serial and parallel runners allow for composing complex workflows from simpler executables. Steps are defined with a RefConfig that supports inline commands or references to other executables:

type SerialRefConfig struct {
  Cmd string
  Ref Ref
  Args []string

  If string          // Expression to conditionally skip the step
  Retries int
  ReviewRequired bool // Prompts the user before continuing
}

Execution and result handling is managed by the internal engine.Engine interface. The current implementation includes retry logic, error handling, and result aggregation.

Execution Environment and State

Environment Inheritance Hierarchy:

Environment variables are provided to the running executable in the following order:

1. System environment variables (lowest priority)
2. Dotenv files (.env, workspace-specific)
3. Flow context variables (FLOW_WORKSPACE_PATH, FLOW_NAMESPACE, etc.)
4. Executable params (secrets, prompts, static values)
5. Executable args (command-line arguments)
6. CLI --param overrides (highest priority)

State Management

There are two ways state can be managed when composing workflows:

• Cache Store: Key-value persistence across executions with scoped lifetime. Values set outside executables persist globally; values set within executables are cleaned up on completion. Uses bbolt for cross-process storage.
• Temporary Directories: Isolated scratch space (f:tmp) with automatic cleanup and shared access across serial/parallel workflow steps.

File System Access

By default, the working directory is the directory containing the flow file that defines the executable. This can be configured using special prefixes: // (workspace root), ~/ (user home), f:tmp (temporary).

There is no automatic sandboxing. Executables inherit full user permissions. Flow assumes users understand their workflows’ scope and potential for system modification, prioritizing automation flexibility over execution isolation. Containerized execution is a planned future improvement.

See the executable guide and state management for usage details.

Performance and Caching

Flow uses eager discovery with multi-level caching to keep response times fast. Workspace scanning runs up front and is cached to disk, with in-memory caching layered on top for quick lookups. The cache is invalidated and refreshed via flow sync or the --sync flag.

Note to self: Some performance testing needed to validate sub-100ms discovery targets across large workspace trees.

For implementation details, see the DeepWiki reference.

Getting Values Into a Process

Everything reaches an executable as an environment variable. There are four sources:

Each can write to envKey or, when something needs a real file on disk, to outputFile, which is cleaned up after the run.

Arguments are separate from parameters and come from the command line, either positionally (pos: 1) or as flags (flag: name), with a type and an optional default:

flow build container -- v1.2.3 --publish=true

Resolution runs highest to lowest: a --param override, then the executable’s params, then its args, then the surrounding shell environment. Parent values propagate into children in serial and parallel workflows.

Paths get their own small vocabulary, which keeps definitions portable: // is the workspace root, ~/ is home, ./ is relative to the flowfile, $VAR expands from the environment, and f:tmp is a temp directory created once per run and cleaned up after.

Containers

A step can declare an image and run there instead of on the host:

exec:
  cmd: pytest -q
  container:
    image: python:3.13-alpine

The runtime is Docker or Podman, auto-detected unless pinned. The workspace mounts at /workspace by default, additional volumes use the same path prefixes as everything else, and the FLOW_* variables come along automatically. Secrets go in through a temporary --env-file rather than the command line, so they never appear in the container’s argv.

Conditions and State

Steps can be skipped with an if expression evaluated against os, arch, env, store, and a ctx object carrying the current workspace, namespace and flowfile paths. Conditions are the one place where a $("command") shell escape is available.

The store is a small key-value cache with two lifetimes, and the distinction matters more than it looks:

• Global, set outside a run with flow cache set, persists until cleared.
• Execution, set from inside an executable, is cleared automatically when the parent finishes. A serial workflow can pass state between its own steps without leaking it.

Two more things exist at the step level because workflows meet reality: retries: N, and reviewRequired: true, which pauses for a human before continuing.