AGENTS.md - shep
Not logged in

Shep is an Ansible-like orchestration engine implemented in Go. It focuses on a plugin-forward architecture using the HashiCorp plugin model for process-isolated extensions and FlatBuffers for efficient, cross-language payloads over stdin/stdout.

Primary goals:

Quick Reference

make build          # Build shep and all plugins
make test           # Run all tests
make gen-schemas    # Regenerate FlatBuffers Go code
make prettier       # Format markdown files (run after editing .md)
make timeline       # Show recent fossil history

Project Structure

cmd/shep/              # Host CLI application
plugins/               # Plugin implementations
  basic/               # Reference plugin (echo capability)
  inventory/           # Inventory parser plugin
internal/
  fbs/shep/            # Generated FlatBuffers code (do not edit)
  host/                # Plugin manager and lifecycle
  protocol/            # Message framing and FlatBuffer builders
  logger/              # Logging infrastructure
flatbuffers/           # FlatBuffers schema definitions
  domain/              # Domain types (Inventory, Host, Group, etc.)
  capabilities/        # Capability request/response types
docs/                  # Architecture documentation
testdata/              # Test fixtures

Unbreakable Rules for Agents

refs

Architecture Rules

Interface Design: Two Layers

There are two distinct layers in the system with different design principles:

Layer Principle What it contains
Protocol Capability-shaped Envelope, Handshake, routing metadata
Domain Strongly typed Inventory, Task, Host, ExecuteResult, etc.

The capability-shaped principle applies to the protocol layer. The domain layer should be strongly typed.

Protocol Layer (capability-shaped)

The protocol handles plugin discovery, routing, and framing. It should NOT know about specific behaviors.

// GOOD: open-ended capability routing
table Envelope {
  protocol_version:int = 1;
  capability:string;      // "parse", "schedule", "execute" - open-ended
  correlation_id:string;
  is_response:bool;
  payload:[ubyte];        // typed by context, not by enum
}

table Capability {
  name:string;
  version:int;
}

table Handshake {
  protocol_version:int = 1;
  plugin_name:string;
  capabilities:[Capability];
}

Domain Layer (strongly typed)

Domain objects flow as payloads. They are strongly typed FlatBuffers schemas.

// flatbuffers/domain/inventory.fbs
table Host {
  name:string;
  address:string;
  port:int = 22;
  vars:[HostVar];
}

table Inventory {
  hosts:[Host];
  groups:[Group];
}

// flatbuffers/domain/task.fbs
table Task {
  name:string;
  module:string;
  args:[TaskArg];
  when:string;
  become:bool;
}

Capability Request/Response Types (strongly typed)

Each capability has well-defined input and output types:

// flatbuffers/capabilities/parse.fbs
table ParseRequest {
  file_path:string;
  content:[ubyte];
}

table ParseResponse {
  inventory:Inventory;
  tasks:TaskList;
}

The Payload Is Not Untyped

The capability name acts as a type discriminator:

capability="parse"    → payload is ParseRequest or ParseResponse
capability="schedule" → payload is ScheduleRequest or ScheduleResponse
capability="execute"  → payload is ExecuteRequest or ExecuteResponse

This is like HTTP: the Content-Type header tells you how to interpret the body. HTTP doesn't have a closed enum of message types.

The Wrong Way (baking behavior into protocol)

Do NOT design the protocol wrapper to know about all behaviors:

// BAD: closed enum ties host to specific message types
enum MessageType : byte { Parse, Schedule, Execute }

// BAD: one field per behavior
table Message {
  type:MessageType;
  parse_request:ParseRequest;
  schedule_request:ScheduleRequest;
  execute_request:ExecuteRequest;
}

This forces schema changes and host code changes for every new capability.

Adding a New Capability

  1. Define domain types (if new)
  2. Define capability request/response types
  3. Write the plugin
  4. No changes to Envelope, no changes to host routing code

Schema Organization

flatbuffers/
├── protocol/         # Envelope, Handshake (minimal, capability-shaped)
├── domain/           # Inventory, Task, Host (strongly typed)
└── capabilities/     # ParseRequest/Response, etc. (typed per capability)

See docs/shep-plugin-modeling.md for detailed examples.

FlatBuffers Generated Code vs Go Structs

Plugins define their own Go structs (e.g., Inventory, Host, Group in plugins/inventory/parser.go) that mirror the FlatBuffers-generated types in internal/fbs/shep/domain/.

Why duplicate? FlatBuffers-generated types are low-level accessors for serialized binary data, not idiomatic Go structs:

Plugin structs serve as the working model for parsing and business logic. FlatBuffers types are only used at serialization boundaries:

YAML file → Go structs (parse/manipulate) → FlatBuffers (serialize) → wire
wire → FlatBuffers (deserialize) → Go structs (use) → output

This separation keeps plugin code idiomatic and testable while FlatBuffers handles the wire format.

FlatBuffers Style Rules

Follow the official FlatBuffers style guide: https://flatbuffers.dev/schema/

Example:

namespace shep;

enum Status : byte { Ok = 0, Failure = 1 }

table ExecuteTaskRequest {
  task_name:string;
  exit_code:int;
  is_changed:bool;
}

Adding a New Plugin

  1. Create plugins/<name>/main.go with handshake and message loop
  2. Define FlatBuffers schemas if needed:
    • Domain types in flatbuffers/domain/<name>.fbs
    • Capability types in flatbuffers/capabilities/<name>.fbs
  3. Update Makefile to build the plugin and compile schemas
  4. Run make build to generate code and build
  5. Write tests in plugins/<name>/*_test.go

See plugins/basic/ for a minimal example or plugins/inventory/ for a complete implementation with parsing and FlatBuffer serialization.

Testing

make test                           # Run all tests
go test -v ./plugins/inventory/...  # Test specific package
go test -v ./... -run TestName      # Run specific test

Integration tests in internal/host/ build plugins to temp directories and test the full lifecycle (handshake → invoke → shutdown).