Shep — Architecture Design
Not logged in

Status: Implementing

1. Overview

Shep is an Ansible-like orchestration engine implemented in Go. It focuses on a plugin-forward architecture using the HashiCorp plugin model for process-isolated extensions and FlatBuffers for efficient, cross-language payloads over stdin/stdout.

Primary goals:

Non-goals:

2. Constraints & Assumptions

3. High-level Components

Logging:

4. Plugin Model (HashiCorp go-plugin)

Design decisions:

Plugin discovery:

Handshake and lifecycle:

Handshake schema:

table Handshake {
  protocol_version:int;
  plugin_name:string;
  capabilities:[string];
}

Isolation & security:

Versioning & compatibility:

5. FlatBuffers Schema Strategy

Why FlatBuffers:

Schema management:

Style guide (per https://flatbuffers.dev/schema/):

Message design principles:

Example message flows (high-level):

6. Communication Flow and Data Exchange

Sequence (simple task run):

  1. Controller resolves inventory and selects target hosts.
  2. Controller ensures a Connection plugin is available for transport to the host.
  3. Controller prepares an ExecuteTaskRequest FlatBuffer and writes it to the Action plugin's stdin.
  4. Plugin runs task and writes ExecuteTaskReply to stdout.
  5. Controller reads the reply, records status, and handles logging.

7. Data Model

Ansible compatibility focus:

Deferred for later:

Controller stores runtime state in-memory; optional persistence can be added later (e.g., local SQLite for job history).

8. Security Considerations

Note: Detailed security mechanisms (secret handling, plugin signing, network isolation) are deferred beyond v1.

9. Developer Experience & Tooling

Note: Advanced developer tooling is deferred beyond v1.

10. Project Layout (recommended)

Recommended repo layout (root: shep/ is this repo):

shep/
├── cmd/
│   └── shep/                 # main CLI entrypoint
│       └── main.go
├── internal/                  # shared internal packages
│   ├── host/                  # plugin host logic (orchestration, handshake)
│   ├── fbs/                   # FlatBuffers-generated code (shared schemas)
│   ├── util/                  # misc helpers
│   └── test/                  # shared test helpers
├── plugins/                   # core plugins (all Go binaries)
│   ├── parser/                # parses Ansible-compatible YAML/JSON
│   │   ├── main.go
│   │   ├── plugin.go          # plugin-specific logic
│   │   ├── testdata/          # test inputs/outputs for parser plugin
│   │   └── README.md
│   └── ...                    # other plugins (file, command, etc.)
├── flatbuffers/               # FlatBuffers schema files
│   ├── shep.fbs
│   └── generate.go            # optional script to generate Go code
├── docs/
│   ├── cli.md                 # shep CLI usage
│   └── plugins.md             # overview of all plugins
├── testdata/                  # global testdata (playbooks, example inputs)
├── go.mod
├── go.sum
└── Makefile                   # build/test/documentation targets

11. CI/CD and Testing

12. Roadmap / Milestones

Phase 0 — Spec & PoC

Phase 1 — MVP (Local Execution)

Phase 2 — Remote Execution & Ecosystem

13. Migration and Compatibility

Note: Deprecation policy deferred beyond v1.

14. Appendix: Example FlatBuffers Schema

The following is a compact example of the kinds of messages used; expand and normalize in flatbuffers/.

namespace shep;

enum Status : byte { Ok = 0, Failure = 1 }

enum LogLevel : byte { Debug = 0, Info = 1, Warn = 2, Error = 3 }

table Metadata {
  protocol_version:int = 1;
  message_type:string;
  correlation_id:string;
}

table KeyValue {
  key:string;
  value:string;
}

table ExecuteTaskRequest {
  meta:Metadata;
  task_name:string;
  args:[KeyValue];
  host:string;
  connection_info:[KeyValue];
}

table ExecuteTaskReply {
  meta:Metadata;
  status:Status;
  exit_code:int;
  stdout:[ubyte];  // UTF-8 encoded
  stderr:[ubyte];  // UTF-8 encoded
  changed:bool;
  error_message:string;
}

table LogMessage {
  meta:Metadata;
  level:LogLevel;
  message:string;
  timestamp:long;  // Unix timestamp in milliseconds
}

root_type ExecuteTaskRequest;

Notes:

15. Next Steps

File: shep-arch.md