The Problem

We established that plugin interfaces should be capability-shaped (open-ended routing with opaque payloads) rather than domain-shaped (closed enums with behavior baked into the protocol). But this creates a tension:

• We don't want [ubyte] blobs everywhere with no type safety
• We do want strongly-typed domain objects like Inventory, Task, Host
• How do we reconcile these?

The Key Insight: Two Layers

There are two distinct layers in the system:

1. Protocol Layer — How plugins are discovered, invoked, and communicate
2. Domain Layer — What data flows between components

The capability-shaped principle applies to the protocol layer. The domain layer should absolutely be strongly typed.

┌─────────────────────────────────────────────────────────────┐
│  Protocol Layer (capability-shaped)                         │
│  - Envelope { capability, correlation_id, payload }         │
│  - Handshake { capabilities[] }                             │
│  - Routing is by capability name, not message type enum     │
├─────────────────────────────────────────────────────────────┤
│  Domain Layer (strongly typed)                              │
│  - Inventory { groups[], hosts[] }                          │
│  - Task { name, module, args, when, ... }                   │
│  - ExecutionResult { status, stdout, stderr, changed }      │
│  - These ARE the payload contents                           │
└─────────────────────────────────────────────────────────────┘

How It Works in Practice

1. Define Domain Types in FlatBuffers

These are not protocol messages. They are data structures that can be serialized and passed as payloads.

// flatbuffers/domain/inventory.fbs
namespace shep.domain;

table HostVar {
  key:string;
  value:string;  // or use a Value union for typed values
}

table Host {
  name:string;
  address:string;
  port:int = 22;
  vars:[HostVar];
  groups:[string];
}

table Group {
  name:string;
  hosts:[string];      // host names, resolved later
  children:[string];   // child group names
  vars:[HostVar];
}

table Inventory {
  hosts:[Host];
  groups:[Group];
}

// flatbuffers/domain/task.fbs
namespace shep.domain;

table TaskArg {
  key:string;
  value:string;
}

table Task {
  name:string;
  module:string;           // e.g., "ansible.builtin.shell"
  args:[TaskArg];
  when:string;             // condition expression
  register:string;         // variable name to store result
  become:bool;
  become_user:string;
  delegate_to:string;
  ignore_errors:bool;
  retries:int;
  delay:int;
  tags:[string];
}

table TaskList {
  tasks:[Task];
}

2. Define Capability Request/Response Types

Each capability has well-defined input and output types. These are also in FlatBuffers, but they reference the domain types.

// flatbuffers/capabilities/parse.fbs
namespace shep.cap.parse;

import "domain/inventory.fbs";
import "domain/task.fbs";

// Input to the "parse" capability
table ParseRequest {
  file_path:string;
  content:[ubyte];        // raw file bytes if not using path
  format:string;          // "yaml", "json", "ini"
}

// Output from the "parse" capability
table ParseResponse {
  inventory:shep.domain.Inventory;
  tasks:shep.domain.TaskList;
  // errors, warnings, etc.
}

// flatbuffers/capabilities/schedule.fbs
namespace shep.cap.schedule;

import "domain/inventory.fbs";
import "domain/task.fbs";

table ScheduleRequest {
  inventory:shep.domain.Inventory;
  tasks:shep.domain.TaskList;
  limit:[string];         // host patterns to limit to
  tags:[string];          // tags to run
  skip_tags:[string];     // tags to skip
}

// A single unit of work to execute
table ScheduledTask {
  task:shep.domain.Task;
  target_host:shep.domain.Host;
  resolved_vars:[shep.domain.HostVar];
}

table ScheduleResponse {
  queue:[ScheduledTask];
}

// flatbuffers/capabilities/execute.fbs
namespace shep.cap.execute;

import "domain/task.fbs";
import "domain/inventory.fbs";

enum ExecutionStatus : byte {
  Ok = 0,
  Failed = 1,
  Skipped = 2,
  Unreachable = 3,
  Changed = 4
}

table ExecuteRequest {
  task:shep.domain.Task;
  host:shep.domain.Host;
  vars:[shep.domain.HostVar];
  check_mode:bool;
  diff_mode:bool;
}

table ExecuteResponse {
  status:ExecutionStatus;
  changed:bool;
  stdout:string;
  stderr:string;
  return_code:int;
  start_time:long;
  end_time:long;
  // facts gathered, if any
  facts:[shep.domain.HostVar];
}

3. The Envelope Carries Typed Payloads

The envelope's payload:[ubyte] contains a serialized domain type. The capability name tells you which type it is.

Capability "parse"  → payload is ParseRequest/ParseResponse
Capability "schedule" → payload is ScheduleRequest/ScheduleResponse
Capability "execute" → payload is ExecuteRequest/ExecuteResponse

This is not "untyped data and pray." The capability name is a type tag that tells the receiver exactly how to deserialize the payload.

4. Go Code Pattern

// Host sends a request to the "parse" capability
func (m *Manager) Parse(filePath string, content []byte) (*ParseResponse, error) {
    // Build the typed request
    req := buildParseRequest(filePath, content)

    // Send via envelope (capability-based routing)
    respBytes, err := m.Invoke("parse", uuid.New().String(), req)
    if err != nil {
        return nil, err
    }

    // Deserialize the typed response
    return parseParseResponse(respBytes), nil
}

// Plugin handles the request
func handleParseCapability(payload []byte) []byte {
    req := parseParseRequest(payload)

    // Do the actual parsing
    inventory, tasks := parseYAML(req.Content())

    // Build typed response
    return buildParseResponse(inventory, tasks)
}

The Flow: Parser → Scheduler → Executor

                    ┌─────────────────┐
  playbook.yml ───▶│  parse          │───▶ Inventory + TaskList
                    │  (plugin)       │
                    └─────────────────┘
                            │
                            ▼
                    ┌─────────────────┐
  Inventory ───────▶│  schedule       │───▶ [ScheduledTask]
  TaskList          │  (plugin)       │     (queue of work)
                    └─────────────────┘
                            │
                            ▼
                    ┌─────────────────┐
  ScheduledTask ───▶│  execute        │───▶ ExecuteResponse
                    │  (plugin)       │     (status, stdout, etc.)
                    └─────────────────┘

Each arrow is:

1. Host builds a typed request FlatBuffer
2. Host sends via Invoke(capability, correlationID, payload)
3. Plugin receives, deserializes the typed request
4. Plugin does work
5. Plugin builds typed response FlatBuffer
6. Host receives, deserializes the typed response

Why This Is Different From Domain-Shaped

Domain-shaped (wrong):

// The MESSAGE WRAPPER knows about all behaviors
enum MessageType { Parse, Schedule, Execute }
table Message {
  type:MessageType;
  parse_request:ParseRequest;
  schedule_request:ScheduleRequest;
  execute_request:ExecuteRequest;
}

Adding a new capability requires changing the Message table, the enum, and regenerating code. The host must have a switch statement for every type.

Capability-shaped (right):

// The MESSAGE WRAPPER only knows about routing
table Envelope {
  capability:string;    // "parse", "schedule", "execute", or any future one
  correlation_id:string;
  is_response:bool;
  payload:[ubyte];      // contains the typed request/response
}

Adding a new capability requires:

1. Define new domain types (if needed)
2. Define new capability request/response types
3. Write the plugin
4. No changes to Envelope, no changes to host routing code

The Principle Restated

• Protocol layer: capability-shaped (open-ended, data-driven routing)
• Domain layer: strongly typed (FlatBuffers schemas for all data)
• Payloads are typed by context: the capability name determines the type

The envelope is like HTTP — the Content-Type header (capability name) tells you how to interpret the body (payload). HTTP doesn't have a closed enum of message types; it has typed content with a runtime type tag.

Schema Organization

flatbuffers/
├── protocol/
│   ├── envelope.fbs      # Envelope, Handshake, Shutdown, Log
│   └── capability.fbs    # Capability declaration type
├── domain/
│   ├── inventory.fbs     # Host, Group, Inventory
│   ├── task.fbs          # Task, TaskList
│   └── vars.fbs          # Variable types, Value union
└── capabilities/
    ├── parse.fbs         # ParseRequest, ParseResponse
    ├── schedule.fbs      # ScheduleRequest, ScheduleResponse
    └── execute.fbs       # ExecuteRequest, ExecuteResponse

The host core imports protocol/ and domain/. Plugins import protocol/, domain/, and the specific capabilities/ they implement.

Type Safety in Go

The host can provide type-safe wrappers:

type ParseCapability interface {
    Parse(req *ParseRequest) (*ParseResponse, error)
}

type ScheduleCapability interface {
    Schedule(req *ScheduleRequest) (*ScheduleResponse, error)
}

type ExecuteCapability interface {
    Execute(req *ExecuteRequest) (*ExecuteResponse, error)
}

// Generic invoke still exists for extensibility
func (m *Manager) Invoke(capability, corrID string, payload []byte) ([]byte, error)

// But typed methods provide safety for known capabilities
func (m *Manager) Parse(req *ParseRequest) (*ParseResponse, error) {
    bytes, err := m.Invoke("parse", uuid.New().String(), req.Bytes())
    if err != nil {
        return nil, err
    }
    return ParseParseResponse(bytes), nil
}

New capabilities can use Invoke() directly with their own types. Known capabilities get type-safe wrappers. Best of both worlds.

Summary

Concern	Approach
Plugin discovery	Capability-shaped (declare capabilities at handshake)
Message routing	Capability-shaped (route by capability name string)
Protocol framing	Typed but minimal (Envelope, Handshake, Log, Shutdown)
Domain data	Strongly typed (Inventory, Task, Host, ExecuteResult)
Capability I/O	Strongly typed (ParseRequest/Response, etc.)
Extensibility	Add new schemas, no protocol changes

The envelope is not "put bytes in a bag." It's "put a typed FlatBuffer in a typed envelope where the capability name is the type discriminator."