BOP

BOP is my Bespoke Orchestration Program that is a personal experiment of coordination agentic AI workflows. the system contemplates multiple bounded agents coordinated by a top layer agent which aims to manage context, token economy, and hallucinations. agents use RAG to ensure source data provenance and knowledge store.

under wraps for now

Current Status

Initial development complete. See PRD below.

Boosting Open Playbook (BOP)

v0.5 Product Requirements Document

1. Executive Summary

Boosting Open Playbook (BOP) is a grounded workflow orchestration and reasoning system designed to operationalize reusable concepts across technical, analytical, and decision-making workflows.

BOP combines:

The system is intentionally designed to avoid:

BOP is fundamentally:

It is NOT intended to function as:

2. Product Vision

Vision Statement

Create a trusted cognitive operating system that:

The long-term goal is to create:

3. Core Architectural Philosophy

3.1 Canonical State Principle

All persistent operational knowledge must exist in explicit system state.

Canonical state includes:

Agents may not maintain hidden operational memory.

The system should avoid:

3.2 Provenance-First Design

Every meaningful output must be traceable.

All recommendations, conclusions, and generated artifacts should reference:

The system prioritizes:

over convenience or autonomy.

3.3 Grounded Reasoning Only

Agents may only reason from:

Agents may NOT:

Unsupported information must be labeled explicitly.

3.4 Explicit Uncertainty Principle

The system must explicitly distinguish between:

When evidence is insufficient, the system should:

The system should never fabricate certainty.

3.5 Human Augmentation Principle

BOP exists to augment human cognition and workflows.

The system should:

The system should NOT:

3.6 Constrained Orchestration Principle

Structured workflows are preferred over unrestricted autonomy.

The system should favor:

The system should avoid:

3.7 Retrieval Quality Over Model Sophistication

System quality depends more on:

than on:

The system should prioritize:

before increasing orchestration complexity.

3.8 Token Efficiency Principle

BOP should use LLM tokens efficiently while preserving:

Token efficiency is important but subordinate to:

The system should minimize unnecessary token usage through:

The system should avoid:

Token optimization must never:

3.9 Operationalization Principle

System quality depends more on:

than on:

The system should prioritize:

before increasing orchestration complexity.

4. System Goals

4.1 Primary Goals

4.2 Secondary Goals

4.3 Non-Goals

BOP is NOT intended to:

5. LLM-Agnostic Architecture

5.1 Architectural Principle

Agents must remain provider-neutral.

An agent is defined by:

An agent is NOT defined by:

Model selection is an implementation detail handled by the Model Adapter Layer.

5.2 Model Adapter Layer

The Model Adapter Layer abstracts provider-specific APIs.

Initial provider:

Future providers may include:

Responsibilities:

Suggested interface:

model.generate(
    messages=messages,
    response_schema=schema,
    tools=allowed_tools
)

5.3 Provider Capability Registry

The system should explicitly track provider capabilities.

Example:

providers:
  openai:
    supports_tools: true
    supports_json_schema: true
    supports_file_search: true

  local_model:
    supports_tools: false
    supports_json_schema: weak
    supports_file_search: false

The runtime should route tasks appropriately.

5.4 Standard Agent Specification

All agents should use provider-neutral specifications.

Suggested structure:

id: retrieval_agent
role: Retrieve relevant Playbook concepts.

responsibilities:
  - retrieve concepts
  - return evidence
  - avoid unsupported inference

inputs:
  - task
  - workflow_context
  - domain_filters

outputs:
  - retrieved_concepts
  - evidence
  - confidence

rules:
  - Do not infer beyond retrieved sources.
  - Return "No supporting evidence found" if retrieval fails.

allowed_tools:
  - retrieve_concepts

5.5 Standard Input Schema

All agents should receive a standardized task envelope.

{
  "task_id": "...",
  "agent_id": "ux_analyzer",
  "user_request": "...",
  "context": {},
  "artifacts": [],
  "retrieved_concepts": [],
  "constraints": [],
  "prior_outputs": []
}

5.6 Standard Output Schema

All agents should return structured outputs.

{
  "facts": [],
  "assumptions": [],
  "inferences": [],
  "recommendations": [],
  "open_questions": [],
  "evidence": [],
  "confidence": "LOW|MEDIUM|HIGH",
  "next_actions": []
}

This schema is central to hallucination mitigation.

6. Workflow Orchestration Architecture

6.0 Workflow Lifecycle State Machine

All workflows should follow an explicit lifecycle state machine.

Suggested lifecycle:

DRAFT
↓
PLANNED
↓
AWAITING_APPROVAL
↓
RUNNING
↓
BLOCKED
↓
AWAITING_USER_INPUT
↓
AWAITING_APPROVAL
↓
COMPLETED
↓
ARCHIVED

Failure states:

FAILED
CANCELLED
INVALID

The workflow lifecycle should remain:

Workflow state transitions should be explicitly logged.

6.1 Supervisor Philosophy

The Supervisor should function primarily as:

The Supervisor should NOT function as:

The Supervisor should:

6.1b Supervisor Execution Semantics

The Supervisor should follow an explicit orchestration procedure.

Suggested execution sequence:

Interpret request
↓
Classify workflow complexity
↓
Estimate token budget
↓
Determine required evidence
↓
Determine required agents
↓
Construct workflow DAG
↓
Validate constraints
↓
Present workflow plan
↓
Await user approval
↓
Execute subtasks
↓
Validate outputs
↓
Resolve or escalate conflicts
↓
Synthesize final output
↓
Propose storage opportunities
↓
Finalize workflow

The Supervisor should remain:

The Supervisor should avoid:

6.2 Supervisor Planning and Approval Rule

In v0.2, the Supervisor must always show its execution plan before starting a workflow.

The Supervisor must wait for user approval before invoking agents or executing the workflow.

This approval-first behavior is intentional. It prioritizes:

Future versions may relax this requirement once the system proves reliability through repeated successful workflows.

The plan should include:

6.2b Workflow Complexity Classification

Before constructing a workflow DAG, the Supervisor should classify workflow complexity.

Suggested workflow modes:

Direct Response

Use when:

Characteristics:

Retrieval-Augmented Workflow

Use when:

Characteristics:

Orchestrated Workflow

Use when:

Characteristics:

Mutation Workflow

Use when:

Characteristics:

The Supervisor should prefer the lowest-complexity workflow capable of safely accomplishing the user goal.

6.3 Token Budget Management

The Supervisor should estimate token usage before workflow execution.

The estimate should include:

The Supervisor should manage token usage automatically unless the user provides a hard cap.

Token budget estimates should remain visible to the user in the workflow plan.

The system should record actual token usage after workflow execution so that future estimates can improve.

6.3a Supervisor Execution Loop (v0.5)

A minimal, deterministic loop for planning, approval, and execution:

PLAN → SHOW PLAN → AWAIT APPROVAL → EXECUTE SUBTASKS → VALIDATE → SYNTHESIZE

Pseudo-flow:

plan = supervisor.plan(user_request)
show(plan)
approved = await_user_approval()
if not approved: abort
state = init_state(plan)
for subtask in plan.subtasks:
  result = execute(subtask, state)
  state = update(state, result)
  if result.status == 'FAILED': handle_failure(state)
final = supervisor.synthesize(state)
return final

All steps must be logged and inspectable.

6.3b Workflow Modes (Decision Heuristics)

The Supervisor must choose the simplest viable mode:

Heuristics (initial):

If no artifacts and low ambiguity → Direct
If concepts likely improve answer → Retrieval-Augmented
If multiple artifacts/steps needed → Orchestrated
If persistent changes requested → Mutation

The Supervisor should estimate token usage before workflow execution.

The estimate should include:

The Supervisor should manage token usage automatically unless the user provides a hard cap.

Token budget estimates should remain visible to the user in the workflow plan.

The system should record actual token usage after workflow execution so that future estimates can improve.

The token estimation model should eventually learn from:

Token optimization remains subordinate to:

6.4 Agent Registry Constraint

In v0.2, the Supervisor may only assign tasks to predefined agents in the agent registry.

The Supervisor may NOT create ad hoc one-off agents during normal workflow execution.

However, the Supervisor may suggest that a new reusable agent should be created when repeated workflow needs indicate that one would be useful.

At the user’s explicit direction, the system may help define a new reusable agent specification.

New reusable agents should be added through an explicit design/update workflow, not created silently during execution.

6.5 Conflict Handling

In v0.2, material conflicts must be shown to the user.

The Supervisor should not silently resolve material disagreements between:

When a material conflict is detected, the Supervisor should present:

The Supervisor should then ask the user how to proceed.

Future versions may introduce a more formal conflict-resolution framework that allows the Supervisor to resolve low-risk conflicts automatically.

6.5b Subtask Contract Model

All subtasks should follow an explicit contract structure.

Suggested structure:

subtask:
  id:
  purpose:
  assigned_agent:
  dependencies:
  inputs:
  expected_outputs:
  token_budget:
  evidence_required:
  approval_required:
  retry_policy:
  timeout_policy:
  status:

Subtasks should remain:

Subtasks should not mutate global workflow state directly.

Global state mutation should occur through controlled orchestration transitions.

6.5c Retry and Degradation Model

The system should support graceful degradation when workflows encounter failures.

Potential failure categories:

Suggested retry/degradation sequence:

Failure detected
↓
Retry if retry policy allows
↓
Attempt fallback strategy
↓
Request clarification if needed
↓
Escalate to user
↓
Gracefully terminate workflow if unresolved

The system should avoid:

Fallback behavior should remain visible and logged.

6.6 Workflow Logging Format

In v0.2, workflows should be logged as YAML.

YAML is preferred initially because it is:

Workflow logs should capture:

Future versions may convert workflow logs to JSON or another structured representation if needed for performance, tooling, or interoperability.

6.7 Specialized Agent Philosophy

Agents should remain narrowly scoped and domain-specific.

Examples:

The system should avoid generalized mega-agents.

Specialized agents improve:

6.8 Workflow-Centric Design

The system should prioritize workflows over isolated prompts.

Workflows should:

The workflow DAG should be treated as a first-class system object.

6.9 Human-in-the-Loop Design

Human oversight is considered a core architectural feature.

The system should:

6.10 Trust Accrual Principle

System trust depends on:

Trust should be treated as a primary system asset.

The system should prioritize preserving trust over maximizing autonomy.

7. High-Level System Components

7.1 Supervisor Agent

Responsibilities:

The Supervisor is the only agent with visibility into:

The Supervisor should remain lightweight and orchestration-focused.

The Supervisor must operate within strict rules:

7.2 Retrieval Agent

Responsibilities:

The Retrieval Agent must remain retrieval-only.

It should never perform speculative reasoning.

7.3 Analysis Agents

Examples:

Responsibilities:

Outputs must include:

7.4 Generation Agents

Examples:

Outputs must:

7.5 Storage Agent

Responsibilities:

The Storage Agent must NEVER auto-store concepts.

All persistent mutations require explicit approval.

8. Shared Infrastructure

8.0 Interaction Layer Philosophy

BOP should remain interface-flexible.

The orchestration and reasoning layers should remain decoupled from the user interface layer.

Potential interaction surfaces include:

The system should expose stable orchestration primitives independent of interface.

The UX layer should:

The core orchestration architecture should not depend on any specific LLM chat interface.

8.1 Playbook Repository

Canonical knowledge store.

Technology:

8.1a Concept Deduplication (v0.5)

On storage:

Decisions:

User confirmation required for merge decisions

8.2 Vector Store

Semantic retrieval layer.

Responsibilities:

The vector store is NOT the source of truth.

It is a retrieval acceleration layer only.

8.2b Retrieval Ranking Strategy

[existing content preserved]

8.2c Retrieval Stopping Rule (v0.5)

Simple bounded retrieval:

Algorithm:

8.2d Context Packing Strategy (v0.5)

Context structure:

context:
  full_concepts: []
  summary_concepts: []

Domain Relevance

Boost concepts matching:

Example:

A UX workflow should strongly prefer UX concepts over generic development concepts.

Provenance Confidence

Boost concepts with:

Concept Freshness

Moderately boost:

while avoiding excessive recency bias.

Concept Specificity

Prefer concepts that are:

over vague conceptual guidance.

Workflow Context Matching

Boost concepts matching:

Historical Reuse Performance

Track:

This creates a continuously improving retrieval layer.

Retrieval Budgeting

The retrieval system should optimize for:

The system should:

Suggested Ranking Formula

Conceptually:

FinalScore =
  SemanticSimilarity
  + DomainWeight
  + WorkflowContextWeight
  + ProvenanceWeight
  + HistoricalPerformanceWeight
  + FreshnessWeight
  + SpecificityWeight

Weights should remain configurable and observable.

Retrieval Explainability

The retrieval layer should expose WHY concepts were retrieved.

Example:

This improves:

Semantic retrieval layer.

Responsibilities:

The vector store is NOT the source of truth.

It is a retrieval acceleration layer only.

8.3 Runtime Layer

The Runtime Layer should:

The Runtime Layer should remain deterministic and inspectable.

8.4 Validation Layer

Raw model outputs should never be trusted directly.

The Validation Layer should validate:

8.4a Validation + Repair Loop (v0.5)

On failure:

  1. detect error (invalid JSON, missing fields, missing evidence)
  2. attempt repair (max 2 retries)
  3. revalidate
  4. if still failing → escalate or fail subtask

Repair constraints:

8.5 Logging and Audit Trail

The system should log:

This is required for:

9. Hallucination Mitigation Framework

9.1 Grounding Rules

Agents may only use:

Any unsupported claim must be labeled.

9.2 Provenance Requirements

Every meaningful claim should identify:

9.3 Assumption Tracking

Assumptions must:

9.4 Supervisor Validation

The Supervisor should validate:

before synthesis.

9.5 Human Escalation

When evidence quality is insufficient:

The system should never fabricate certainty.

10. Example Workflows

10.1 UX Review of iOS App (PRD + Code) — v0.5 Canonical Workflow

Goal

Evaluate an iOS app’s UX by comparing PRD intent with actual code behavior, grounded in Playbook concepts, and produce actionable recommendations and optional follow-up actions.

Workflow Mode

Inputs

Outputs

Agent Set (v0.5)

Optional (future / gated):

High-Level DAG

parse_prd ─┐
            ├─> retrieve_ux_concepts ─┐
parse_code ─┘                         ├─> ux_analysis ──> synthesize_results
                                       └─> gap_analysis ─┘

Optional branch:

build_and_run_simulator → observe_runtime → ux_runtime_analysis

Subtasks (Canonical Contracts)

1) parse_prd

2) parse_code

3) retrieve_ux_concepts

4) gap_analysis

5) ux_analysis

6) (optional) runtime branch

7) synthesize_results

8) propose_concepts (optional)

Token Budget (Example Heuristic)

Validation & Repair

Conflict Handling

If conflicts detected (e.g., PRD vs runtime vs concepts):

Next-Step Options (Post-Synthesis)

Supervisor should present explicit options:

Logging (YAML)

Capture:

Notes / Constraints

11. Initial Technical Architecture

Initial Stack

Language:

Storage:

Retrieval:

Execution:

LLM:

Initial Execution Model

Initial orchestration should remain:

Avoid parallelization initially.

Correctness and observability matter more than speed.

12. Suggested Repository Structure

playbook/

Suggested workflow log location:

workflows/logs/\.yaml

Suggested reusable agent specification location:

agents/\.yaml

13. Success Metrics

Reliability

Metrics:

Reuse

Metrics:

Efficiency

Metrics:

Trust

Metrics:

14. Risks

14.1 Over-Orchestration

Too many agents create:

Mitigation:

14.2 Hallucination Cascades

Speculative outputs propagating downstream.

Mitigation:

14.3 Knowledge Pollution

Low-quality concepts degrading retrieval quality.

Mitigation:

14.4 Context Explosion

Too many retrieved concepts reducing signal quality.

Mitigation:

15. Roadmap

v0.5

Goals:

v0.6

Goals:

v0.3

Goals:

v0.4

Goals:

v0.5

Goals:

16. Gap Analysis — What Prevents This From Being v1.0

16.0 Supervisor Maturity Gap

Current weakness:

The Supervisor is still primarily described at the conceptual orchestration level.

Needed for v1.0:

The v0.2 Supervisor should intentionally remain conservative:

v1.0 should relax these constraints only after reliability is demonstrated.

16.1 Missing Formal Workflow State Model

Current weakness:

Workflow state is described conceptually but not formally specified.

Needed for v1.0:

This is one of the biggest missing architectural pieces.

16.2 Missing Provenance Data Model

Current weakness:

Provenance requirements exist philosophically but not structurally.

Needed for v1.0:

Without this, provenance becomes inconsistent.

16.3 Missing Retrieval Ranking Strategy

Current weakness:

Retrieval exists but ranking quality is unspecified.

Needed for v1.0:

Poor retrieval quality will eventually dominate system quality.

16.4 Missing Concept Lifecycle Management

Current weakness:

Concepts are stored but not managed over time.

Needed for v1.0:

16.5 Missing Agent Evaluation Framework

Current weakness:

No formal mechanism exists to measure agent quality.

Needed for v1.0:

16.6 Missing Tool Governance Model

Current weakness:

Tool usage is insufficiently constrained.

Needed for v1.0:

Current weakness:

Tool usage is insufficiently constrained.

Needed for v1.0:

16.8 Missing Observability Infrastructure

Current weakness:

Logging exists conceptually but observability is incomplete.

Needed for v1.0:

16.9 Missing Prompt Governance

Current weakness:

Prompt construction and lifecycle management are underspecified.

Without governance, prompts risk becoming:

Prompt drift can create:

Needed for v1.0:

Centralized Prompt Repository

Prompts should exist as explicit versioned assets.

Suggested structure:

prompts/
  shared/
    grounding_rules.md
    provenance_rules.md
    output_schema.md

  agents/
    retrieval_agent.md
    ux_analyzer.md
    supervisor.md

  workflows/
    resume_workflow.md
    ux_review_workflow.md

Prompt Layering Model

Prompt composition should follow explicit hierarchy.

Suggested layering:

SYSTEM LAYER
↓
WORKFLOW LAYER
↓
AGENT LAYER
↓
TASK LAYER
↓
USER INPUT

Higher layers should constrain lower layers.

Shared Grounding Rules

Core hallucination mitigation rules should be centralized.

Examples:

Agents should inherit these rules consistently.

Prompt Versioning

Prompts should support:

Prompt changes should be treated similarly to:

Prompt Testing

The system should support:

Prompt Observability

The system should log:

This is required for:

Prompt Injection Resistance

The architecture should eventually support:

This becomes increasingly important as workflows grow more complex.

16.10 Missing Scalability Philosophy

Current weakness:

The system assumes small-scale sequential execution.

Needed for v1.0:

16.11 Missing UX Philosophy for Human Interaction

Current weakness:

Human interaction model is underdefined.

Needed for v1.0:

17. Final Philosophy

BOP should prioritize:

The system should behave more like:

and less like:

The long-term value of BOP will depend primarily on: