Generating Output Data - Cyberdesk Docs

Overview

When a Cyberdesk workflow reaches a terminal state, you often want structured data as output - not just “task completed.” This page explains how Cyberdesk transforms various types of captured data into structured JSON output that matches your schema.

This transformation process works whether your workflow runs from scratch or uses cached trajectories. Dynamic tools like focused_action and extract_prompt always capture fresh data, even during trajectory replay.

The Output Data Pipeline

During workflow execution, Cyberdesk captures data from multiple sources: Observations - Dynamic data extracted by:

focused_action - Context-aware decisions and extractions
extract_prompt - Vision-based data extraction (sync, batch, or run-scoped)

Runtime Values - Variables set by:

copy_to_clipboard - Clipboard-based extraction
upsert_runtime_values - Direct variable setting (from async extract_prompt)
focused_action - Variable assignment via {{variable}} syntax

At terminal completion, the Transformation Agent combines all observations and runtime values to generate structured output_data JSON matching your schema.

Components of Output Data

1. Output Schemas

Define the structure of your expected output data when creating a workflow. Example Schema:

{
  "type": "object",
  "properties": {
    "patient_mrn": {
      "type": "string"
    },
    "vital_signs": {
      "type": "object",
      "properties": {
        "blood_pressure": {
          "type": "string"
        },
        "heart_rate": {
          "type": "number"
        },
        "temperature": {
          "type": "number"
        }
      }
    },
    "medications": {
      "type": "array"
    },
    "lab_results": {
      "type": "array"
    }
  }
}

Format: output_schema must be valid JSON Schema. The one special passthrough marker is {"only_runtime_values": true}, which skips transformation and returns the runtime values map as-is.

Purpose:

Defines expected structure and data types
Guides the transformation agent
Enables validation and type checking
Makes output predictable and consistent

Optional but Recommended: You don’t need an output schema for every workflow, but having one ensures consistent, structured data that’s easy to parse and use in downstream systems.

2. Observations

Captured during workflow execution by multiple tools:

focused_action - Dynamic observations and decisions
extract_prompt - Vision-based data extraction (sync, batch, or run-scoped)

How They’re Created: Via focused_action:

"Use focused_action to extract the patient vital signs from the screen and 
note: blood pressure, heart rate, temperature, and oxygen saturation."

Via extract_prompt (any mode):

"Take screenshot with extract_prompt='Extract patient vital signs as JSON: 
{blood_pressure, heart_rate, temperature, oxygen_saturation}'"

What Gets Stored:

The timestamp when the observation was made
The extracted observation text, or an error when extraction fails
The screenshot captured at that moment
Source metadata such as cached/extraction mode details
In focused actions and some extraction paths, the instruction that produced the observation

Example Observation Entries: From focused_action:

{
  "timestamp": "2024-01-15T14:30:22Z",
  "instruction": "Extract patient vital signs",
  "observation": "Blood Pressure: 120/80 mmHg, Heart Rate: 72 bpm, Temperature: 98.6°F, O2 Saturation: 98%",
  "screenshot": "base64_image_data",
  "cached": false
}

From extract_prompt (with source field):

{
  "timestamp": "2024-01-15T14:30:25Z",
  "instruction": "screenshot.extract: Extract medications as JSON array",
  "observation": "[{\"name\": \"Aspirin\", \"dosage\": \"81mg\"}, ...]",
  "screenshot": "base64_image_data",
  "cached": false,
  "source": "screenshot_extract",
  "zoom_bounding_box": [100, 200, 500, 600]
}

Key Characteristics:

Dynamic: Re-evaluated on every run, even in cached workflows
Multi-source: From focused_action, extract_prompt (sync/batch/run)
Contextual: Includes the instruction for clarity
Visual: Screenshot attached for verification
Structured: Consistently formatted for transformation

3. Runtime Values

Set during workflow execution via:

focused_action with {{variable}} syntax
copy_to_clipboard with key names
extract_prompt with process_async (batch or run) and upsert_runtime_values

How They’re Created: Via Focused Action:

"Use focused_action to find the invoice number on screen and save it as `{{invoice_number}}`"

Via Copy to Clipboard:

"Triple-click the account number and use copy_to_clipboard with key name 'account_number'"

Via Async Extraction (batch or run scoped):

"Take screenshot with extract_prompt='Extract customer_id and order_total as 
runtime variables using upsert_runtime_values' and process_async='batch'"

"Take screenshot with extract_prompt='Extract customer_id and order_total as 
runtime variables using upsert_runtime_values' and process_async='run'"

What Gets Stored:

{
  "invoice_number": "INV-2024-001",
  "account_number": "1234567890",
  "customer_id": "CUST-5678",
  "order_total": 1250.00
}

Key Characteristics:

Immediate: Available as {{variable_name}} in subsequent workflow steps
Flexible: Can be strings, numbers, or simple objects
Persistent: Included in final output transformation
Reusable: Can be used multiple times in workflow

Runtime values are perfect for:

IDs and reference numbers needed later in workflow
Values used in file naming or path construction
Data that determines workflow branching
Key metrics that appear in multiple places in output

4. The Transformation Agent

At the end of a terminal run, if an output schema is defined, the transformation agent converts all captured observations and runtime values into structured JSON.

This includes success, task_failed, cancelled, and error runs. If a run ends early, Cyberdesk still attempts to generate schema-shaped output_data from the captured data and any schema defaults.

Input to Transformation Agent:

Your defined output schema
All observations (from focused_action and extract_prompt)
All runtime values (from copy_to_clipboard, upsert_runtime_values, focused_action)

Process:

Transformation Agent receives:
  - Output Schema: {patient_mrn: string, vitals: object, ...}
  - Observations: [
      {instruction: "Extract vitals", observation: "BP: 120/80, HR: 72..."},
      {instruction: "screenshot.extract: medications", observation: "[{name: 'Aspirin'...}]"}
    ]
  - Runtime Values: {patient_mrn: "MRN12345", patient_age: 45}
  
Transformation Agent analyzes and maps:
  - "patient_mrn" → use runtime value "MRN12345"
  - "vitals" → extract from observation "Blood pressure 120/80..."
  - "medications" → extract from observation "Medications: aspirin..."
  - "lab_results" → use extraction result
  
Transformation Agent outputs:
  {
    "patient_mrn": "MRN12345",
    "vital_signs": {
      "blood_pressure": "120/80",
      "heart_rate": 72,
      "temperature": 98.6
    },
    "medications": ["aspirin 81mg daily", "lisinopril 10mg daily"],
    "lab_results": [...]
  }

System Prompt (simplified):

You are a data transformation assistant. Extract structured data from 
observations and runtime values, formatting according to the provided JSON schema.

Be precise and only include information actually captured. If a required field 
cannot be determined, use null or appropriate default value.

Runtime values are extracted using copy_to_clipboard or focused actions - they 
often represent important identifiers that should be included in output.

Key Characteristics:

Intelligent: Maps observations to schema fields semantically
Comprehensive: Includes runtime values automatically
Type-aware: Converts strings to numbers, arrays, etc. as needed
Validated: Ensures output matches schema structure

Output Data Optimization Features

The transformation agent supports two powerful optimization features to reduce LLM-induced lossiness and improve efficiency.

1. Direct Runtime Values Output

If you’ve already collected exactly what you need via runtime variables and don’t need any LLM transformation, you can skip the transformation step entirely. How to Use: Set your output schema to:

{"only_runtime_values": true}

This immediately returns the runtime values map as-is, without any LLM processing. Example:

"Navigate to order details page.

Triple-click Order ID and use copy_to_clipboard with key name 'order_id'
Triple-click Customer ID and use copy_to_clipboard with key name 'customer_id'

Take screenshot with extract_prompt='Extract all order line items, shipping 
details, and payment info. Store as runtime variables using upsert_runtime_values' 
and process_async='run'

Navigate to next page..."

Runtime Values Collected:

{
  "order_id": "ORD-2024-5678",
  "customer_id": "CUST-1234",
  "line_items": [...],
  "shipping_details": {...},
  "payment_info": {...}
}

Output Schema: {"only_runtime_values": true} Result: The runtime values are returned exactly as collected, with zero lossiness. When to Use:

✅ All data is already in runtime values
✅ You want zero LLM-induced modifications
✅ Data structure is already exactly what you need
✅ You’re using focused actions or extract_prompt extensively to set runtime values

Benefits:

⚡ Instant - no LLM transformation call
🎯 Zero lossiness - exact values preserved
💰 Cheaper - no transformation tokens
🔒 Predictable - no chance of LLM hallucination

2. Automatic Runtime Value Referencing

The transformation agent is smart enough to reference existing runtime values directly instead of regenerating them, reducing lossiness and token usage. How It Works: When transforming observations into output data, the transformation agent can automatically detect when a value should come from runtime values instead of being regenerated. It uses internal template syntax to reference these values, which are then deterministically substituted with the exact values. Example: Runtime Values Collected:

{
  "long_transcript": "This is a very long transcript of a customer call that spans multiple paragraphs and contains detailed conversation history...",
  "customer_id": "CUST-5678",
  "order_items": [
    {"sku": "WIDGET-A", "name": "Premium Widget", "price": 29.99},
    {"sku": "GADGET-B", "name": "Super Gadget", "price": 49.99}
  ]
}

Observations:

[
  {
    "instruction": "Extract customer sentiment",
    "observation": "Customer was satisfied with service, rated 9/10"
  }
]

Output Schema:

{
  "type": "object",
  "properties": {
    "customer_id": {"type": "string"},
    "sentiment": {"type": "string"},
    "rating": {"type": "number"},
    "full_transcript": {"type": "string"},
    "order_summary": {"type": "object"}
  }
}

Final Output (automatic optimization):

{
  "customer_id": "CUST-5678",
  "sentiment": "satisfied",
  "rating": 9,
  "full_transcript": "This is a very long transcript of a customer call that spans multiple paragraphs and contains detailed conversation history...",
  "order_summary": {
    "items": [
      {"sku": "WIDGET-A", "name": "Premium Widget", "price": 29.99},
      {"sku": "GADGET-B", "name": "Super Gadget", "price": 49.99}
    ],
    "item_count": 2
  }
}

The transformation agent automatically references customer_id, long_transcript, and order_items from runtime values instead of regenerating them, ensuring exact values are preserved. Benefits:

🎯 Zero lossiness - exact runtime values preserved
⚡ Faster - less content to generate
💰 Cheaper - fewer output tokens
🔒 Reliable - no chance of LLM typos in long values

This optimization happens automatically - you don’t need to do anything special. The transformation agent is instructed to use internal referencing when appropriate.

Complete Flow Example

Healthcare Workflow

1. Define Output Schema:

{
  "type": "object",
  "properties": {
    "patient_mrn": {
      "type": "string"
    },
    "date_of_birth": {
      "type": "string"
    },
    "vital_signs": {
      "type": "object",
      "properties": {
        "blood_pressure": {
          "type": "string"
        },
        "heart_rate": {
          "type": "number"
        },
        "temperature": {
          "type": "number"
        }
      }
    },
    "current_medications": {
      "type": "array"
    },
    "latest_lab_results": {
      "type": "object",
      "properties": {
        "test_date": {
          "type": "string"
        },
        "results": {
          "type": "array"
        }
      }
    }
  }
}

2. Workflow Execution:

"Log into EHR system with {username} and {$password}.

Navigate to patient search and search for {patient_name}.

In the patient demographics section:
- Triple-click on the Medical Record Number and use copy_to_clipboard 
  with key name 'patient_mrn'
- Triple-click on Date of Birth and use copy_to_clipboard with key name 'date_of_birth'

Navigate to Vitals tab.
Use focused_action to extract current vital signs and note: blood pressure, 
heart rate, and temperature.

Navigate to Medications tab.
Take screenshot with extract_prompt='Extract all current medications as JSON 
array with fields: name, dosage, frequency' and process_async='batch'

Navigate to Lab Results tab.
Take screenshot with extract_prompt='Extract most recent lab results as JSON 
with test_date and results array' and process_async='run'

Continue with documentation workflow."

3. Data Captured During Execution: Runtime Values (from copy_to_clipboard):

{
  "patient_mrn": "MRN12345",
  "date_of_birth": "1985-03-15"
}

Focused Observations (from focused_action):

[
  {
    "timestamp": "2024-01-15T14:30:22Z",
    "instruction": "Extract current vital signs",
    "observation": "Blood Pressure: 120/80 mmHg, Heart Rate: 72 bpm, Temperature: 98.6°F",
    "screenshot": "...",
    "cached": false
  }
]

Extraction Results (from extract_prompt):

[
  {
    "scope": "batch",
    "prompt": "Extract all current medications as JSON array",
    "result": [
      {"name": "Aspirin", "dosage": "81mg", "frequency": "once daily"},
      {"name": "Lisinopril", "dosage": "10mg", "frequency": "once daily"}
    ]
  },
  {
    "scope": "run",
    "prompt": "Extract most recent lab results",
    "result": {
      "test_date": "2024-01-10",
      "results": [
        {"test": "CBC", "value": "Normal", "reference": "Normal"},
        {"test": "Glucose", "value": "95", "reference": "70-100"}
      ]
    }
  }
]

4. Transformation (at run completion): The transformation agent receives all captured data and the schema, then produces:

{
  "patient_mrn": "MRN12345",
  "date_of_birth": "1985-03-15",
  "vital_signs": {
    "blood_pressure": "120/80",
    "heart_rate": 72,
    "temperature": 98.6
  },
  "current_medications": [
    {"name": "Aspirin", "dosage": "81mg", "frequency": "once daily"},
    {"name": "Lisinopril", "dosage": "10mg", "frequency": "once daily"}
  ],
  "latest_lab_results": {
    "test_date": "2024-01-10",
    "results": [
      {"test": "CBC", "value": "Normal", "reference": "Normal"},
      {"test": "Glucose", "value": "95", "reference": "70-100"}
    ]
  }
}

This structured output is now available via the API and can be used by downstream systems!

Run-Scoped Extraction with Runtime Variables

Run-scoped extractions have a unique capability: they can both store runtime variables AND provide comprehensive observations.

The Extraction Agent Loop

When using process_async="run", the extraction becomes a proper agent with access to upsert_runtime_values: System Prompt:

You are an extraction assistant. You have two capabilities:

1. Call upsert_runtime_values to store specific extracted values that should 
   be available throughout the workflow as `{{key_name}}` placeholders.

2. Provide final observations as text describing what you see on screen.

You can do BOTH: store specific values AND provide observations, or just do 
one or the other. Your final text message will be recorded as the extraction result.

Example: Store Key Fields + Comprehensive Extraction

"Navigate to order details page.

Take screenshot with extract_prompt='Extract order_id and customer_id and 
store them as runtime variables using upsert_runtime_values. Then extract 
complete order details including all line items, shipping info, payment 
details, and order history as detailed JSON.' and process_async='run'

Continue generating shipping label using `{{order_id}}` and `{{customer_id}}` in the label.

The full order details will be available in final output."

What Happens:

Extraction Agent Analyzes Screenshot

Calls upsert_runtime_values:

{
  "order_id": "ORD-2024-5678",
  "customer_id": "CUST-1234"
}

Provides Detailed Observation:

{
  "order_id": "ORD-2024-5678",
  "customer_id": "CUST-1234",
  "line_items": [
    {"sku": "WIDGET-A", "quantity": 2, "price": 29.99},
    {"sku": "GADGET-B", "quantity": 1, "price": 49.99}
  ],
  "shipping": {
    "address": "123 Main St, City, State 12345",
    "method": "Standard",
    "tracking": "TRACK123456"
  },
  "payment": {
    "method": "Credit Card",
    "last_4": "4242",
    "amount": 109.97
  },
  "order_history": [...]
}

Runtime Values Immediately Available:
- {{order_id}} can be used in shipping label
- {{customer_id}} can be used in customer lookup
Full Details in Final Output:
- Complete observation included in transformation
- Both runtime values and detailed data in output_data JSON

Benefits

This pattern gives you:

Immediate access to key identifiers via runtime variables
Non-blocking extraction of comprehensive data
Single extraction instead of multiple separate calls
Flexible output tailored to your needs

Best Practices

1. Design Your Schema First

Before writing workflow prompts, define your output schema:

{
  "type": "object",
  "properties": {
    "primary_id": {
      "type": "string"
    },
    "core_data": {
      "type": "object"
    },
    "details": {
      "type": "array"
    },
    "metadata": {
      "type": "object"
    }
  }
}

2. Use the Right Tool for Each Data Type

Data Type	Best Tool	Example
IDs, Numbers (copyable)	copy_to_clipboard	Account numbers, order IDs
Dynamic decisions	focused_action	Status checks, validations
Large extractions	extract_prompt with `process_async="run"`	Analytics, comprehensive data
List processing	extract_prompt with `process_async="batch"`	Scrolling through tables

3. Set Runtime Variables for Key Identifiers

If a value is used multiple times or in file naming, make it a runtime variable:

"Extract customer_id and save as `{{customer_id}}` for use in:
- File naming: Report`{{customer_id}}`.pdf
- API calls: GET /api/customers/`{{customer_id}}`
- Output data: customer_id field"

4. Use Focused Actions for Critical Observations

Use focused_action when:

The observation requires decision-making
You need to verify something visually
The data determines workflow branching
You want to ensure dynamic re-evaluation in cached runs

5. Use Run-Scoped Extraction for Output-Only Data

If data is only needed in final output (not for navigation), use run-scoped:

"Take screenshot with extract_prompt='Extract complete report data' 
and process_async='run'

This doesn't block the workflow - extraction happens in background."

6. Request JSON Format

Always request JSON for structured data:

extract_prompt='Extract order data as JSON: {order_id: string, items: array, 
total: number, status: string}'

7. Use Async Extraction for Runtime Variables

When using process_async (batch or run), extraction agents can call upsert_runtime_values:

"Take screenshot with extract_prompt='Extract order_id as runtime variable 
using upsert_runtime_values, then describe the order details' and process_async='batch'"

This stores {{order_id}} for later use while also providing comprehensive observations.

8. Include Type Information

Help the transformation agent by specifying types:

extract_prompt='Extract metrics as JSON: {revenue: number, customers: number, 
growth_rate: number (as percentage), categories: array of strings}'

Common Patterns

Pattern 1: ID + Details

Extract ID first (fast), then comprehensive details (async):

"Triple-click the Order ID and use copy_to_clipboard with key name 'order_id'

Take screenshot with extract_prompt='Extract complete order details as JSON' 
and process_async='run'

Continue workflow using `{{order_id}}` for file naming. Details in final output."

Pattern 2: Decision + Data

Make decision synchronously, extract data asynchronously:

"Take screenshot with extract_prompt='Extract account status: Active, Suspended, 
or Closed' to determine next steps.

If Active, take screenshot with extract_prompt='Extract complete account history 
and analytics as detailed JSON' and process_async='run'

Continue with appropriate workflow path."

Pattern 3: Iterative Extraction + Summary

Extract from multiple views, then summarize:

"For each page in report:
- Take screenshot with extract_prompt='Extract page data as JSON' and 
  process_async='batch'
- Go to next page

After all pages, take screenshot with extract_prompt='Extract summary statistics 
and totals' and process_async='run'

All data available in final output."

Pattern 4: Mixed Sources

Combine all extraction methods:

"Extract customer data using optimal method for each field:

Copyable fields (fast):
- copy_to_clipboard for customer_id
- copy_to_clipboard for account_number

Dynamic observation (for decisions):
- focused_action to check account status and extract current balance

Comprehensive data (for output):
- extract_prompt with process_async='run' for complete transaction history

All values combined in final structured output."

Output Data Access

Via API

After workflow completes, access output_data:

from cyberdesk import CyberdeskClient, RunCreate
import time

client = CyberdeskClient(api_key="your_api_key")

# Create and wait for run
response = client.runs.create_sync(
    RunCreate(
        workflow_id="workflow_123",
        input_values={"patient_name": "John Doe"},
    )
)
run = response.data

while run.status in ["scheduling", "running"]:
    time.sleep(2)
    run = client.runs.get_sync(run.id).data

# Access structured output if it was generated
if run.output_data is not None:
    output_data = run.output_data
    print(f"Patient MRN: {output_data['patient_mrn']}")
    print(f"Vitals: {output_data['vital_signs']}")
    print(f"Medications: {output_data['current_medications']}")

Via Webhooks

Receive output_data when run completes:

from fastapi import FastAPI, Request
from cyberdesk.webhooks import verify_webhook, RunCompletedEvent

app = FastAPI()

@app.post("/webhooks/cyberdesk")
async def handle_webhook(request: Request):
    # Verify webhook signature
    body = await request.body()
    signature = request.headers.get("x-cyberdesk-signature")
    
    verify_webhook(body, signature, webhook_secret="your_secret")
    
    # Parse event
    data = await request.json()
    event = RunCompletedEvent.from_dict(data)
    
    output_data = event.run.output_data
    if output_data is not None:
        # Process output_data
        print(f"Received output: {output_data}")
    
    return {"ok": True}

Troubleshooting

Output Data is None

Causes:

No output schema defined
Transformation failed

Solutions:

Define an output schema in workflow settings
Use schema defaults when you want terminal runs with limited captured data to still produce structured output
Check run logs for transformation errors

Missing Fields in Output

Causes:

Field not captured during workflow
Field name mismatch between schema and observations
Transformation couldn’t map observation to field

Solutions:

Verify observations contain the expected data
Use clear, descriptive field names in schema
Request JSON with explicit field names in extractions

Incorrect Data Types

Causes:

Schema specifies number but observation has string
Vision model returned unexpected format

Solutions:

Specify types in extraction prompts: “Extract age as number”
Use runtime values for precise extractions
Request “as JSON with types: {field: number}“

Runtime Variables Not in Output

Cause:

Transformation agent didn’t include them

Solution:

Runtime values are automatically included - check schema field names match variable names
If mismatch, transformation agent will try to map semantically

Summary

Output data generation in Cyberdesk is a powerful pipeline that combines:

Observations - Dynamic data from focused_action and extract_prompt
Runtime Values - Immediate identifiers and metrics
Transformation Agent - Intelligent mapping to your schema

All observations (whether from focused_action or extract_prompt) are stored together and transformed into structured output. By understanding this pipeline and using the right tools for each data type, you can build workflows that produce consistent, structured output data ready for integration with any downstream system. Note: The observations list contains data from both focused_action and extract_prompt tools. They’re all stored together and transformed into the final output. For detailed information, see:

Trajectories 101 - How caching and replay work
Focused Action - Dynamic observations
Copy to Clipboard - Fast clipboard extraction
Extract Prompt - Vision-based extraction
Async Extraction Patterns - Performance optimization

​Overview

​The Output Data Pipeline

​Components of Output Data

​1. Output Schemas

​2. Observations

​3. Runtime Values

​4. The Transformation Agent

​Output Data Optimization Features

​1. Direct Runtime Values Output

​2. Automatic Runtime Value Referencing

​Complete Flow Example

​Healthcare Workflow

​Run-Scoped Extraction with Runtime Variables

​The Extraction Agent Loop

​Example: Store Key Fields + Comprehensive Extraction

​Benefits

​Best Practices

​1. Design Your Schema First

​2. Use the Right Tool for Each Data Type

​3. Set Runtime Variables for Key Identifiers

​4. Use Focused Actions for Critical Observations

​5. Use Run-Scoped Extraction for Output-Only Data

​6. Request JSON Format

​7. Use Async Extraction for Runtime Variables

​8. Include Type Information

​Common Patterns

​Pattern 1: ID + Details

​Pattern 2: Decision + Data

​Pattern 3: Iterative Extraction + Summary

​Pattern 4: Mixed Sources

​Output Data Access

​Via API

​Via Webhooks

​Troubleshooting

​Output Data is None

​Missing Fields in Output

​Incorrect Data Types

​Runtime Variables Not in Output

​Summary

Overview

The Output Data Pipeline

Components of Output Data

1. Output Schemas

2. Observations

3. Runtime Values

4. The Transformation Agent

Output Data Optimization Features

1. Direct Runtime Values Output

2. Automatic Runtime Value Referencing

Complete Flow Example

Healthcare Workflow

Run-Scoped Extraction with Runtime Variables

The Extraction Agent Loop

Example: Store Key Fields + Comprehensive Extraction

Benefits

Best Practices

1. Design Your Schema First

2. Use the Right Tool for Each Data Type

3. Set Runtime Variables for Key Identifiers

4. Use Focused Actions for Critical Observations

5. Use Run-Scoped Extraction for Output-Only Data

6. Request JSON Format

7. Use Async Extraction for Runtime Variables

8. Include Type Information

Common Patterns

Pattern 1: ID + Details

Pattern 2: Decision + Data

Pattern 3: Iterative Extraction + Summary

Pattern 4: Mixed Sources

Output Data Access

Via API

Via Webhooks

Troubleshooting

Output Data is None

Missing Fields in Output

Incorrect Data Types

Runtime Variables Not in Output

Summary