Skip to main content

What is Extract Prompt?

Extract Prompt is a powerful vision-based extraction tool you add on top of a screenshot, that uses advanced AI vision models to read, interpret, and extract data from screenshots. Unlike clipboard-based extraction which requires selectable text, Extract Prompt can extract data from any visible content—including images, PDFs, charts, tables, and complex layouts.
In your prompts, use the screenshot action with the extract_prompt parameter to trigger this feature.
Trajectories & Extract Prompt: Extract prompts work seamlessly with Cyberdesk’s trajectory caching system. When a trajectory is replayed, extract prompts re-execute to capture fresh data from the current screen, ensuring dynamic data extraction even in cached workflows.

Why Extract Prompt Exists

Cyberdesk offers three complementary methods for data extraction: Clipboard Extraction (copy_to_clipboard)
  • Direct copy via Ctrl+C, deterministic and instant
  • Best for: Selectable text fields (IDs, numbers, dates)
  • Limitation: Only works with copyable text
Focused Action (focused_action)
  • Dynamic decision-making with vision-based extraction
  • Best for: Runtime decisions, conditional logic, setting runtime variables
  • Use when: You need to make decisions during workflow execution
Extract Prompt (This Tool)
  • Pure vision-based extraction with flexible async processing
  • Best for: Large-scale data extraction, non-copyable content, parallel processing
  • Use when: Extracting data for output, not needed for navigation decisions
Extract Prompt excels at extracting data from:
  • Non-selectable or non-copyable text
  • Images, PDFs, scanned documents
  • Charts, graphs, and visualizations
  • Complex tables and multi-column layouts
  • Dynamic content that changes between runs

How It Works

Basic Flow

  1. Agent takes a screenshot (optionally with zoom)
  2. Screenshot is sent to a strong vision model with your extraction instruction
  3. Vision model reads the screen and extracts the requested data
  4. Extracted text is returned as the tool result

With Async Processing

When using process_async, extractions can run in the background:
  • Batch scope: Extractions run in parallel within current tool call batch
  • Run scope: Extractions run for entire workflow lifetime, only awaited at the end

The process_async Parameter

The process_async parameter controls when and how extractions are processed:

Synchronous (Default)

  • When: process_async not set, false, or None
  • Behavior: Blocks until extraction completes
  • Use for: Single extractions where you need immediate results
  • Runtime values: If your prompt explicitly says to save or store something, the extraction agent can use upsert_runtime_values before returning its final text result

Batch-Scoped Async

  • When: process_async=true or process_async="batch"
  • Behavior: When the screenshot runs inside an actual batched tool phase, extraction runs in parallel with other batch extractions and all complete before the next agent step
  • Use for: Scrolling through lists, extracting from multiple views in sequence
  • Fallback: If Cyberdesk executes that screenshot as a standalone tool call instead of a batch, it falls back to synchronous extraction
  • Special: The extraction agent can still call upsert_runtime_values when your prompt explicitly asks it to save or store values (see below)

Run-Scoped Async

  • When: process_async="run"
  • Behavior: Extraction runs completely in background for entire run, only awaited at final output
  • Use for: Large extractions not needed for navigation, maximum parallelism
  • Requirement: The workflow must have an output_schema; otherwise Cyberdesk returns an error and asks you to use synchronous or batch mode instead
  • Special: The extraction agent can still call upsert_runtime_values when your prompt explicitly asks it to save or store values (see below)

When to Use Each Mode

Use Synchronous When:

  • Extracting a single value you need immediately
  • The extraction result determines next steps
  • Speed is not critical (< 5 extractions total)
  • You want simple, predictable behavior
Example:

Use Batch-Scoped Async When:

  • Scrolling through lists or paginated content
  • Extracting from multiple sequential views
  • Extractions don’t depend on each other
  • Results should be ready for next agent decision
Example:

Use Run-Scoped Async When:

  • Extracting large amounts of data
  • Extraction is only needed for final output, not navigation
  • You want maximum performance (fully non-blocking)
  • You may want to store specific values as runtime variables mid-extraction
Example:

Runtime Values with Extract Prompt

All extract_prompt modes use the extraction agent, and that agent can call upsert_runtime_values when your prompt explicitly tells it to save or store runtime values. Async modes add concurrency and background execution, but runtime-value storage itself is not limited to async.

The upsert_runtime_values Tool

The extraction agent can:
  1. Store specific values as runtime variables via upsert_runtime_values
  2. Provide final observations as text
  3. Do BOTH: Store values AND provide observations
For a dedicated reference on upsert_runtime_values, including merge/append operators and focused-action usage, see Upsert Runtime Values.

System Prompt Behavior

The extraction agent receives guidance like this:

Use Cases for Async Extraction with Runtime Variables

Batch-Scoped: Store and Observe During List Processing:
Run-Scoped: Store Specific Fields, Observe the Rest:
Pure Observation (No Runtime Variables):
Multiple Runtime Variables with Observation:
Extract Prompt can both store runtime values and return observations in any mode. Choose sync when you need the answer immediately, batch when you want parallel work inside a batched tool phase, and run-scope when the result is only needed for final structured output.

Real-World Examples

Healthcare: Patient Data Extraction

Synchronous (Simple):
Batch-Scoped (List Processing):
Run-Scoped (Large Extraction):

E-Commerce: Product Catalog Extraction

Batch-Scoped with Runtime Variable:
Run-Scoped (Full Catalog):

Finance: Transaction Data

Synchronous (Decision-Making):
Run-Scoped with Multiple Runtime Variables:

Insurance: Claims Processing

Batch-Scoped (Multiple Claims):
Run-Scoped (Detailed Claim):

Formatting Guidelines

Always Request JSON for Structured Data

Best Practice: Always request strict JSON with explicit keys for structured data extraction.
Good:
Also Good:
Avoid:

Multi-Field Extraction Templates

Simple Object:
Array of Objects:
Nested Structure:

Integration with Output Schemas

When your workflow has an output schema defined, extracted data automatically flows into the final structured output.

How It Works

  1. Define Output Schema in workflow settings:
  1. Extract Data During Workflow:
  1. Automatic Transformation:
  • At run completion, extraction results (+ any runtime variables + focused action observations) are automatically transformed to match your output schema
  • No manual output construction needed!
Optimization Tip: If all your data is already in runtime values and you want zero LLM transformation, set your output schema to {"only_runtime_values": true} to return runtime values directly. The transformation agent also automatically references existing runtime values instead of regenerating them, reducing lossiness. Learn more about output optimization.

Example: Healthcare Record Extraction

Output Schema:
Workflow:

Comparison with Other Extraction Methods

When to Use Each

Use Extract Prompt When:
  • Extracting large amounts of data for output
  • Working with non-copyable content (images, PDFs, charts)
  • Need parallel/async processing for speed
  • Extraction doesn’t affect navigation decisions
  • Want to store runtime values while also returning an extraction result
Use Focused Action When:
  • Making dynamic decisions during workflow
  • Conditional logic based on screen content
  • Selecting from lists based on runtime criteria
  • Verification steps that determine next actions
Use Copy to Clipboard When:
  • Text is selectable and copyable
  • Need deterministic, byte-exact extraction
  • Speed is critical
  • Working with simple text fields (IDs, numbers, dates)

Advanced Patterns

Hybrid Extraction Strategy

Combine all three methods for optimal performance:

Cascading Extractions

Extract summary data first, then detailed data based on results:

Progressive Detail Extraction

Start with high-level extraction, add detail as needed:

Runtime Variable Integration

Use run-scoped extraction to set variables for later workflow steps:

Model Override (Optional)

You can optionally specify which model to use for extraction by adding the model parameter:
Unlike focused_action, any vision-capable model works for extract_prompt—it doesn’t need computer use capabilities since it only reads the screen without performing actions.
In the prompt editor, type model="" or use the / slash menu and select “Model Override” to access the model picker. See Model Configuration for details on per-action model overrides and using different models for cost optimization.

Error Handling and Best Practices

Clear Instructions

Be Specific: The clearer your extraction prompt, the better the results.
Good:
Avoid:

Field Type Specification

Always specify expected data types:

Handling Missing Data

Instruct the model how to handle missing fields:

Vision Model Limitations

Vision Model Considerations:
  • Complex tables may require multiple extractions
  • Very small text may not be readable (use zoom if needed)
  • Similar-looking characters (0/O, 1/l) may be confused
  • For critical data, consider verification steps

Zoom for Better Accuracy

Performance Optimization

Use Async When Possible

Performance Tip: If extraction results aren’t needed for navigation and your workflow already has an output_schema, use process_async="run" for maximum parallelism and speed.

Batch Similar Extractions

Instead of:
Do this:

Minimize Synchronous Extractions

Before (Slow):
After (Fast):

Common Pitfalls

Avoid These Mistakes:
  1. Using synchronous for large extractions that aren’t needed for navigation
  2. Not specifying JSON format for structured data
  3. Vague instructions that lead to unpredictable results
  4. Extracting copyable text instead of using copy_to_clipboard
  5. Not using async when extracting from multiple pages/views

❌ Incorrect Usage

✅ Correct Usage

Complete Workflow Example

Scenario: E-Commerce Order Processing

Output Schema:
Workflow Instructions:
Result:
  • Order ID and email extracted instantly via clipboard
  • Large extraction runs in background while workflow continues
  • Runtime variables available for inventory and notification steps
  • All data automatically formatted to output schema
  • Maximum performance with hybrid approach

Summary

Extract Prompt is your go-to tool for vision-based data extraction with flexible async processing:
  • Synchronous: Simple, immediate extractions for decision-making
  • Batch-Scoped Async: Parallel processing within tool batches (great for lists)
  • Run-Scoped Async: Fully non-blocking extraction for entire run lifetime
Combine with focused_action for dynamic decisions, copy_to_clipboard for fast clipboard extraction, and looping tools for efficient batch processing to create highly efficient workflows.
Remember: Choose the right tool for each task—clipboard for copyable text (fast), extract_prompt for large-scale/async extraction (flexible), focused_action for decisions (dynamic), and loops for repetitive patterns (efficient).