What is Extract Prompt?
Extract Prompt is a powerful vision-based extraction tool you add on top of a screenshot, that uses advanced AI vision models to read, interpret, and extract data from screenshots. Unlike clipboard-based extraction which requires selectable text, Extract Prompt can extract data from any visible content—including images, PDFs, charts, tables, and complex layouts.In your prompts, use the
screenshot action with the extract_prompt parameter to trigger this feature.Trajectories & Extract Prompt: Extract prompts work seamlessly with Cyberdesk’s trajectory caching system. When a trajectory is replayed, extract prompts re-execute to capture fresh data from the current screen, ensuring dynamic data extraction even in cached workflows.
Why Extract Prompt Exists
Cyberdesk offers three complementary methods for data extraction: Clipboard Extraction (copy_to_clipboard)
- Direct copy via Ctrl+C, deterministic and instant
- Best for: Selectable text fields (IDs, numbers, dates)
- Limitation: Only works with copyable text
focused_action)
- Dynamic decision-making with vision-based extraction
- Best for: Runtime decisions, conditional logic, setting runtime variables
- Use when: You need to make decisions during workflow execution
- Pure vision-based extraction with flexible async processing
- Best for: Large-scale data extraction, non-copyable content, parallel processing
- Use when: Extracting data for output, not needed for navigation decisions
- Non-selectable or non-copyable text
- Images, PDFs, scanned documents
- Charts, graphs, and visualizations
- Complex tables and multi-column layouts
- Dynamic content that changes between runs
How It Works
Basic Flow
- Agent takes a screenshot (optionally with zoom)
- Screenshot is sent to a strong vision model with your extraction instruction
- Vision model reads the screen and extracts the requested data
- Extracted text is returned as the tool result
With Async Processing
When usingprocess_async, extractions can run in the background:
- Batch scope: Extractions run in parallel within current tool call batch
- Run scope: Extractions run for entire workflow lifetime, only awaited at the end
The process_async Parameter
Theprocess_async parameter controls when and how extractions are processed:
Synchronous (Default)
- When:
process_asyncnot set,false, orNone - Behavior: Blocks until extraction completes
- Use for: Single extractions where you need immediate results
- Runtime values: If your prompt explicitly says to save or store something, the extraction agent can use
upsert_runtime_valuesbefore returning its final text result
Batch-Scoped Async
- When:
process_async=trueorprocess_async="batch" - Behavior: When the screenshot runs inside an actual batched tool phase, extraction runs in parallel with other batch extractions and all complete before the next agent step
- Use for: Scrolling through lists, extracting from multiple views in sequence
- Fallback: If Cyberdesk executes that screenshot as a standalone tool call instead of a batch, it falls back to synchronous extraction
- Special: The extraction agent can still call
upsert_runtime_valueswhen your prompt explicitly asks it to save or store values (see below)
Run-Scoped Async
- When:
process_async="run" - Behavior: Extraction runs completely in background for entire run, only awaited at final output
- Use for: Large extractions not needed for navigation, maximum parallelism
- Requirement: The workflow must have an
output_schema; otherwise Cyberdesk returns an error and asks you to use synchronous or batch mode instead - Special: The extraction agent can still call
upsert_runtime_valueswhen your prompt explicitly asks it to save or store values (see below)
When to Use Each Mode
Use Synchronous When:
- Extracting a single value you need immediately
- The extraction result determines next steps
- Speed is not critical (< 5 extractions total)
- You want simple, predictable behavior
Use Batch-Scoped Async When:
- Scrolling through lists or paginated content
- Extracting from multiple sequential views
- Extractions don’t depend on each other
- Results should be ready for next agent decision
Use Run-Scoped Async When:
- Extracting large amounts of data
- Extraction is only needed for final output, not navigation
- You want maximum performance (fully non-blocking)
- You may want to store specific values as runtime variables mid-extraction
Runtime Values with Extract Prompt
Allextract_prompt modes use the extraction agent, and that agent can call upsert_runtime_values when your prompt explicitly tells it to save or store runtime values. Async modes add concurrency and background execution, but runtime-value storage itself is not limited to async.
The upsert_runtime_values Tool
The extraction agent can:- Store specific values as runtime variables via
upsert_runtime_values - Provide final observations as text
- Do BOTH: Store values AND provide observations
For a dedicated reference on
upsert_runtime_values, including merge/append operators and focused-action usage, see Upsert Runtime Values.System Prompt Behavior
The extraction agent receives guidance like this:Use Cases for Async Extraction with Runtime Variables
Batch-Scoped: Store and Observe During List Processing:Extract Prompt can both store runtime values and return observations in any mode. Choose sync when you need the answer immediately, batch when you want parallel work inside a batched tool phase, and run-scope when the result is only needed for final structured output.
Real-World Examples
Healthcare: Patient Data Extraction
Synchronous (Simple):E-Commerce: Product Catalog Extraction
Batch-Scoped with Runtime Variable:Finance: Transaction Data
Synchronous (Decision-Making):Insurance: Claims Processing
Batch-Scoped (Multiple Claims):Formatting Guidelines
Always Request JSON for Structured Data
Good:Multi-Field Extraction Templates
Simple Object:Integration with Output Schemas
When your workflow has an output schema defined, extracted data automatically flows into the final structured output.How It Works
- Define Output Schema in workflow settings:
- Extract Data During Workflow:
- Automatic Transformation:
- At run completion, extraction results (+ any runtime variables + focused action observations) are automatically transformed to match your output schema
- No manual output construction needed!
Example: Healthcare Record Extraction
Output Schema:Comparison with Other Extraction Methods
When to Use Each
Use Extract Prompt When:- Extracting large amounts of data for output
- Working with non-copyable content (images, PDFs, charts)
- Need parallel/async processing for speed
- Extraction doesn’t affect navigation decisions
- Want to store runtime values while also returning an extraction result
- Making dynamic decisions during workflow
- Conditional logic based on screen content
- Selecting from lists based on runtime criteria
- Verification steps that determine next actions
- Text is selectable and copyable
- Need deterministic, byte-exact extraction
- Speed is critical
- Working with simple text fields (IDs, numbers, dates)
Advanced Patterns
Hybrid Extraction Strategy
Combine all three methods for optimal performance:Cascading Extractions
Extract summary data first, then detailed data based on results:Progressive Detail Extraction
Start with high-level extraction, add detail as needed:Runtime Variable Integration
Use run-scoped extraction to set variables for later workflow steps:Model Override (Optional)
You can optionally specify which model to use for extraction by adding themodel parameter:
focused_action, any vision-capable model works for extract_prompt—it doesn’t need computer use capabilities since it only reads the screen without performing actions.
Error Handling and Best Practices
Clear Instructions
Good:Field Type Specification
Always specify expected data types:Handling Missing Data
Instruct the model how to handle missing fields:Vision Model Limitations
Zoom for Better Accuracy
Performance Optimization
Use Async When Possible
Batch Similar Extractions
Instead of:Minimize Synchronous Extractions
Before (Slow):Common Pitfalls
❌ Incorrect Usage
✅ Correct Usage
Complete Workflow Example
Scenario: E-Commerce Order Processing
Output Schema:- Order ID and email extracted instantly via clipboard
- Large extraction runs in background while workflow continues
- Runtime variables available for inventory and notification steps
- All data automatically formatted to output schema
- Maximum performance with hybrid approach
Summary
Extract Prompt is your go-to tool for vision-based data extraction with flexible async processing:- Synchronous: Simple, immediate extractions for decision-making
- Batch-Scoped Async: Parallel processing within tool batches (great for lists)
- Run-Scoped Async: Fully non-blocking extraction for entire run lifetime