Skip to main content

​
Writing Effective Prompts

When creating workflows, clarity and specificity are paramount. Your prompts should be:
  • Unambiguous: Leave no room for interpretation
  • Specific: Include exact details like file paths, button names, or expected values
  • Actionable: Use clear, directive language

​
Example of Good vs Poor Prompting

"Download the report"

​
Understanding Variables

Cyberdesk supports three types of variables in your workflows:

​
Input Variables: {variable}

  • When defined: Before the workflow starts
  • Purpose: Pass dynamic data like patient names, dates, or IDs
  • Example: {patient_name}, {report_date}

​
Runtime Variables: {{variable}}

  • When defined: During workflow execution (ONLY by focused actions)
  • Purpose: Store values discovered during the workflow for later use
  • Example: {{extracted_invoice_id}}, {{found_file_path}}
  • Important: Can only be set by focused_action, but once set can be used anywhere
Key Difference: Input variables are replaced before the agent sees them, runtime variables are set during execution and used by subsequent steps, and sensitive variables are passed verbatim (as aliases) to tools but their plaintext values are never exposed to the agent; they are resolved only during actual computer actions.

​
Sensitive Variables: {$variable}

  • When defined: At run creation (via sensitive_input_values)
  • Purpose: Handle secrets like passwords, SSNs, tokens without exposing plaintext to the agent or logs
  • Example: {$password}, {$ssn}
  • Behavior:
    • Kept as {$variable} in prompts and passed verbatim to tools
    • Stored securely in a third‑party vault (Basis Theory) only for the duration of the run
    • Never logged, never shown on the dashboard, and never sent to LLMs
    • Resolved to plaintext only at the last mile (e.g., when typing), then deleted from the vault after run completion

​
Utilizing Prompt Images

Images can anchor your instructions with visual context. Add screenshots alongside your text in the dashboard editor to help the agent recognize ambiguous icons, menu locations, or tricky UI flows (especially in legacy apps).
  • Place images near the paragraph they illustrate for best results
  • Use images as guidance – the agent will still take fresh screenshots during execution
  • Keep images focused on the relevant UI area; include multiple images if needed
Runtime access to prompt images is available via the grab_reference_image tool by filename (tail only). After analyzing any reference or zoomed image, the agent should take a fresh full screenshot before issuing coordinate actions. When specifying zoom regions, consider leaving a small margin so labels/edges aren’t cropped.
Use your OS’s native Markup/annotation tools to add red circles and arrows around the exact UI elements you want the AI to focus on. Simple visual callouts dramatically improve recognition.

​
Understanding Trajectories and Cached Tools

Cyberdesk’s intelligent caching system uses trajectories to record successful workflow executions and replay them deterministically. When a workflow has an approved trajectory, it can complete in seconds instead of minutes. To maximize this feature while still doing dynamic actions, you must properly prompt the agent to use special dynamic tools.
More on Trajectories: Learn more in Trajectories 101.

​
Dynamic Tools in Cached Workflows

Even during cached trajectory replay, certain tools execute dynamically to capture fresh data: By explicitly prompting the agent to use these tools, you ensure dynamic data capture even when the rest of the workflow is cached.

​
The Essential Workflow Tools

​
1. Focused Action

Capture dynamic content or make context-aware decisions during cached runs.

When to use

  • Selecting different items from a list each run
  • Extracting dynamic data that changes between runs
  • Making decisions based on screen content
In prompts, refer to it as "focused_action"

​
2. Mark File for Export

Export files from the remote machine after workflow completion.

When to use

  • Downloading generated reports
  • Extracting processed files
  • Saving workflow outputs
In prompts, refer to it as "mark_file_for_export"

​
3. Save Screenshot as Run Attachment

Capture important moments as screenshots accessible after the run.

When to use

  • Documenting visual confirmation
  • Capturing patient X-rays or medical images
  • Recording transaction confirmations
In prompts, refer to it as "save_screenshot_as_run_attachment"

​
4. Grab Reference Image

Retrieve a user-provided prompt image during execution by filename (tail only).

When to use

  • Provide visual context to disambiguate UI elements
  • Help the agent recognize specific screens, dialogs, or popups
In prompts, refer to it as "grab_reference_image". After analyzing, take a fresh full screenshot before coordinate actions. See the dedicated page for details.

​
5. Copy to Clipboard

Execute Ctrl+C and automatically save clipboard contents as a runtime variable.

When to use

  • Extracting text from legacy systems
  • Capturing non-selectable UI elements
  • Getting data from read-only fields
  • Quick text extraction with manual selection
In prompts, refer to it as "copy_to_clipboard"

​
6. Screenshot with Extract Prompt

Vision-based data extraction with flexible async processing modes (synchronous, batch-scoped, or run-scoped) and ability to set a runtime value

When to use

  • Non-copyable content (images, PDFs, charts, tables)
  • Parallel extraction across multiple views
  • When navigation doesn’t depend on extraction results
  • Quickly setting one off runtime values
In prompts, refer to it as "extract_prompt" parameter on screenshot action. For parallel processing, use process_async="batch" or process_async="run". See the dedicated page for full details on async modes.

​
Quick Start: Simple Extraction

"Take a screenshot with extract_prompt='Extract customer name, email, and 
phone number as JSON: {name: string, email: string, phone: string}'"

​
Parallel Extraction Across Multiple Screens

Make extraction 3-5x faster by using process_async="batch" when scrolling or navigating: 
"In one batched tool call, extract all product data by scrolling through the catalog:
- Take screenshot with extract_prompt='Extract all visible products as JSON 
  array' and process_async='batch'
- Scroll down
- Take screenshot with extract_prompt='Extract all visible products as JSON 
  array' and process_async='batch'
- Repeat until bottom

All extractions process in parallel before the next agent step."

​
Run-Scoped Extraction (Fully Non-Blocking)

For large extractions that don’t affect navigation, use process_async="run": 
"Take screenshot with extract_prompt='Extract complete dashboard analytics 
as detailed JSON, saving as a runtime value.' and process_async='run'.

Continue with other workflow tasks. The extraction runs in background and 
will be included in final output automatically."
Learn More: See the Extract Prompt page for detailed documentation on async modes, runtime variable integration, and advanced patterns.

​
7. Execute Terminal Command

Run PowerShell commands on the Windows machine.

When to use

  • System administration tasks
  • File operations
  • API calls or data processing
In prompts, refer to it as "execute_terminal_command"

​
8. Declare Task Failed

Terminate workflow execution when failure conditions are met.

When to use

  • Error handling
  • Validation failures
  • Preventing unnecessary continuation
In prompts, refer to it as "declare_task_failed"

​
Quick Examples

​
Using Focused Action

"When you reach the patient list, use focused_action to select the patient 
whose name matches {patient_name} and extract their ID number"

​
Exporting Files

"After generating the report, use mark_file_for_export to export the file 
located at C:\Reports\output.pdf"

​
Capturing Screenshots

"Once the payment confirmation appears, use save_screenshot_as_run_attachment 
to capture the confirmation screen as 'payment_confirmation.png'"

​
Copying to Clipboard

"Triple-click on the account number field to select it, then use 
copy_to_clipboard with key name 'account_number' to save it as 
{{account_number}} for use in the next form"

​
Running Terminal Commands

"Use execute_terminal_command to run 'Get-ChildItem C:\Exports\*.pdf | 
ConvertTo-Json' to list all PDF files in the exports folder"
For long-running commands: 
"Use execute_terminal_command with duration=120 (2 minutes) to run the report 
generation script and wait for completion before proceeding"

​
Handling Failures

"If the login screen shows 'Account Locked' or 'Invalid License', 
use declare_task_failed to terminate with message 'Unable to access system'"

​
Next Steps

For detailed information about each tool, including advanced usage patterns and real-world examples, explore the individual tool documentation pages:

Focused Action

Dynamic observations and decisions

Mark File for Export

File extraction from workflows

Save Screenshot

Visual documentation

Extract Prompt

Vision-based extraction with optional async extraction

Copy to Clipboard

Copy text and save as runtime variable

Grab Reference Image

Use prompt images at runtime

Execute Terminal

PowerShell automation

Declare Task Failed

Error handling