Skip to main content

​
What Are Trajectories?

Trajectories are recorded sequences of actions that Cyberdesk captures when your workflow runs successfully. Think of them as “learned paths” through your software—once recorded, they can be replayed deterministically, making subsequent workflow executions much faster and more reliable.
A trajectory is essentially a cache of a successful workflow execution, including every action taken, every screen state observed, and every decision made by the AI agent.

​
How Trajectories Work

​
1. Initial Run: Recording

When you run a workflow for the first time (or when no matching trajectory exists): 
User triggers workflow → AI agent executes → Actions recorded → Trajectory saved
What Gets Recorded:
  • Actions taken (click, type, etc.)
  • Screen state before each action (pre-check snapshot)
  • Agent’s thought process for each step
  • Final results and observations
  • Display dimensions (screen resolution)
  • Input values used during the original run
Example Trajectory Step: 
{
  "func_name": "click",
  "args": [850, 450],
  "kwargs": {},
  "pre_check_snapshot": {
    "screenshot": "base64_encoded_image",
    "last_agent_thought": "I need to click the Submit button to proceed",
    "custom_cache_detection_instructions": "The Submit button must be visible and enabled"
  },
  "result": "Clicked at coordinates (850, 450)"
}

​
2. Subsequent Runs: Replay with Validation

When you run the same workflow again, Cyberdesk attempts to replay the trajectory: 
Load approved trajectories → Validate screen state → Replay if match → Complete in seconds
Validation Process:
  1. Load: System loads all approved trajectories for this workflow and resolution
  2. Filter: Filters trajectories that match the current step sequence
  3. Compare: For each step, compares current screen against cached screenshot
  4. Decision: AI determines if screens match well enough to proceed
  5. Replay: If match, executes the cached action
  6. Fallback: If no match, falls back to agent for that step
Performance Impact:
  • Cache Hit: Workflow completes in seconds (no AI reasoning needed)
  • Cache Miss: Falls back to AI agent for that step, records new trajectory from that point

​
The Approval Process

Critical: Newly created trajectories are NOT automatically approved. They must be manually reviewed and approved before they can be used in future workflow executions.

​
Why Approval is Required

Trajectories capture exact coordinates, keyboard inputs, and action sequences. Before allowing automatic replay:
  • Verify Correctness: Ensure the recorded actions actually worked
  • Check for Errors: Confirm no mistakes were captured in the trajectory
  • Review Observations: Validate that focused actions extracted correct data
  • Inspect Coordinates: Ensure clicks hit the right UI elements
Safety First: Unapproved trajectories sit in your trajectory library but are never used for replay, preventing potentially incorrect actions from running automatically.

​
How to Approve Trajectories

In the Dashboard:
  1. Navigate to Workflows → Select your workflow
  2. Click the Trajectories tab
  3. Review the trajectory:
    • Check each step’s screenshot and action
    • Verify focused action observations
    • Inspect coordinates and inputs
  4. Click the Approval toggle or checkbox
  5. Trajectory is now available for future runs
Best Practice: Review and approve trajectories promptly after successful runs to maximize caching benefits.

​
Editing Trajectories

You can edit trajectories in the dashboard to fix coordinates, update observations, or refine cache detection.

​
What You Can Edit

Step Actions:
  • Coordinates for clicks and drags
  • Text to type
  • Scroll amounts
  • Screenshot zoom areas
Pre-Check Snapshots:
  • Custom cache detection instructions
  • Expected screen states
Focused Action Observations:
  • Extracted data and observations
  • Runtime variable assignments

​
How to Edit

  1. Open a trajectory in the trajectory viewer
  2. Expand the step you want to edit
  3. Click the edit icon on specific fields
  4. Make your changes
  5. Save the trajectory
Important: Changes take effect immediately on approved trajectories. If you’re making significant changes, consider unapproving the trajectory first, testing it, then re-approving.

​
Editing Focused Actions: Critical Workflow Update

⚠️ HIGHLY RECOMMENDED: When you edit a focused_action observation or instruction in a trajectory, you should also update the underlying workflow prompt to match.

​
Why This Matters

When a cache miss occurs, the system falls back to running the AI agent using your workflow’s main prompt. If the trajectory has diverged from the prompt, you’ll get inconsistent behavior: Scenario:
  • Trajectory: focused_action extracts “patient_mrn, date_of_birth, primary_diagnosis”
  • Workflow Prompt: Says to extract “patient_mrn and date_of_birth” only
  • Cache Hit: Works perfectly (uses trajectory)
  • Cache Miss: Falls back to prompt, extracts different fields, causes output schema mismatch

​
The Right Approach

When editing focused actions in trajectories:
  1. Edit the Trajectory:
    • Update the focused action instruction
    • Modify the expected observation
    • Adjust runtime variables if needed
  2. Edit the Workflow Prompt:
    • Update the same focused_action instruction in the main prompt
    • Ensure consistency between trajectory and prompt
    • Test with cache disabled to verify prompt works
  3. Test Both Paths:
    • Run with cache hit (uses trajectory)
    • Run with cache disabled (uses prompt)
    • Verify both produce same results
Example: If you change trajectory from: 
focused_action: "Extract patient MRN only"
To: 
focused_action: "Extract patient MRN and date of birth"
Also update workflow prompt from: 
"Use focused_action to extract the patient MRN"
To: 
"Use focused_action to extract the patient MRN and date of birth"
Pro Tip: When making significant trajectory edits, temporarily disable cache detection to test the workflow with the updated prompt before re-enabling caching.

​
Cache Detection: Enabling and Disabling

Cache detection is always active by default, but you have control over it per workflow run.

​
When Cache Detection is Active (Default)

Every workflow run attempts to use approved trajectories: 
Run Start → Load Approved Trajectories → Validate Each Step → Replay if Match → Agent if Miss
Benefits:
  • ⚡ Massive speed improvements (cache hits complete instantly)
  • 🎯 Consistent execution (same actions every time)

​
Disabling Cache Detection

You can disable cache detection for runs by unapproving all approved trajectories in the dashboard. Use Cases for Disabling:
  • Testing workflow prompt changes
  • Debugging trajectory mismatches
  • Verifying prompt works without cache
  • Development and testing
Re-Enabling:
  • Cache detection re-enables automatically for subsequent runs
  • Per-run control means you can selectively test without cache

​
Custom Cache Detection Instructions

To improve cache hit rates and reduce false negatives/positives, you can add Custom Cache Detection Instructions to trajectory steps.

​
What They Do

Custom instructions guide the cache validation AI by providing human context: 
"The Submit button must be visible and enabled. 
Ignore the order number displayed, as it will differ between runs."

​
Where to Add Them

  1. Open a trajectory in the trajectory viewer
  2. Expand a step to view details
  3. Find “Custom Cache Detection Instructions” in the Pre-check Snapshot
  4. Add your guidance
  5. Save changes

​
When to Use Them

Add custom instructions when:
  • ✅ A step frequently has false cache misses due to minor differences
  • ✅ Dynamic content (timestamps, IDs) causes unnecessary validation failures
  • ✅ Specific UI elements are critical while others are cosmetic
  • ✅ You need to specify tolerances for acceptable variations
  • ✅ Recovery agent needs context about validation criteria
Example: 
"The patient list must show at least 5 entries. Patient names will vary 
based on {search_term}. Focus on the list structure, not specific names."

​
Impact on Cache Hit Rates

Well-written custom instructions can:
  • Increase cache hit rates by 20-40%
  • Reduce false negatives (rejecting valid matches)
  • Improve recovery agent performance
  • Provide valuable context for debugging
See the full Custom Cache Detection Instructions guide for detailed examples and best practices.

​
Trajectory Lifecycle

​
Phase 1: Creation

  • Workflow runs with AI agent
  • Actions recorded in real-time
  • Trajectory saved to database
  • Status: Unapproved, not used yet

​
Phase 2: Review

  • View trajectory in dashboard
  • Inspect each step and screenshot
  • Verify focused action observations
  • Check coordinates and inputs

​
Phase 3: Approval

  • Approve if trajectory is correct
  • Status: Approved, ready for use

​
Phase 4: Usage

  • Future runs load this trajectory
  • Each step validated before replay
  • Cache hits = fast execution
  • Cache misses = fallback to agent

​
Phase 5: Maintenance

  • Edit as needed (coordinates, instructions)
  • Update workflow prompt if editing focused actions
  • Test with cache disabled after significant changes
  • Unapprove if major edits needed, re-approve after testing

​
Multiple Trajectories Per Workflow

A single workflow can have multiple approved trajectories: Why Multiple Trajectories?
  • Different Input Patterns: Trajectory for “new patient” vs “existing patient”
  • UI Variations: Trajectory for each possible screen layout
  • Resolution Differences: Trajectory for 1920x1080 vs 1280x720
  • Branching Paths: Different trajectories for different workflow branches
How Selection Works:
  1. System loads all approved trajectories for this workflow
  2. Filters by display resolution (must match exactly)
  3. For each trajectory, validates screen state at first step
  4. Selects the trajectory with the best match
  5. Replays selected trajectory step by step
  6. Falls back to agent if no trajectory matches
Parallel Filtering:
  • Multiple trajectories validated simultaneously
  • Fast selection even with dozens of trajectories
  • Most recent trajectories prioritized

​
Best Practices

​
1. Approve Promptly

Approve successful trajectories quickly to start benefiting from cache hits on subsequent runs.

​
2. Keep Prompts in Sync

When editing trajectories (especially focused actions), update the workflow prompt to match: 
Trajectory Edit: focused_action observation changed
         ↓
Workflow Prompt: Update focused_action instruction
         ↓
Test: Run with cache disabled
         ↓
Verify: Both paths produce same results
         ↓
Re-approve: Trajectory ready for use

​
3. Use Custom Instructions Strategically

Add custom instructions to steps that:
  • Frequently fail validation unnecessarily
  • Have dynamic content that’s acceptable to ignore
  • Require nuanced validation logic

​
4. Test Before Approving

Run the workflow at least once successfully before approving the trajectory:
  • Verify all actions completed correctly
  • Check output data matches expectations
  • Ensure focused actions extracted correct values
  • Inspect coordinates hit the right UI elements

​
5. Maintain Trajectory Hygiene

  • Unapprove trajectories that are outdated or incorrect
  • Delete trajectories that are no longer relevant
  • Edit trajectories when UI changes are minor (coordinate adjustments)
  • Re-record when UI changes are major (new flow needed)

​
6. Monitor Cache Hit Rates

Track your workflow’s cache hit rate in the dashboard:
  • High hit rate (>80%) = well-optimized trajectories
  • Low hit rate (<50%) = may need custom instructions or prompt updates
  • Check which steps frequently miss cache

​
7. Consider Resolution

Trajectories are resolution-specific:
  • Record trajectories at your most common resolution
  • If you use multiple resolutions, you’ll need multiple trajectories
  • Display dimensions must match exactly for replay

​
Common Scenarios

​
Scenario 1: UI Element Moved

Problem: Button moved from (850, 450) to (850, 480) Solution:
  1. Open trajectory in viewer
  2. Find the click step
  3. Edit coordinates from (850, 450) to (850, 480)
  4. Save trajectory
  5. Test with cache enabled
Quick Fix: Minor coordinate adjustments don’t require prompt changes.

​
Scenario 2: Focused Action Needs More Data

Problem: Trajectory extracts “customer_id” but you now need “customer_id and email” Solution:
  1. Edit trajectory focused_action observation to include email
  2. IMPORTANT: Update workflow prompt to match: 
    Before: "Use focused_action to extract customer_id"
After: "Use focused_action to extract customer_id and email"
  3. Test with cache disabled to verify prompt works
  4. Test with cache enabled to verify trajectory works
  5. Re-approve if unapproved
Critical: Prompt must match trajectory for consistent cache miss behavior.

​
Scenario 3: Dynamic Content Causes False Misses

Problem: Cache detection fails because timestamps/IDs differ Solution:
  1. Open trajectory step
  2. Add custom cache detection instruction: 
    "The form structure must match with 3 input fields. 
Ignore the order ID shown - it will differ per run."
  3. Save trajectory
  4. Future runs will ignore order ID differences
See Custom Cache Detection Instructions for more examples.

​
Scenario 4: Workflow Prompt Changed Significantly

Problem: You updated the workflow prompt with new steps or different logic Solution:
  1. Unapprove all existing trajectories (they’re now outdated)
  2. Run workflow with cache disabled (force fresh recording)
  3. Review new trajectory
  4. Approve new trajectory
  5. Old trajectories can be deleted or kept for reference
Don’t Edit: When prompts change significantly, recording a fresh trajectory is better than trying to edit the old one.

​
Scenario 5: Want to Test Without Cache

Problem: Need to verify prompt works independently of cache Solution:
  1. Trigger run with cache detection disabled
  2. Agent executes from scratch using workflow prompt
  3. New trajectory is recorded (if successful)
  4. Review and approve new trajectory if it’s better
Use Case: Regression testing, prompt validation, debugging

​
Trajectory Data Structure

​
High-Level Structure

{
  "id": "uuid",
  "workflow_id": "uuid",
  "is_approved": false,
  "dimensions": {"width": 1920, "height": 1080},
  "original_input_values": {"patient_name": "John Doe"},
  "trajectory_data": [
    {
      "func_name": "click",
      "args": [x, y],
      "kwargs": {},
      "pre_check_snapshot": { ... },
      "result": "..."
    },
    {
      "func_name": "type",
      "args": ["search text"],
      "kwargs": {},
      "pre_check_snapshot": { ... },
      "result": "..."
    },
    ...
  ]
}

​
Pre-Check Snapshot

Captured before each action: 
{
  "screenshot": "base64_image",
  "last_agent_thought": "I can see the login form...",
  "custom_cache_detection_instructions": "Login button must be visible",
  "coordinates": [[x1, y1], [x2, y2], ...],
  "step_signature_hash": "unique_hash"
}

​
Focused Action Steps

Special handling for dynamic observations: 
{
  "func_name": "focused_action",
  "args": ["Extract the patient MRN"],
  "kwargs": {
    "is_cached_action": true,
    "cached_thought": "I need to find the MRN field"
  },
  "pre_check_snapshot": { ... },
  "result": "Patient MRN: MRN12345"
}
When editing focused actions, remember to update the workflow prompt to match!

​
Performance Benefits

​
Without Trajectories (Every Run Uses AI)

Run 1: AI agent from scratch (45 seconds)
Run 2: AI agent from scratch (45 seconds)
Run 3: AI agent from scratch (45 seconds)
Average: 45 seconds per run

​
With Trajectories (Cache Hits)

Run 1: AI agent + record trajectory (45 seconds) → Trajectory approved
Run 2: Replay trajectory (5 seconds) ✅ Cache hit
Run 3: Replay trajectory (5 seconds) ✅ Cache hit
Average: 18 seconds per run, improving to ~5 seconds as cache hits increase
Typical Improvement: 5-10x faster execution after trajectory approval

​
Partial Cache Hits

Even partial cache hits provide benefits: 
Steps 1-5: Replay from trajectory (2 seconds)
Step 6: Cache miss, use AI agent (8 seconds)
Steps 7-10: Record new actions (continue with agent)

Total: 10 seconds (vs 45 seconds from scratch)

​
Trajectory Strategies

​
Strategy 1: Single Golden Trajectory

Approach: One approved trajectory per workflow Best For:
  • Highly deterministic workflows
  • Consistent UI layouts
  • Same input pattern every time
  • Minimal variation between runs
Pros: Simple, easy to maintain Cons: Any variation causes cache miss

​
Strategy 2: Multi-Path Trajectories

Approach: Multiple approved trajectories for different scenarios Best For:
  • Workflows with branching logic
  • Different input patterns (new vs existing records)
  • UI variations (different screen layouts)
  • Different resolution targets
Pros: Higher cache hit rates, handles variation Cons: More trajectories to maintain

​
Strategy 3: Progressive Refinement

Approach: Start with one trajectory, add more as variations are discovered Best For:
  • New workflows where patterns emerge over time
  • Workflows with occasional edge cases
  • Gradual optimization
Process:
  1. Approve first successful trajectory
  2. Monitor cache miss patterns
  3. Identify common variations
  4. Record and approve trajectories for those variations
  5. Cache hit rate improves over time

​
Troubleshooting

​
Low Cache Hit Rate

Symptoms: Most runs fall back to agent instead of using cache Possible Causes:
  • No approved trajectories
  • Custom instructions too strict
  • UI changed since trajectory was recorded
  • Resolution mismatch
  • Input patterns vary significantly
Solutions:
  • Approve successful trajectories
  • Add custom instructions to tolerate minor differences
  • Re-record trajectories after UI changes
  • Ensure consistent display resolution
  • Consider multiple trajectories for different input patterns

​
Trajectory Approval Uncertainty

Question: “Should I approve this trajectory?” Checklist:
  • ✅ Workflow completed successfully
  • ✅ Output data is correct
  • ✅ Focused actions extracted expected values
  • ✅ All clicks hit the right UI elements
  • ✅ No errors in the run logs
  • ✅ Screenshots show correct screens at each step
If all checks pass → Approve
If any concern → Review more carefully or run again to verify

​
Trajectory Editing vs Re-Recording

Edit Trajectory When:
  • Minor coordinate adjustments (button moved slightly)
  • Updating custom cache detection instructions
  • Refining focused action observations (if prompt also updated)
  • Small corrections that don’t change workflow logic
Re-Record Trajectory When:
  • Major UI redesign
  • Workflow prompt changed significantly
  • Different action sequence needed
  • New steps added or removed
  • Complete workflow refactor
Rule of Thumb: If you’re changing more than 3-4 steps or the workflow logic has changed, re-record instead of editing.

​
Advanced Topics

​
Parameterized Trajectories

Trajectories can be parameterized to handle variations: Original Input Values: 
{
  "patient_name": "John Doe",
  "date": "2024-01-15"
}
Replay with Different Values: 
{
  "patient_name": "Jane Smith",
  "date": "2024-01-16"
}
The trajectory replays the same action sequence but with new values substituted.

​
Trajectory Versioning

Trajectories are immutable once approved:
  • Edits create new versions (updated_at timestamp changes)
  • Old trajectory data is overwritten (not versioned separately)
  • Consider unapproving before major edits

​
Resolution Filtering

Trajectories are automatically filtered by display resolution:
  • Exact Match Required: 1920x1080 trajectory won’t work on 1280x720
  • Why: Coordinates are absolute pixel positions
  • Solution: Record trajectories at each resolution you use

​
Summary

Trajectories are Cyberdesk’s intelligent caching system that dramatically speeds up workflow execution:
  • Recording: Automatic during initial runs
  • Approval: Required before trajectories can be used
  • Validation: AI compares screen states before each replay
  • Editing: Supported, but keep prompts in sync (especially for focused actions)
  • Custom Instructions: Enhance cache detection accuracy
  • Cache Control: Enable/disable per run as needed
Key Takeaway: Approve good trajectories, keep prompts synchronized when editing, and use custom instructions to optimize cache hit rates. For advanced cache validation, see Custom Cache Detection Instructions.