Desktop Agent
Control desktop apps via mouse and keyboard, capture screenshots, teach AI tasks by demonstration, and automate workflows with saved reusable tasks.
Desktop AI Agent Skill
Use this skill when the user wants to:
- Control desktop applications
- Teach the AI to perform tasks
- Have the AI learn from demonstrations
- Automate desktop workflows
Commands
Take Screenshot
from desktop_agent import get_agent
agent = get_agent()
agent.capture_to_file("screenshot.png")
Get Mouse Position
agent = get_agent()
pos = agent.get_mouse_position() # Returns (x, y)
Mouse Control
agent = get_agent()
agent.move_to(x, y) # Move mouse
agent.click(x, y) # Click
agent.double_click(x, y) # Double click
agent.right_click(x, y) # Right click
agent.drag_to(x, y) # Drag
Keyboard Control
agent = get_agent()
agent.type("Hello world") # Type text
agent.press("enter") # Press key
agent.hotkey("ctrl", "c") # Ctrl+C
Teaching Mode
from desktop_agent import get_agent
from desktop_agent.teacher import TaskTeacher
agent = get_agent()
teacher = TaskTeacher(agent)
# Start teaching
teacher.start_teaching("my_task")
# Record actions
teacher.record_click() # Records current mouse position
teacher.record_click(100, 200) # Records specific position
teacher.record_type("text")
teacher.record_press("enter")
teacher.record_hotkey("ctrl", "s")
teacher.record_wait(2) # Wait 2 seconds
# Show steps
teacher.show_steps()
# Save or cancel
teacher.finish_teaching()
teacher.cancel_teaching()
Running Learned Tasks
agent = get_agent()
tasks = agent.list_tasks() # List all tasks
agent.execute_task("task_name") # Run a task
Files
desktop_agent/__init__.py- Core agentdesktop_agent/teacher.py- Teaching systemlearned_tasks/- Saved task definitions
Notes
- AI can see screen and control mouse/keyboard
- User can teach by demonstration
- Tasks are saved as JSON and reusable
- Use with caution - can control any application
Related skills
- Downloads
- 1.8kAll time
- Security audit
- Review
- Last updated
- 6mo ago
- Current version
- v1.0.0
- License
- MIT-0
