# Building Your Agent from Scratch Source: https://docs.hub.agentsea.ai/advanced_tutorial A detailed guide from zero to GUI surfer with multiple approaches In this guide, we'll show how to create your own agent using SurfKit and some of the techniques we came up along the way to help your agent navigate GUIs and accomplish its goals. ## Prerequisites * Install `poetry` (see [Poetry docs](https://python-poetry.org/docs/)). * Install `surfkit` (see [Quickstart](./quickstart)). * Set up your local or cloud environment (see [Configuration](./configuration)). * Install Tesseract on your machine (see [Tesseract docs](https://tesseract-ocr.github.io/tessdoc/Installation.html)). ## Creating an Agent Creating a dummy agent that follows the SurfKit protocol is super easy: ``` mkdir surfhamster && cd surfhamster surfkit new ``` The last command will ask you to answer a few questions: ``` Enter agent name: SurfHamster Describe the agent: The AI agent that can navigate GUIs and do tasks in them. Enter git user reference (Your Name ): Enter docker image repo: Enter icon url (https://tinyurl.com/y5u4u7te): ``` Feel free to leave the docker image repo empty for now and use a standard icon. When you run these commands, an Agent project will get initialized inside the folder you created. It contains all the components you need to give it a try: ``` surfkit create tracker -n hound001 -r docker ``` ``` surfkit create device -n eve001 ``` ``` surfkit create agent -n robby001 -r process ``` ``` surfkit solve "Find a cool cat image in the internet" \ --agent robby001 --device eve001 --tracker hound001 \ --starting-url "google.com" ``` If the browser tab with a VM desktop and an agent log opens and the agent starts solving the task, congratulations: you just created your first agent! ## How does it work? ### Code Let's briefly look inside the repo. There are a bunch of files there, but the most critical files are the following: * `agent.yaml` is a configuration file; it contains a few self-explanatory sections: * you will need to change the docker image repo later when you're ready to publish it; * you may also want to change the icon; * by default, your agent will run locally; * this agent is designed to work with a “desktop” device as are the other GUI-driven agents that we build; * `server.py` is just a utility class used to host the server with the agent process; * `agent.py` is the main class that implements the logic of the agent: * at the beginning of the task execution we explain to the MLLM/LLM what this agent is expected to do; * then we enter the loop: * given the task and the history of the chat, as well as the state of the desktop, we ask an MLLM/LLM to give us the next action and the reason for it; * the next action is returned by MLLM/LLM as a JSON, which is checked against the available device and executed upon it; * we exit the loop when either the MLLM/LLM returns an action marked as "result" (which means that it thinks that the task is solved), or the maximum amount of iterations is reached (30 by default). ### Architecture This type of agent works with a Desktop device. The Desktop device has an interface that allows it to control the VM (in the cloud or a local one) via a mouse and a keyboard programmatically. You've just created it above with `surfkit create device`. When the agent works to solve a given task, it uses the device. The device implements the `Tool` interface and therefore has a schema of actions that the agent can take at any give time. We get this schema in JSON format, ask an MLLM/LLM to return the next action in that format, and then on each step of solving a task, we have an action that can be passed back to the device to be executed. For example, the action returned by MLLM/LLM, might look like this: ```json theme={null} { "observation": "The mouse cursor is positioned inside the Google search bar.", "reason": "To find a cat image, we need to type the search query 'cool cat images' into the search bar.", "action": { "name": "type_text", "parameters": { "text": "cool cat images" } } } ``` When this object gets returned to a device, the device can execute it: in this case, type the text "cool cat images" using the keyboard. The actions include the low-level operations of a mouse and a keyboard, like moving the mouse, clicking on coordinates, typing letters, and sending key commands. They also include taking the screenshot and getting the current mouse coordinates, which helps an MLLM to choose the next action towards the task completion. The agent we just created works using these primitives: typing text and clicking the mouse. Cool, right? However, there is one little problem. ### Problem And this is a big problem: at the time of this tutorial (June 2024, "gpt-4o" was released not so long ago), all frontier MLLMs are horrible at identifying coordinates of the object on a screenshot. They can reason quite well about what should be clicked or typed to achieve their goal, but they can't return correct coordinates for that value. So it's time to make this agent better with some additional tricks. Our goal is to help them **convert an idea on what should be clicked into actual, correct screen coordinates**. To do that, we need to expand the toolset of our device and add a semantic layer to it, so that instead of "click on (400, 350)" our agent will return something like to "click on the big 'Search' button at the bottom of the screen". From the code point of view, we'll do the following: * We'll introduce the `SemanticDesktop` class, which is essentially a wrapper around the Desktop that we already have, and it inherits all the actions that it provides. * We'll then update the `Agent` class to use the `SemanticDesktop` alongside the actual `Desktop`: we want the actions to be taken from and by `SemanticDesktop` and translated to the low-level `Desktop` operations when needed; we also want to keep using the screenshotting and mouse-clicking abilities from `Desktop`; * After that, we'll introduce a new action to the `SemanticDesktop`: `click_object`. ## Adding SemanticDesktop {" "} You can find the code for this step of the tutorial [here](https://github.com/agentsea/surfhamster/tree/1fc5562be20c2731974222405160aef2f4168717).{" "} First, we need to refine our `SemanticDesktop`. It will interit from `Tool`, which would allow us pass it to the MLLM. See the full code for `tool.py` [here](https://github.com/agentsea/surfhamster/blob/1fc5562be20c2731974222405160aef2f4168717/surfhamster/tool.py). The most interesting part of this class is that we add a new method, `click_object`: ```python theme={null} @action def click_object(self, description: str, type: str) -> None: """Click on an object on the screen Args: description (str): The description of the object including its general location, for example "a round dark blue icon with the text 'Home' in the top-right of the image", please be a generic as possible type (str): Type of click, can be 'single' for a single click or 'double' for a double click. If you need to launch an application from the desktop choose 'double' """ info = self.desktop.info() screen_size = info["screen_size"] self._click_coords(screen_size["x"] // 2, screen_size["y"] // 2, "single") ``` As you can see, it is not so smart at the moment: it simply returns the coordinates in middle of the screen. Don't worry, we'll work on it later! As we now have this class, we can update the `Agent` class too. See the full code for `agent.py` [here](https://github.com/agentsea/surfhamster/blob/1fc5562be20c2731974222405160aef2f4168717/surfhamster/agent.py). Note that we replace some of the usages of the `Desktop` device by the `SemanticDesktop` device, but not all of them. The best way to explain it is that we get observations from the `Desktop` (a screenshot and mouse coordinates), but we run the actions of and by the `SemanticDesktop`. We also remove some actions we don't need our agent to know: ```python theme={null} tools = semdesk.json_schema( exclude_names=[ "move_mouse", "click", "drag_mouse", "mouse_coordinates", "take_screenshot", "open_url", "double_click", ] ) ``` If you run the agent now, you'll notice in the console logs, that the schema (the available actions) have changed, and the agent can now return a new kind of action: ```python theme={null} { "observation": "The cursor is near the address bar and the search tab at the top left of the Google homepage.", "reason": "To search for a cool cat image, the next step is to type the search query into the Google search box located in the center of the screen.", "action": { "name": "click_object", "parameters": { "description": "the Google search box in the center of the screen", "type": "single" } } } ``` The only problem now is that the implementation of this `click_object` function is still pretty dumb. So let's fix that now. ## Adding Grid {" "} You can find the code for this step of the tutorial [here](https://github.com/agentsea/surfhamster/tree/4a80d5f1fab16d61a0669ffca1507f6768c2b8dd). You'll need to add the fonts that you can see in the repository.{" "} There are many ways to assist an MLLM in picking the right location of the object on a screenshot. None of them are perfect (to the best of our knowledge at the moment of writing this tutorial), but combining a few in one agent can get you pretty high accuracy. Let's start with something simple. We call this approach "The Grid". The idea is to put a bunch of dots with numbers in the corners of the cells on the NxN grid on a screen. Honestly, it's easier to show than to explain: Google results page covered with grid If we desaturate the original screenshot and put this grid on top, we can ask an MLLM which dot is the closest one to the place the agent wants to click (for example, a search bar or a button). In order to do that, we'll start a tiny thread with an MLLM (outside of the main thread) just to address this question. We then simply convert the number that an MLLM returns back to the coordinates on a screen. First, we need to define a bunch of utility functions to generate this grid, merge it with the main image, and also convert images to and from b64 because it's the only image format gpt-4o accepts. See the code for `image.py` [here](https://github.com/agentsea/surfhamster/blob/4a80d5f1fab16d61a0669ffca1507f6768c2b8dd/surfhamster/image.py). Now, we can update the `click_object` function to give it some more power and perception: ```python theme={null} @action def click_object(self, description: str, type: str) -> None: """Click on an object on the screen Args: description (str): The description of the object including its general location, for example "a round dark blue icon with the text 'Home' in the top-right of the image", please be a generic as possible type (str): Type of click, can be 'single' for a single click or 'double' for a double click. If you need to launch an application from the desktop choose 'double' """ if type != "single" and type != "double": raise ValueError("type must be'single' or 'double'") color_number = os.getenv("COLOR_NUMBER", "yellow") color_circle = os.getenv("COLOR_CIRCLE", "red") click_hash = hashlib.md5(description.encode()).hexdigest()[:5] class ZoomSelection(BaseModel): """Zoom selection model""" number: int = Field( ..., description=f"Number of the dot closest to the place we want to click.", ) current_img_b64 = self.desktop.take_screenshot() current_img = b64_to_image(current_img_b64) img_width, img_height = current_img.size # number of "cells" along one side; the numbers are in the corners of those "cells" n = 10 thread = RoleThread() prompt = f""" You are an experienced AI trained to find the elements on the screen. You see a screenshot of the web application. I have drawn some big {color_number} numbers on {color_circle} circles on this image to help you to find required elements. Please tell me the closest big {color_number} number on a {color_circle} circle to the center of the {description}. Please note that some circles may lay on the {description}. If that's the case, return the number in any of these circles. Please return you response as raw JSON following the schema {ZoomSelection.model_json_schema()} Be concise and only return the raw json, for example if the circle you wanted to select had a number 3 in it you would return {{"number": 3}} """ self.task.post_message( role="assistant", msg=f"Clicking '{type}' on object '{description}'", thread="debug", images=[image_to_b64(current_img)], ) image_path = os.path.join(self.img_path, f"{click_hash}_current.png") current_img.save(image_path) img_width, img_height = current_img.size screenshot_b64 = image_to_b64(current_img) self.task.post_message( role="assistant", msg=f"Current image", thread="debug", images=[screenshot_b64], ) grid_path = os.path.join(self.img_path, f"{click_hash}_grid.png") create_grid_image( img_width, img_height, color_circle, color_number, n, grid_path ) merged_image_path = os.path.join( self.img_path, f"{click_hash}_merge.png" ) merged_image = superimpose_images(image_path, grid_path, 1) merged_image.save(merged_image_path) merged_image_b64 = image_to_b64(merged_image) self.task.post_message( role="assistant", msg=f"Merged image", thread="debug", images=[merged_image_b64], ) msg = RoleMessage( role="user", text=prompt, images=[merged_image_b64], ) thread.add_msg(msg) response = router.chat( thread, namespace="zoom", expect=ZoomSelection, agent_id="SurfHamster", retries=1 ) if not response.parsed: raise SystemError("No response parsed from zoom") self.task.add_prompt(response.prompt) zoom_resp = response.parsed self.task.post_message( role="assistant", msg=f"Selection {zoom_resp.model_dump_json()}", thread="debug", ) console.print(JSON(zoom_resp.model_dump_json())) chosen_number = zoom_resp.number # We convert the chosen number into screen coordinates # of the corresponding dot on the grid x_cell = (chosen_number - 1) // (n - 1) + 1 y_cell = (chosen_number - 1) % (n - 1) + 1 cell_width = img_width // n cell_height = img_height // n click_x = x_cell * cell_width click_y = y_cell * cell_height self.task.post_message( role="assistant", msg=f"Clicking coordinates {click_x}, {click_y}", thread="debug", ) self._click_coords(x=click_x, y=click_y, type=type) return ``` It looks like a lot is going on here, but if you look closely, we're just doing a few simple steps: * We generate the image with the grid, same as shown above. * We craft the prompt to instruct our MLLM to return to us exactly what we need: the number of the closest dot. * We run the prompt and get our result. * We convert the number back to screen coordinates. * Along the way, we record the stuff in the "debug" channel of our agent, so that you can see what exactly is going on, in the UI. You can find full code for `tool.py` [here](https://github.com/agentsea/surfhamster/blob/4a80d5f1fab16d61a0669ffca1507f6768c2b8dd/surfhamster/tool.py). When you run the agent now, you can see the images with the grid that it generates, in the debug tab. The MLLM picks the correct number pretty reliably. This method is obviously more intelligent than picking the middle of the screen. However, there is a good chance the bot misses the correct spot because the element we're interested in is right under the dot. To address this issue, we add a new capability, zooming in. We zoom in and scale up the part of the screenshot surrounding the chosen dot. You can see the implementation in the [SurfSlicer](https://github.com/agentsea/surfslicer) agent. ## Adding Tesseract {" "} You can find the code for this step of the tutorial [here](https://github.com/agentsea/surfhamster/tree/167e73de555986eec76bc25b57c2e433b37ee92f).{" "} As noted above, you can achieve the best results in your agent by combining many methods. One very simple but powerful idea is to use plain old fashioned OCR to find the text elements whenever it makes sense and click on them. But we use it with a twist. **Not only does OCR return the text, it returns the position of the text.** In case there is no text to click on (because the object is an icon, for example) or the OCR engine doesn't find any text we need (because it's a white text on a blue background, for example), we fall back to the Grid. But if we can find the text, it gives us two benefits: * Finding text with a bounding box using Tesseract is exceptionally fast in comparison to OpenAI API calls: You get the result in a fraction of a second. * The bounding box is very accurate: we can safely click in the middle of the coordinates and be sure that we hit the right oject. Google search page with a button highlighted First of all, install `pytesseract`: ``` poetry add pytesseract ``` Now, we need another bunch of utility methods, to run `Tesseract` and to find bounding boxes for a given text. See the code for `ocr.py` [here](https://github.com/agentsea/surfhamster/blob/167e73de555986eec76bc25b57c2e433b37ee92f/surfhamster/ocr.py). When we have this, we update `click_object`. We move the grid-related logic to a separate method, add a similar one with the OCR-related logic, and update the main action method like this: ```python theme={null} @action def click_object(self, description: str, type: str) -> None: """Click on an object on the screen Args: description (str): The description of the object including its general location, for example "a round dark blue icon with the text 'Home' in the top-right of the image", please be a generic as possible type (str): Type of click, can be 'single' for a single click or 'double' for a double click. If you need to launch an application from the desktop choose 'double' """ if type != "single" and type != "double": raise ValueError("type must be'single' or 'double'") coords = self._ocr_based_click(description, type) if coords is None: coords = self._grid_based_click(description, type) click_x = coords["x"] click_y = coords["y"] self.task.post_message( role="assistant", msg=f"Clicking coordinates {click_x}, {click_y}", thread="debug", ) self._click_coords(x=click_x, y=click_y, type=type) return ``` Grab the complete code for the final version of `tool.py`[here](https://github.com/agentsea/surfhamster/blob/167e73de555986eec76bc25b57c2e433b37ee92f/surfhamster/tool.py). If you look closely on the debug channel now, you'll see that our agent tries to use OCR whenever it makes sense, and if this operation succeeds, it goes on with the next iteration; if it doesn't succeed, it falls back to the grid approach. ## What's next? Now it's your turn! There are a lot of techniques that we've personally tried with different level of success; to name a few: * Locating elements on a page with Grounding Dino. * Cutting the image into pieces and compositing them on a new image with numbers alongside the various pieces. * Zooming into the Grid 2-3 times with new numbers. * Layering coordinates over a screenshot * Upscaling a screenshot with a GAN * OCR, as noted above, but with some tweaks. * Many more... We strongly believe that the key of the success of the agent is mixing and matching a bunch of techniques, including everything from classical ML to deep learning to the most bleeding edge features of frontier models, spiced up with traditional programming. So get in there and try your own techniques! Get creative. Get tricky. Think of it as outthinking the model to get what you want. We can't wait to see what you come up with! # API Reference Source: https://docs.hub.agentsea.ai/agentd/api # Installation Guide Source: https://docs.hub.agentsea.ai/agentd/installation In case you prefer not to use AgentDesk `AgentD` is currently tested on the Ubuntu 22.04 cloud image. ## Prerequisites We recommend using one of our base vms which are already configured. * [Qemu](https://www.qemu.org/download/) -- if you want to run the desktop locally. * [GCP CLI](https://cloud.google.com/sdk/docs/install) or [AWS CLI](https://aws.amazon.com/cli/) -- if you want it to run remotely. ## Qemu For Qemu, download the qcow2 image: ```bash theme={null} wget https://storage.googleapis.com/agentsea-vms/jammy/latest/agentd-jammy.qcow2 ``` To use the image, make a [cloud-init](https://cloud-init.io/) iso with our user-data. See this [tutorial](https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html), below is how it looks on MacOS: ```bash theme={null} xorriso -as mkisofs -o cidata.iso -V "cidata" -J -r -iso-level 3 meta/ ``` Then the image can be ran with Qemu: ```bash theme={null} qemu-system-x86_64 -nographic -hda ./agentd-jammy.qcow2 \ -m 4G -smp 2 -netdev user,id=vmnet,hostfwd=tcp::6080-:6080,hostfwd=tcp::8000-:8000,hostfwd=tcp::2222-:22 \ -device e1000,netdev=vmnet -cdrom cidata.iso ``` Once running, the `agentd` service can be accessed with: ```bash theme={null} curl localhost:8000/health ``` To login to the machine: ```bash theme={null} ssh -p 2222 agentsea@localhost ``` ## AWS For AWS, use public AMI `ami-01a893c1530453073`. Create a cloud-init script with your ssh key: ```yaml theme={null} #cloud-config users: - name: agentsea sudo: ['ALL=(ALL) NOPASSWD:ALL'] groups: sudo ssh_authorized_keys: - your-ssh-public-key package_upgrade: true ``` ```bash theme={null} aws ec2 run-instances \ --image-id ami-01a893c1530453073 \ --count 1 \ --instance-type t2.micro \ --key-name $KEY_NAME \ --security-group-ids $SG_NAME \ --subnet-id $SUBNET_NAME \ --user-data file://path/to/cloud-init-config.yaml ``` ## GCE For GCE, use the public image `ubuntu-22-04-20240208044623`. ```bash theme={null} gcloud compute instances create $NAME \ --machine-type "n1-standard-1" \ --image "ubuntu-22-04-20240208044623" \ --image-project $PROJECT_ID \ --zone $ZONE \ --metadata ssh-keys="agentsea:$(cat path/to/your/public/ssh/key.pub)" ``` ## Custom If you want to install on a fresh Ubuntu VM, use the a [cloud images base](https://cloud-images.ubuntu.com/jammy/current/) qcow2 image. ```bash theme={null} curl -sSL https://raw.githubusercontent.com/agentsea/agentd/main/remote_install.sh | sudo bash ``` # Introduction Source: https://docs.hub.agentsea.ai/agentd/intro `AgentD` is a powerful daemon designed to make a desktop OS accessible to AI agents. By exposing an HTTP API, `AgentD` allows for seamless interactions between a desktop environment and AI-driven applications or scripts. ## Features * **Mouse and Keyboard Control:** Simulate mouse movements, clicks, and keyboard inputs. * **Web Browser Control:** Open URLs and interact with web content through a Chromium-based browser. * **Screen Capture:** Take screenshots of your desktop for analysis or record-keeping. * **Session Recording:** Record and replay desktop sessions to capture workflows or for debugging purposes. ## Getting Started To get started with `AgentD`, follow these simple steps: 1. **Installation:** * For a quick start, we recommend using one of our pre-configured VMs which come with `AgentD` pre-installed. This is the easiest way to get up and running without worrying about dependencies or configuration. * Alternatively, if you prefer to install `AgentD` on your own Ubuntu VM, you can use our remote installation script. This is suitable for users who want more control over the installation process or need to integrate `AgentD` into an existing setup. * See details in [Installation](./installation) section. 2. **Usage:** * Once `AgentD` is installed and the VM is launched, you can start interacting with its desktop through the HTTP API. The API allows you to control the mouse and keyboard, manage web browser sessions, capture screenshots, and much more. * To check if `AgentD` is running correctly, you can send a request to the `/health` endpoint. A successful response indicates that AgentD is ready to accept commands. 3. **API Endpoints:** * `AgentD` provides a rich set of API endpoints to interact with the desktop. Here are some of the key functionalities: * Mouse and Keyboard Control: `/move_mouse`, `/click`, `/type_text`, etc. * Web Browser Control: `/open_url` * Screen Capture: `/screenshot` * Session Recording: `/recordings`, `/recordings/{session_id}/stop`, etc. For more detailed information on how to use `AgentD` and its API, please refer to the full [API documentation](https://agentsea.github.io/agentd/index.html) and examples provided in our [GitHub repository](https://github.com/agentsea/agentd). # GitHub Source: https://docs.hub.agentsea.ai/agentd/repo # API Reference Source: https://docs.hub.agentsea.ai/agentdesk/api # CLI Documentation Source: https://docs.hub.agentsea.ai/agentdesk/cli The AgentDesk CLI provides a command-line interface to manage desktop environments programmatically. Here are the primary switches and their usage: ## Commands ### `create` Creates a new desktop environment. * **Options**: * `--name`: The name of the desktop to create. Defaults to a generated name. * `--provider`: The provider type for the desktop. Options are 'ec2', 'gce', 'qemu' and 'docker'. Default is 'docker'. * `--image`: The image to use for the desktop. Defaults to Ubuntu Jammy. * `--memory`: The amount of memory (in GB) for the desktop. Default is 4. * `--cpu`: The number of CPU cores for the desktop. Default is 2. * `--disk`: The disk size for the desktop. Format as '\gb'. Default is '30gb'. * `--reserve-ip`: Whether to reserve an IP address for the desktop. Default is False. * `--ssh-key`: The SSH key for the desktop. Optional. ### `get` Retrieves information about one or all desktops. * **Options**: * `--name`: The name of the desktop to retrieve. If not provided, all desktops will be listed. * `--provider`: The provider type for the desktop. Optional. ### `delete` Deletes a specified desktop. * **Arguments**: * `name`: The name of the desktop to delete. ### `view` Opens a browser view of the specified desktop. * **Arguments**: * `name`: The name of the desktop to view. ### `refresh` Refreshes the provider information. * **Arguments**: * `provider`: The provider type for the desktop. ### `stop` Stops the specified desktop. * **Arguments**: * `name`: The name of the desktop to stop. ### `start` Starts the specified desktop. Doesn't apply for local desktops: use `create` to start a local desktop. * **Arguments**: * `name`: The name of the desktop to start. ### `clear_cache` Clears the cache directory. For more detailed information on each command, including options and examples, refer to the CLI help by running `agentdesk --help` or `agentdesk --help`. # Drawing Toy Source: https://docs.hub.agentsea.ai/agentdesk/demo A simple example of using AgentDesk # Introduction Source: https://docs.hub.agentsea.ai/agentdesk/intro AgentDesk provides full-featured Desktop environments which can be programatically controlled by AI agents. ## Features * Built on [AgentD](https://github.com/agentsea/agentd) – a runtime daemon which exposes a REST API for interacting with the desktop. * Implements the [DeviceBay Protocol](https://github.com/agentsea/devicebay). * Provides a CLI and a Python library. * The Desktops can be run locally or in the cloud. ## Motivation Why do we want this? Simple. APIs are not always available and they can be incredibly expensive to use. Agents that can use GUIs with ease have a massive advantage operating mobile phones, desktops and SaaS applications. They can work with it just like a human. GUI navigation makes any program accessible and programmable to an agent, which offers tremendous potential to gather information, automate complex, open ended tasks and control your desktop. Almost all the work in this area is currently focused on helping agents to work in browsers, but many apps aren't available on the web. That's why we created AgentDesk. It allows you to run VMs locally and in the cloud, and to control them using a Python SDK and CLI. This gives you a tremendously solid foundation for advanced GUI controlling agents. Check out an example of a complex GUI-based agent [here](https://github.com/agentsea/surfpizza). Read on to learn how to use AgentDesk. ## Installation `pip install agentdesk` If you run local VMs, you need Docker to run the containers with Desktop GUI. You also need [QEMU](https://www.qemu.org/) if you are creating QEMU desktops instead of Docker desktops. ## Quick Start: local run ```python theme={null} from agentdesk import Desktop # Create a local VM desktop = Desktop.local() # Launch the UI for it desktop.view(background=True) # Open a browser to Google desktop.open_url("https://google.com") # Take actions on the desktop desktop.move_mouse(500, 500) desktop.click() img = desktop.take_screenshot() ``` ## Running in GCP and in AWS ```python theme={null} desktop = Desktop.gce() ``` ```python theme={null} desktop = Desktop.aws() ``` ## Explore Further Playing a simple browser game Using GPT-4V to nagivate through UI Find out how to use AgentDesk via CLI Find out how to use AgentDesk Python library # GPT-4V Desktop Explorer Source: https://docs.hub.agentsea.ai/agentdesk/note A more elaborate example # GitHub Source: https://docs.hub.agentsea.ai/agentdesk/repo # Configuring AWS to create desktop VMs Source: https://docs.hub.agentsea.ai/configuration/aws Agents are designed to be capable of operating on desktop VMs. Running a full VM locally can consume a fair amount of resources, it's often preferred to run the VM in the cloud. This section will walk you through how to set up a AWS account from scratch to run a VM. ## Prerequisites * A valid email address * A credit card for billing (AWS offers a free tier for new users) ## Step 1: Create an AWS Account 1. Go to the [AWS](https://aws.amazon.com/) website. 2. Click on the **Create an AWS Account** button. 3. Follow the on-screen instructions to create a new AWS account. 4. Set up your billing information to access the free tier. ## Step 2: Create Access Keys for Your IAM User 1. Sign in to the [AWS Management Console](https://aws.amazon.com/console/). 2. Go to the **IAM** service from the Services menu. 3. Click on **Users** in the left-hand menu. 4. Select your user name from the list. 5. Click on the **Security credentials** tab. 6. Scroll down to the **Access keys** section and click on **Create access key**. 7. Download the CSV file containing the user’s access key ID and secret access key. ## Step 3: Install and Configure AWS CLI 1. Download and install the [AWS CLI](https://aws.amazon.com/cli/). 2. Open a terminal or command prompt. 3. Configure the AWS CLI with your user’s access key ID and secret access key: ``` aws configure ``` Follow the prompts to enter your access key ID, secret access key, default region name, and default output format. ## Step 4: Verify Permissions and Create an EC2 Instance 1. Verify that the CLI is authenticated and can access the EC2 service: ``` aws ec2 describe-instances ``` ## Additional Considerations: Quotas By default, AWS provides sufficient quotas for most users. However, if you need to increase quotas for your account: 1. Go to the **Service Quotas** service from the Services menu. 2. Find the quota you need to increase (e.g., `Running On-Demand Standard (A, C, D, H, I, M, R, T, Z) instances`). 3. Click on the quota, then click on **Request quota increase**. 4. Fill out the request form and submit it. AWS will review your request and notify you of the outcome. ## Conclusion You have now successfully created an AWS account, set up the necessary permissions, and authenticated the CLI to create EC2 instances. For more advanced configurations and management, refer to the [AWS documentation](https://docs.aws.amazon.com/). If you have not already run through our [Quickstart](https://docs.hub.agentsea.ai/quickstart) to setup surfkit, please do that now. Once you've done that, you're ready to create an AWS cloud VM for an agent to use with the following command: ``` surfkit create device --provider ec2 -n my-surfkit-vm ``` Replace `my-surfkit-vm` with whatever friendly name you want to use. # Configuring an Image Repository to Publish Your Agents Source: https://docs.hub.agentsea.ai/configuration/docker In order to publish an agent, you will need an image repository to store your agent's Docker image. Any image repository that can be made public will work. In this section we will walk through creating an image repository on [Google Artifact Registry](https://cloud.google.com/artifact-registry) from scratch. ## Prerequisites * A valid email address * A credit card for billing (GCP offers a free tier and \$300 in free credits for new users) ## Step 1: Create a GCP Account 1. Go to the [Google Cloud Platform](https://cloud.google.com/) website. 2. Click on the **Get started for free** button. 3. Follow the on-screen instructions to create a new Google account or sign in with an existing Google account. 4. Set up your billing information to access the free tier and free credits. ## Step 2: Set Up a New Project 1. Once logged into the GCP Console, click on the **Select a project** dropdown at the top of the page. 2. Click on **New Project**. 3. Enter a project name, select a billing account, and choose a location. 4. Click on **Create**. ## Step 3: Enable the Artifact Registry API 1. In the GCP Console, go to the **Navigation menu** > **API & Services** > **Library**. 2. Search for **Artifact Registry API**. 3. Click on **Artifact Registry API** and then click on **Enable**. ## Step 4: Ensure Necessary Permissions 1. As the owner of the project, you already have the necessary permissions to manage Artifact Registry repositories. If you are using a different account, ensure that it has the **Editor** role at the project level. ## Step 5: Install and Configure Google Cloud SDK (gcloud CLI) 1. Download and install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). 2. Open a terminal or command prompt. 3. Initialize the SDK with the following command: ``` gcloud init ``` 4. Follow the on-screen prompts to log in with your Google account and set the default project. ## Step 6: Authenticate the CLI Using Application-Default Login 1. Authenticate the gcloud CLI with your user account: ``` gcloud auth application-default login ``` Follow the on-screen prompts to log in with your Google account. This command sets up application-default credentials for the CLI. ## Step 7: Configure Docker to Use Google Artifact Registry 1. Install Docker if it is not already installed. Follow the instructions on the [Docker website](https://docs.docker.com/get-docker/) to install Docker for your operating system. 2. Configure Docker to authenticate with Google Artifact Registry: ``` gcloud auth configure-docker us-central1-docker.pkg.dev ``` This command updates the Docker configuration to use the credentials from the gcloud CLI. Replace `us-central1` with the region where your repository is located. ## Step 8: Create a Repository 1. Create an Artifact Registry repository: ``` gcloud artifacts repositories create REPOSITORY_NAME --repository-format=docker --location=LOCATION ``` Replace `REPOSITORY_NAME`with the desired name of your repository and`LOCATION`with your preferred GCP region (e.g.,`us-central1`). ## Step 9: Make the Repository Public 1. In the GCP Console, go to the **Navigation menu** > **Artifact Registry** > **Repositories**. 2. Click on the repository you want to make public. 3. Click on the **Permissions** tab. 4. Click on **Add principal**. 5. In the **New principals** field, enter `allUsers`. 6. In the **Select a role** dropdown, choose **Artifact Registry** > **Artifact Registry Reader**. 7. Click on **Save**. ## Step 10: Verify the Image in Google Artifact Registry 1. In the GCP Console, go to the **Navigation menu** > **Artifact Registry** > **Repositories** > **REPOSITORY\_NAME**. 2. You should see the image you pushed listed under your repository. ## Additional Considerations: Quotas By default, GCP provides sufficient quotas for most users. However, if you need to increase quotas for your project: 1. Go to the **Navigation menu** > **IAM & Admin** > **Quotas**. 2. Filter by the quota you need to increase (e.g., `Artifact Registry storage`). 3. Select the quota and click on **Edit Quotas**. 4. Fill out the request form and submit it. Google will review your request and notify you of the outcome. ## Conclusion You have now successfully created a GCP account, set up the necessary permissions, configured the CLI, created an Artifact Registry repository, and made it public. For more advanced configurations and management, refer to the [GCP documentation](https://cloud.google.com/docs). You can proceed to create an agent and use the registry when prompted for it. ``` mkdir newagent && cd newagent surfkit new ``` Then once the agent is ready ``` surfkit publish --build ``` # Configuring GCP to create desktop VMs Source: https://docs.hub.agentsea.ai/configuration/gcp Agents are designed to be capable of operating on desktop VMs. Running a full VM locally can consume a fair amount of resources, it's often preferred to run the VM in the cloud. This section will walk you through how to set up a GCP account from scratch to run a VM. ## Prerequisites * A valid email address * A credit card for billing (GCP offers a free tier and \$300 in free credits for new users) ## Step 1: Create a GCP Account 1. Go to the [Google Cloud Platform](https://cloud.google.com/) website. 2. Click on the **Get started for free** button. 3. Follow the on-screen instructions to create a new Google account or sign in with an existing Google account. 4. Set up your billing information to access the free tier and free credits. ## Step 2: Set Up a New Project 1. Once logged into the GCP Console, click on the **Select a project** dropdown at the top of the page. 2. Click on **New Project**. 3. Enter a project name, select a billing account, and choose a location. 4. Click on **Create**. ## Step 3: Enable the Compute Engine API 1. In the GCP Console, go to the **Navigation menu** > **API & Services** > **Library**. 2. Search for **Compute Engine API**. 3. Click on **Compute Engine API** and then click on **Enable**. ## Step 4: Ensure Necessary Permissions 1. As the owner of the project, you already have the necessary permissions to manage Artifact Registry repositories. If you are using a different account, ensure that it has the **Editor** role at the project level. ## Step 5: Install and Configure Google Cloud SDK (gcloud CLI) 1. Download and install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). 2. Open a terminal or command prompt. 3. Initialize the SDK with the following command: ```sh theme={null} gcloud init ``` 4. Follow the on-screen prompts to log in with your Google account and set the default project. **NOTE:** If you already have other gcloud accounts configured for your work or for other projects you should see something like the following: ``` Settings from your current configuration [default] are: container: cluster: surfkit-demo core: account: roko@basilisk.ai disable_usage_reporting: 'False' project: some-project Pick configuration to use: [1] Re-initialize this configuration [default] with new settings [2] Create a new configuration [3] Switch to and re-initialize existing configuration: [surfkit-demo] ``` The easiest is to **choose \[2] to create a new configuration which will you keep your old configurations.** However, if you do not wish to keep old accounts you can choose to \[1] to re-initialize the settings. If you choose two you will be prompted to name the configuration, so name it something clear to differentiate it from other configs and then you should see the following: ``` Choose the account you would like to use to perform operations for this configuration: [1] roko@basilisk.ai [2] Log in with a new account Please enter your numeric choice: 2 ``` Choose 2 and you will be redirected to your browser to click through the authentication with your Google enabled email account and password. At the end of the of the configuration process you should see an asterix next to your new email address to indicate that it is the default for logging into gcloud. You may run `gcloud config configurations list` to see the list of configurations available and then `gcloud config configurations activate NAME`, where **NAME** is the friendly name you have choosen for any given account in the first column of the list. ## Step 6: Authenticate the CLI Using Application-Default Login 1. Authenticate the gcloud CLI with your user account: ```sh theme={null} gcloud auth application-default login ``` Follow the on-screen prompts to log in with your Google account. This command sets up application-default credentials for the CLI. ### Step 7: Set up the Project in CLI ```sh theme={null} gcloud config set core/project some-project export GOOGLE_CLOUD_PROJECT=some-project ``` ### Step 8: Verify Permissions 1. Verify that the CLI is authenticated and can access the project: ``` gcloud auth list ``` If this is your first GCP account, you will see only one account listed. If you have multiple, you will see an `*` next to the active account like so: ``` Credentialed Accounts ACTIVE ACCOUNT * roko@basalisk.ai eyudkowsky@maxpaperclips.org ``` 2. Verify that the Compute Engine API is enabled: ``` gcloud services list --enabled | grep compute.googleapis.com ``` You should see the following: ``` compute.googleapis.com Compute Engine API ``` 3. Verify that project is set up: ``` gcloud config list ``` ``` [core] account = roko@basalisk.ai disable_usage_reporting = False project = some-project Your active configuration is: [default] ``` ### Additional Considerations: Quotas By default, GCP provides sufficient quotas for most users. However, if you need to increase quotas for your project: 1. Go to the **Navigation menu** > **IAM & Admin** > **Quotas**. 2. Filter by the quota you need to increase (e.g., `CPUs`). 3. Select the quota and click on **Edit Quotas**. 4. Fill out the request form and submit it. Google will review your request and notify you of the outcome. ### Conclusion You have now successfully created a GCP account, set up the necessary permissions, and authenticated the CLI using application-default login to create GCE VM instances. For more advanced configurations and management, refer to the [GCP documentation](https://cloud.google.com/docs). If you have not already run through our [Quickstart](https://docs.hub.agentsea.ai/quickstart) to setup SurfKit, please do that now. Once you've done that, you're ready to create a GCP cloud VM for an agent to use with the following command: ``` surfkit create device --provider gce -n my-surfkit-vm ``` Replace `my-surfkit-vm` with whatever friendly name you want to use. # How to Get Your Agents Running Locally or in the Cloud Source: https://docs.hub.agentsea.ai/configuration/intro Learn how to set up your cloud to work with SurfKit Agentsea is designed to run on your local machine or in the cloud. Agents can run as: * Python processes * Docker containers * Kubernetes pods Desktops can run on: * Docker * QEMU * GCE * EC2 This section will provide an in depth walkthrough on how to set up your cloud to work with SurfKit. For creating Docker desktops, install Docker. Then you can jump straight to [creating devices](../quickstart#create-a-device). Below, we guide you through both Google Cloud (GCP) and Amazon Web Services (AWS) from scratch. Both are supported by SurfKit, and Azure will be coming soon. Installing QEMU on Mac or Linux Configuring GCP to create desktop VMs Configuring AWS to create desktop VMs Configuring Kubernetes to run agents Configuring an image repository to publish your agents # Configuring Kubernetes to run agents Source: https://docs.hub.agentsea.ai/configuration/k8s Running a single agent locally can be useful for testing and development, but it's often preferred to run multiple agents in the cloud. This section will walk you through how to set up a Kubernetes cluster to run agents from scratch. Agentsea works with *any* vanilla Kubernetes cluster, all you need is a working kubeconfig. However, this guide will use [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine) (GKE) as an example for simplicity. ## Prerequisites * A valid email address * A credit card for billing (GCP offers a free tier and \$300 in free credits for new users) * **CRITICAL NOTE:** If you just signed up for your account, you will need to activate your free credits by choosing the blue "Activate" button at the top right of the screen in order to be able to create a Kubernetes cluster, which requires 900 GB of SSD storage by default and trial accounts come with only 500 GBs. **NOTE:** If you've already done the section [Configure GCP to Create a Desktop VM](./gcp), you may skip Step 1 and Step 2. If you already have a pre-existing GCP account that you're using then you can skip to Step 2 where and create a new project for surfkit. If you have an existing project that you want to use, go to Step 3. ## Step 1: Create a GCP Account 1. Go to the [Google Cloud Platform](https://cloud.google.com/) website. 2. Click on the **Get started for free** button. 3. Follow the on-screen instructions to create a new Google account or sign in with an existing Google account. 4. Set up your billing information to access the free tier and free credits. ## Step 2: Set Up a New Project 1. Once logged into the GCP Console, click on the **Select a project** dropdown at the top of the page. 2. Click on **New Project**. 3. Enter a project name, select a billing account, and choose a location. 4. Click on **Create**. ## Step 3: Enable the Kubernetes Engine API 1. In the GCP Console, go to the **Navigation menu** > **API & Services** > **Library**. 2. Search for **Kubernetes Engine API**. 3. Click on **Kubernetes Engine API** and then click on **Enable**. ## Step 4: Ensure Necessary Permissions 1. As the owner of the project, you already have the necessary permissions to manage Artifact Registry repositories. If you are using a different account, ensure that it has the **Editor** role at the project level. ## Step 5: Install and Configure Google Cloud SDK (gcloud CLI) 1. Download and install the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). 2. Open a terminal or command prompt. 3. Initialize the SDK with the following command: ``` gcloud init ``` 4. Follow the on-screen prompts to log in with your Google account and set the default project. ## Step 6: Authenticate the CLI Using Application-Default Login 1. Authenticate the gcloud CLI with your user account: ``` gcloud auth application-default login ``` Follow the on-screen prompts to log in with your Google account. This command sets up application-default credentials for the CLI. ## Step 7: Create a Kubernetes Cluster 1. Verify that the CLI is authenticated and can access the project: ``` gcloud auth list ``` 2. Verify that the Kubernetes Engine API is enabled: ``` gcloud services list --enabled | grep container.googleapis.com ``` You should see the following: ``` container.googleapis.com Kubernetes Engine API ``` 3. Ensure that you have the correct quotas set to create a new cluster. By default SSD\_TOTAL\_GB is set to 500 and a k8s cluster requies 900/ 3a. Go to your [console admin quotas](https://console.cloud.google.com/iam-admin/quotas) to see a list of your current limits. 3b. Type `ssd` into the **Filter** search bar right above the quotas. 3. Create a new Kubernetes cluster: ``` gcloud container clusters create CLUSTER_NAME --zone ZONE ``` Replace `CLUSTER_NAME`with the desired name of your cluster and`ZONE` with your preferred GCP zone. To see a list of existing zones type `gcloud compute zones list` ## Step 8: Configure kubectl to Connect to Your Cluster 1. Get the credentials for your new cluster: ``` gcloud container clusters get-credentials CLUSTER_NAME --zone ZONE ``` Replace `CLUSTER_NAME`and`ZONE` with the same values you used when creating the cluster. 2. Verify that `kubectl` is configured correctly: ``` kubectl get nodes ``` This command should return a list of nodes in your cluster. ## Additional Considerations: Quotas By default, GCP provides sufficient quotas for most users. However, if you need to increase quotas for your project: 1. Go to the **Navigation menu** > **IAM & Admin** > **Quotas**. 2. Filter by the quota you need to increase (e.g., `In-use IP addresses in region`). 3. Select the quota and click on **Edit Quotas**. 4. Fill out the request form and submit it. Google will review your request and notify you of the outcome. ## Conclusion You have now successfully created a GCP account, set up the necessary permissions, and configured the CLI to create and manage a Kubernetes cluster. For more advanced configurations and management, refer to the [GCP documentation](https://cloud.google.com/docs). Now run an agent on Kubernetes ``` surfkit create agent -t pbarker/SurfPizza --runtime kube ``` # Installing QEMU on Mac or Linux Source: https://docs.hub.agentsea.ai/configuration/qemu Learn how to set up QEMU on Mac or Linux ## Installing QEMU on Mac and Linux QEMU is an open-source machine emulator and virtualizer. Here is how you install QEMU on Mac (using Homebrew or MacPorts) and Linux. ### Installing QEMU on Mac #### Using Homebrew Homebrew is a popular package manager for macOS. Follow these steps to install QEMU using Homebrew: 1. **Install Homebrew** (if not already installed): ```sh theme={null} /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` 2. **Install QEMU**: ```sh theme={null} brew install qemu ``` 3. **Verify the installation**: ```sh theme={null} qemu-system-x86_64 --version ``` #### Using MacPorts MacPorts is another package manager for macOS. Follow these steps to install QEMU using MacPorts: 1. **Install MacPorts** (if not already installed): Follow the instructions on the [MacPorts installation page](https://www.macports.org/install.php). 2. **Update MacPorts**: ```sh theme={null} sudo port selfupdate ``` 3. **Install QEMU**: ```sh theme={null} sudo port install qemu ``` 4. **Verify the installation**: ```sh theme={null} qemu-system-x86_64 --version ``` ### Installing QEMU on Linux #### Using Homebrew Homebrew is also available for Linux. Follow these steps to install QEMU using Homebrew: 1. **Install Homebrew** (if not already installed): ```sh theme={null} /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` 2. **Install QEMU**: ```sh theme={null} brew install qemu ``` 3. **Verify the installation**: ```sh theme={null} qemu-system-x86_64 --version ``` #### Using Linuxbrew Linuxbrew is a fork of Homebrew for Linux. Follow these steps to install QEMU using Linuxbrew: 1. **Install Linuxbrew** (if not already installed): ```sh theme={null} sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)" ``` 2. **Add Linuxbrew to your PATH**: ```sh theme={null} test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv) test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv) test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.profile ``` 3. **Install QEMU**: ```sh theme={null} brew install qemu ``` 4. **Verify the installation**: ```sh theme={null} qemu-system-x86_64 --version ``` # Deep Dive into Robbie Gen 2 Source: https://docs.hub.agentsea.ai/deep_dive_gen_2 A detailed explanation of how Robbie Gen 2 works Meet Robbie, our Gen 2 agent. He's available [right now on Github](https://github.com/agentsea/robbie-g2/) and he's completely open source. You can download him immedietly and modify him and send him out into the wide world of the web to do tasks for you. Robbie navigates GUIs to solve tasks for you and he runs on top of various tools from our AgentSea stack. Check out our [Deep Dive video](https://youtu.be/R6rR27I6oFg), and you'll find even deeper dive right here in this article. Let's walk through a bunch of techniques we use to help him on his way, which we've pioneered over the last few months. # The Mind of Robbie While many teams are using the "team of agents" approach, like the excellent CrewAI team, and getting some cool results, we prefer modeling the parts of the brain in our agents, similar to the "thousand brains" approach of making decisions through consensus of different parts of the "mind" of the agent that excel at different pieces of the puzzle. We divide our Gen 2 agent into the following: * Actor * Critic * Neocortex * Body Robbie Gen 2 Architecture The **actor** is the primary decision maker of the agent. He makes all the decisions and takes actions in the real world. The **critic** studies actions and considers whether they were a success or failure as well as alternative paths and ideas. The **neocortex** does a lot of things in the human brain, as it takes up over 70% of the space in our heads, but in our case we use the neocortex to predict the next few actions. The neocortex is the seat of "simulations" into the future, letting us make decisions in real life before we make them and play out how it might go, so we can pick better actions. In later versions we will make a more complex long term and short term prediction system but in this case, just predicting they you have to first type a search into Google and then hit the search button is pretty strong for us already. The **body** takes actions through its tools, in this case the virtual desktop served up by AgentDesk. It's a hacky approximation of some of the concepts in [Yann LeCun's next-gen neural architectures](https://ai.meta.com/blog/yann-lecun-advances-in-ai-research/) and it also takes inspiration from [Richard Sutton's reinforcement learning](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf) work and builds into our own next-gen architecture which we'll share later in the post. Again we're not training novel architectures, but as an applied AI team, we're doing what humans do best, we're abstracting ideas and applying them in new and novel ways. Now let's jump in and see through Robbie's eyes. # How Robbie Sees the World Robbie is a pure multimodal bot. Like a Tesla car, he makes decisions only by what he sees and reads. Most of the agents out there today use something like Playwright to script browser interactions. That's a cool approach and you can easily use Playwright as a loadable device/tool in the AgentSea ToolFuse protocol but this doesn't help if you want to click around a desktop or a mobile app. For that we knew we needed a pure multimodal approach. Just one problem. We discovered that models like GPT-4/4o and Claude Opus/Sonnet are absolutely terrible at three key things: * Returning coordinates * Moving the mouse * Drawing bounding boxes. The reasons shouldn't be surprising. They just weren't trained to do any of these things. But we've pioneered a number of tricks to help the models give us what we need to get through a GUI dynamically. We divide our techniques into "cheap" and "expensive." We don't mean money wise, although these agents can eat tokens like Pac Man so they can be expensive that way too. * By "cheap" we mean quick, dirty and fast. * By "expensive" we mean round trips to the cloud and using heavier/slower frontier models or big open source models to get what we want. We try to use as many cheap methods as possible because they are much faster. There are a number of ways to squeeze amazing results out of smaller, cheaper, faster methods and classical computer vision if you get creative, which is the essence of applied AI. But for approximations of higher intelligence you need the much slower frontier models and there is no getting around it and that means everything slows down considerably because of round trips to the cloud and the slow inference of these models. Robbie uses three major techniques to navigate the web: ### OCR Positioning This is our absolute favorite cheap method and it’s lightning fast when it works. A few months ago I wondered if we could use the position of OCRed text as a way to get coordinates for buttons and links. Turns out we can. Many models and libraries like, [Tesseract](https://pypi.org/project/pytesseract/), support getting text position back. Unfortunately, most of them are not very good at accurately OCRing and giving coordinates. That set us off on a search to find the right model. Some were too big and too slow and most of them just, well, kinda sucked. And then we hit on [EasyOCR](https://github.com/JaidedAI/EasyOCR). It is *God like* at finding text accurately and it is super fast and lightweight. When we first ran it through our test suite of GUI images and it came back with 100% we thought it was a mistake but subsequent tests proved its very strong (though not 100% in real world settings by any means.) EasyOCR If the element on the page has text it is usually easy to find. The MLLM tells us what is it looking for with its next action description and if we find a match we can easily click it quickly without more round trips to the Big Brain in the cloud. EasyOCR ### The Grid at the Center of it All While models are not very good at knowing precisely where to click they are good at knowing approximately where to click. They can tell you that the search button is in the middle of the page or on the lower left corner. So we help the model do this with much better precision by layering a bunch of dots with over the image and ask it to pick the number closest to the thing its looking for right now. Honestly, it’s easier to show than to explain because it makes intuitive sense when you see it: Grid From there we can progressively zoom, based on the number the model returns, overlaying the screenshot again, to get more granular and zero in on what we want to click. In the below screenshot, you see a second level zoom on Twitter as the model looks to find the "compose tweet" button. Grid This approach is slow but it works very, very well. It's biggest downside is it's "expensive" in that it involves as many as three round trips to the cloud to talk to the Big Brain in the sky. ### Region of Interest Region of insert is a hybrid classic computer vision and Big Brain in the cloud approach. It's faster than grid but still slower than OCR. In essence, it involves using canny in opencv to find all the right bounding boxes and then intelligently splitting the regions so that they show entire sections without cutting off parts of the piece of the puzzle the model is looking for at the moment. We then layer all these regions onto a grid that we call the "Jeffries Composite" and asking the model to pick the number that has the element it wants. Again, it is easier to just show you: Composite In older versions of our agents, our desktop slicer often cut right through the element we were looking and so that search bar or button spawned two or three boxes, which confused the model. By intelligently not cutting something that is in the middle of a bounding box, we now get very clear sections of the image to work with 95% of the time. Combined, these three techniques do very well at finding their way around a GUI, even without any model fine tuning (which we're working on too and [you can read about here](https://huggingface.co/collections/agentsea/waveui-6684c5ab7b72cda3a523674c).) Fine tuning will give us a much faster and more robust clicking model and we'll fall back to these classical techniques as needed in v3. ## Running Robbie Gen 2 ### Setup 1. Setup your OpenAI API key: ```sh theme={null} export OPENAI_API_KEY= ``` 2. Install/upgrade SurfKit: ```sh theme={null} pip install -U surfkit ``` 3. Clone the repository and go to the root folder: ```sh theme={null} git clone git@github.com:agentsea/robbie-g2.git && cd robbie-g2 ``` 4. Install dependencies: ```sh theme={null} poetry install ``` ### Creating required entities 5. Create a tracker: ```sh theme={null} surfkit create tracker --name tracker01 ``` 6. Create a device: * If you are using Docker locally: ```sh theme={null} surfkit create device --provider docker --name device01 ``` You can also skip the `provider` flag, because `docker` is the default provider: ```sh theme={null} surfkit create device --name device01 ``` * If you are using QEMU: ```sh theme={null} surfkit create device --provider qemu --name device01 ``` * If you are using GCE: ```sh theme={null} surfkit create device --provider gce --name device01 ``` * If you are using AWS: ```sh theme={null} surfkit create device --provider aws --name device01 ``` 7. Create an agent: ```sh theme={null} surfkit create agent --name agent01 ``` ### Solving a task ```sh theme={null} surfkit solve "Search for common varieties of french ducks" \ --tracker tracker01 \ --device device01 \ --agent agent01 ``` # Introduction Source: https://docs.hub.agentsea.ai/introduction The AgentSea platform lets you build, deploy and share agents with ease. Hero Light Hero Dark ## Introduction The AgentSea platform delivers a collection of libraries and tools for building AI agent apps. We favor the UNIX philosophy of do one thing and do it well. Making our tools easy to use, easy to extend, and easy to mix and match. Use the tools one by one or stack them together into a single agent app. You can also use our tools with other popular frameworks like LlamaIndex and LangChain. Our tools ▼ * [SurfKit](https://github.com/agentsea/surfkit) an orchestartor for building and launching agents locally, in a docker container or in the cloud. Think of it as k8s for agents. * [DeviceBay](https://github.com/agentsea/devicebay) offers pluggable devices ready to be used by AI agents, complete with a UI experience. * [ToolFuse](https://github.com/agentsea/toolfuse) a library that wraps up scripts, 3rd party apps and APIs as `Tool` implementations for agents. * [AgentD](https://github.com/agentsea/agentd) a powerful daemon that makes a Linux desktop OS accessible to your bot, like a remote desktop app but where the agent takes all the actions. * [AgentDesk](https://github.com/agentsea/agentdesk) a library for running `AgentD` powered VMs as `Tool` instances on any cloud. * [Taskara](https://github.com/agentsea/taskara) task management for your agentic systems. * [ThreadMem](https://github.com/agentsea/threadmem) a library for building multi-role persistent threads that keep track of all the messages and dialogues with your agents. * [MLLM](https://github.com/agentsea/mllm) a library for simplifying communication with multiple Large Language Models (LLMs) and multi-modal LLMs. Build your own agent or use our alpha agents. Our initial batch of agents focus on multimodal navigation of GUI interfaces. Our prototypes use a combo of old school computer vision techniques and some new tricks of our own applied AI methods. Our agents ▼ * [SurfPizza](https://github.com/agentsea/surfpizza) an agent that explores by slicing up the screen and returning a composite to the multimodal model so it can pick where to go next. * [SurfSlicer](https://github.com/agentsea/surfslicer) divides up the screen into dots that signify regions and the multimodal model picks the dot closest to what it's looking for and then zooms in and does it again, zeroing in on its target. * **SurfNinja** (coming soon) - A precision-based second gen AI agent. * **SurfMonsta** (coming soon) - Our best performing agent: It combines a number of our techniques, like SurfSlicer regions, the SAM model for bounding and segmenting a GUI, OCR for text positioning and a GAN for upscaling smaller slices of images to give the multimodal model the best resolution. ## Demo