Hey HN, we’re Nitish and Prateek, the founders of Coasty (https://coasty.ai/computer-use). We’re building computer-use agents that can complete workflows inside legacy desktop software and web applications without usable APIs.

Developers send Coasty a natural-language task either through our consumer app or through our API, select a machine or browser environment, and any relevant credentials or files. The agent then operates the interface through screenshots, mouse, and keyboard input, verifies the result, and returns a structured run record with screenshots, actions, outputs, and errors.

Here is a raw demo of an agent completing a workflow in a legacy application(It’s a mockup): https://drive.google.com/file/d/1ZghU_3vsAYhHVz1bsvE0pkvZYk7...

A lot of important software is still difficult to automate. Healthcare teams submit prior authorizations through payer portals, accounting teams enter data into desktop applications, and operations teams move information between internal systems, spreadsheets, and remote desktops. Many of these applications have no API, incomplete APIs, or integrations that take months to build.

The usual alternative is RPA, record a sequence of clicks and replay it. That works when the interface and workflow are predictable, but it often breaks when a button moves, a pop-up appears, a page loads slowly, or the application enters an unexpected state.

Coasty takes a different approach. The agent observes the current screen, decides what action to take, executes it, and then observes the resulting state before continuing. It does not require DOM access, an accessibility tree, selectors, or an application-specific integration, so the same API can operate browsers, remote desktops, and older Windows applications.

A simplified request looks roughly like this:

  run = coasty.runs.create(
      environment="vm_123",
      task="""
      Open the patient record in the billing portal.
      Enter the attached authorization data.
      Do not submit if the member ID or procedure code does not match.
      Return the confirmation number.
      """,
      files=["authorization.pdf"],
      approval_required=["final_submission"]
  )

The response includes the final status, extracted outputs, a replay URL, and a timestamped event log:

  {
    "status": "completed",
    "output": {
      "confirmation_number": "PA-184392"
    },
    "replay_url": "...",
    "events": [
      {
        "type": "verification",
        "field": "member_id",
        "result": "matched"
      }
    ]
  }

The API can also pause a run for human approval, retry from a checkpoint, or return control to the developer when it encounters a condition the workflow did not anticipate.

We started working on this last summer, because we saw that models were getting better at vision but kept seeing a gap between computer-use demos and the reliability needed for production workflows. Getting an agent to complete a task once is fairly straightforward. Getting it to repeat that task, recover from unexpected states, avoid silently entering incorrect data, and produce evidence of what it did is much harder.

We built several layers around the underlying computer-use model. The system tracks the expected state of the workflow, detects when the application has diverged from that state, and can re-plan instead of continuing blindly. Developers can define invariants such as “the patient name must match the source document” or “never submit without approval,” and the agent checks those conditions during the run.

Each run happens in an isolated virtual machine. We expose APIs for provisioning environments, uploading files, starting tasks, streaming events, inserting human approvals, and retrieving the full replay and audit trail. Environments can be kept alive across runs when the application has a long login flow or persistent local state.

One problem we are still working through is the tradeoff between speed and reliability. The agent can move faster by taking fewer observations and verification steps, but that becomes risky in workflows involving patient records, payments, or regulatory filings. We currently bias toward slower execution with more checks and let developers configure approval points and verification policies.

We are initially working with healthcare operations teams because their workflows combine many of the hardest conditions: payer portals, EHRs, PDFs, spreadsheets, remote desktops, and actions where quiet mistakes are expensive. We also expose the same infrastructure through the developer API for teams building their own agents and vertical automation products.

We currently charge based on agent runtime and workflow volume, with separate pricing for dedicated environments and enterprise deployments.

We’d especially appreciate feedback from people who have built and/or used browser agents, RPA systems, desktop automation, or agent infrastructure. We’re curious which parts of the API you would want direct control over, where you would prefer higher-level abstractions, and which failure modes have been hardest in your own automation systems.

If you've hit weird failure modes automating software like this, we want to hear about them. We'll be here all day answering questions and taking notes!

user:	nkov47
created:	December 23, 2024
karma:	18