Ask HN: How have you implemented human-in-the-loop workflows?

37 points by jpochtar 2 years ago | 14 comments

I’ve seen a few startups now re-implement the same things for human-in-the-loop workflows.

These are where a product is backed by a human somewhere in the world working a queue. Content moderation is the classic example, where when a user “flags” a social media someone on the other side of the world looks to decide if it’s violating or not.

I’ve seen each one re-implement reviews (QA), task assignment, worker productivity measurement, Slack notifications, SLA/due dates, and so on.

I’ve written a draft of a framework at http://tinyurl.com/opsqueue-framework, but before building anything want to learn more about how other software engineers have solved this problem.

If you’ve implemented human-in-the-loop workflows before, how did you do it?

mattbrewsbytes 2 years ago |

How is this problem any different than a task/ticket/todo tracker?

I don't see custom built todo trackers as adding value to that workflow. What features are you thinking would require something that any of the 100's of SaaS offerings out there don't do that would be needed?

jpochtar 2 years ago | |

totally— this is for integrating into custom internal tools. Consider content moderation: you wouldn't want mods getting tickets in JIRA then switching to a different tool to action them.

If switching between screens is not a problem, by all means prefer Linear/Asana/Jira/etc!

cotera-grant 2 years ago |

I come from Ruby on Rails world at a medium sized software company. I think we got a lot of leverage from how good our wrapper was around sidekiq. We asked our engineers to break down their business logic into small to medium idempotent "jobs" and then we'd give them a class they could inherit from that would hook up all the alerting / observability / rate limiting / retry / SLA stuff.

I really liked the "Job Queue" abstraction, because it gave our engineers a good mental framework to think about scalable computter processes. Curious if there's some parallel where the workers are humans instead of ruby processes?

jpochtar 2 years ago | |

yes! One of my original thoughts was "Job queue, but for humans"

koliber 2 years ago |

Would it help to think of it as two workflows? The first workflow ends with a human. The second workflow is triggered by the human. To build those two workflows, use whatever tools allow you to build such no-human-in-the-loop workflows.

apercu 2 years ago | |

Wouldn't you want to track the flow through the all components though?

Case management for example - case is created, case is triaged, case is (hopefully) closed. That way there is visibility.

If it's workflow A/tool A, some human using tool B, and then workflow C/tool C, absent integration you lose external visibility at each step, or you have to go check in Tool A to see if work flow made it to Human, and so on.

jpochtar 2 years ago | | |

yeah one flow for all the stages is the way. I like having a "status" column, and have a different task queue depending on the status

fswd 2 years ago |

Yes, I have implemented HITL. There's a task queue(psql skiplock). You make a request (insert row). It builds a "workflow graph" of your request and compiles it, a combination between a FSM and graph program. It runs an iteration on the workgraph, and then if successful, submits the next step in the queue as "pending". the user then inspects and accepts the tasks, which then run another iteration of the workgraph. If the user rejects it they can click "playground" and manually do the step.

jpochtar 2 years ago | |

is the "workflow graph" entirely custom built or is there a library/framework/service for it? Would love to hear more

fswd 2 years ago | | |

sorry I meant to say or explain computational graph, and yes langgraph was reintroduced recently last week.

zurawiki 2 years ago |

What about just using Linear or another off-the-shelf task management software?

Seems like most of the features are covered there

jpochtar 2 years ago | |

Totally— OpsQueue is for custom internal React apps to have integrated task management. If workers can switch between Linear and your internal apps, or you don’t have a custom internal React app, you should use Linear.

In some ways OpsQueue’s goal is to be a headless, API-first Linear.

nekudotayim 2 years ago |

I worked at a CRM/BPO company doing something like this for customer support inquiries, everything was pretty bespoke. We did all the productivity measurement off of, IIRC, Redshift tables fed by Kafka events (probably some airflow in there to ETL) published during task assignment.

oytuntez 2 years ago |

I have my eyes on https://temporal.io/ for similar purposes.