Ask HN: How should non-technical users “pull request” data changes?

3 points by quotient 2 years ago | 2 comments

Curious if anyone else has this problem:

We have some configuration data that our non-technical, business employees that need to update occasionally.

That has to be approved to make sure it's correctly formatted, and then pulled into our production codebase to run.

So we've set up a flow: there's a spreadsheet (CSV) in the github repository. We've taught the non-technical user how to download it, edit it in Excel, and then submit a PR.

We then have an engineer approve the PR (as usual), and then the CSV gets pulled in on our regular deploy cycle. On startup, the server loads the configuration data from the CSV. (For configuration data that we need to persist to the DB, the server updates those rows on startup.)

Obviously this is clunky for everyone involved. Has anyone found an out-of-the-box solution for this? A regular SQL client for non-technical users doesn't work because we want (1) eng approval on data changes and (2) to coordinate data changes with our deploy cycle in general.

theamk 2 years ago |

Given that you want this to be part of PR/deploy cycle, I'd imagine most standalone options would not work. But you can significantly simplify your existing process.

1. Keep the primary copy on server somewhere -- in google sheets, or in Airtable, or even as Excel file on a shared drive. Make sure it has backups (or can be easily re-generated from csv in github repo)

2. Create an automation (github action, jenkins job, whatever) that takes current version of spreadsheet, exports to CSV, and makes a PR with new contents (and possibly submits for review)

3. In normal case, business employee changes primary copy, then starts the automation to create a PR. They can monitor created PR for any errors and/or merge progress.

4. There is a chance of accidental massive damage, like someone deleting the entire column by accident or restructuring data in incompatible way. That's when you step in and manually restore from backup/from csv file in github.

5. There is no locking, two business users might step on each other toes. But you say "only need to update occasionally" so this is hopefully fine for you...

PaulHoule 2 years ago |

What specifically is the problem?

Using git for everything is not a bad idea because then you have consistent ids for each version and you can manage everything together.

You could build something that can manage multiple versions in SQL with an approval process (particularly if the order of the rows doesn't matter) but changes there won't be synchronized with git without more trouble.

Look for little things you can do to streamline the process you already have, for instance you might be able to hide the git manipulation behind some scripts if you haven't already.