Show HN: I'm working on an easy way to add data to files. Thoughts?

Show HN: I'm working on an easy way to add data to files. Thoughts?(github.com)

8 points by padho 8 years ago | 4 comments

mtmail 8 years ago |

You have extractor and loader classes, I think the industry term at least in data warehousing is a https://en.wikipedia.org/wiki/Extract,_transform,_load system. Enrichment sounds cool, too.

I like it, at least more than Zapier and similar pipelines. We have a couple of clients that use our API (geocoding[1]) to enrich their records inside an ETL. Be prepared that some APIs are rate-limited or quota-based, need API keys or other authentication and might return 'over-quota' error messages, never mind latency.

[1] https://geocoder.opencagedata.com/

padho 8 years ago | |

Hi, thanks for your comment and the link to your Geocoder. I think it's exactly the use case that I faced several times in the past.

I just saw that the Geocoder API is free with certain limitations. If I find some free time, I would like to create an enricher for it. Maybe it can help your clients (and other people of course) to save some time :)

lozzo 8 years ago |

my thoughts: I was looking for the way you handle exceptions. And found none. For me any enrichment API needs to be designed to handle exceptional case and make it easy to deal with them.

Say that during enrichment I might want to do some lookups what if one of them is not working...

padho 8 years ago | |

Hi, thanks for your feedback. You’re right. There is no exception handling yet. I also plan to add some kind of state handling, so that jobs can be paused/resumed. What do you mean by making lookups?