S3 Object Lambda: Use Your Code to Process Data as It Is Being Retrieved from S3

S3 Object Lambda: Use Your Code to Process Data as It Is Being Retrieved from S3(aws.amazon.com)

18 points by jonburs 5 years ago | 5 comments

ghoomketu 5 years ago |

I really love all the cool features AWS keeps adding to their offerings but god does the bill adds up so quickly.

I just realised that I was paying almost $100 a month for things I'm now getting for almost free by just choosing alternative providers like Digitalocean, Cloudflare, etc.

- For example AWS charges 50 cents per DNS zone. I have had a lot of domains in route 53 (it was part of a setup script) which added almost $25 to my bill each month. I moved all my domain to DO and it's $0 now

- Then even with S3 things start to add up quickly. Cost associated with data storage and bandwidth is insane compared to spaces which is $5 flat and I've never exceed what comes in it.

- Most expensive were the EBS snapshots or something. I have been AWS for god knows how long and there were just too many snapshots (AMIs) that were there. Don't think you can download/upload then but they started adding quite on my bill.

- Load balancer also adds like $2 or $5 or something per instance if I remember correctly. Not sure what it is now.

Anyway point being if you are a solo dev and you're not too careful about costs (read lazy), AWS bill can be a good chunk of change per month .

llarsson 5 years ago |

Run a lambda every time you GET your object, so it can do data transformations. Seamlessly. You won't even notice until your bill comes.

And you won't feel like optimizing this either, because now you have tools and processes that depends on this behavior.

Seems easier to me to do this via regular Lambdas instead, but also save the data as an automatic cache for some amount of time. S3 can clean it out automatically. So your lambda will either fetch the cached version or compute it for you upon request.

Storage is much cheaper than processing, that's for sure.

CSDude 5 years ago |

Although cool feature, it can be a good way to go broke in 10 minutes

jfengel 5 years ago |

How is it different from writing a lambda and connecting it to S3? Is it just a pathway to make that easier, or is there some speed/cost benefit?

DerivativeWork 5 years ago | |

It's behind the S3 API & endpoints and integrates with AWS IAM. It's not a perfect drop-in fit for an existing application that talks to S3 but much closer than having to adapt it to a Lambda or API Gateway.

Lambda also has the limitation that it can't work with outputs greater than 6 MB and those outputs cannot be streamed - the entire content needs to be prepared before any of it can reach the caller.

I don't think of this as a replacement for direct to Lambda or API Gateway based solutions. Rather, when you need to be able to modify what an existing, possibly difficult to modify, application sees when it talks to S3.