Ask HN: What would you compute on 2000 badly behaved worker nodes? I work for CrowdProcess, and we built a distributed computing platform that runs on Web Workers (badly behaved, i.e. volatile connections with some latency), to monetize websites replacing advertising. We already have +2000 nodes and have been getting pretty good speedups in Monte-Carlo based algorithms. We also have the simplest ever API for a distributed computing platform. What else would you build using it ? |
This is a fairly common idea, and it usually gets shot down. I am surprised you guys made it this far into the process. Unless maybe there is some user opt-in model?
For example, do you know how much more expensive this is (e.g. Wh/Tflop) than traditional datacenter grid computing? Or, how you are essentially charging users without their knowledge? I'm sure the legal system will love that one.
It's as immoral as advertising, maybe even less. In advertising you show up at a web page and see tons of things that you did not want to see or did not bring you to that web page, sometimes shift your focus and annoys you. It's the same with CrowdProcess, except instead of annoying you, we annoy one CPU core. We believe that while being more expensive than traditional datacenter grid computing, it may be less expensive because it only has to outperform ads. We don't compute on all the CPU cores, of course, only on one.
We actually ask websites to tell they're a part of this, but we cannot control what they do because they can simply display:none.
You could certainly just check to see they're using it properly. Do a screen scrape or even have someone hit the page every month or two, ban anyone abusing it by not notifying users.
You've actually managed to convince yourself that, haven't you?
It isn't because:
1: the user is paying for the electricity being wasted by you. A tab left open could be significantly detrimental.
2: it will cause real problems for mobile users who will be wondering why their battery's flat.
An opt-out is under way, one website requested an opt-in but we haven't translated that yet: http://cdn.crowdprocess.com/opt-in.html
Look at botnet owners.
Do you have any clues how to find it?
Operations use MIPs a lot. Usually via an excel spreadsheet :s However people expect a good interface to these problems, its not trivial
Because having thousands of systems as distributed web crawlers would be really really cool.
took me some seconds to understand this subtle one.
If you can't prioritize something as important as running an ethical (and law abiding - take a very close look at the ramifications of unauthorized computer usage, which I think it could be argued you're doing with this platform) business, then you really shouldn't be in business.
The incentives must be: if you do not comply, your content won't be monetized (as would happen with ads).