Introducing Chronos: A Replacement for Cron(nerds.airbnb.com) |
Introducing Chronos: A Replacement for Cron(nerds.airbnb.com) |
We wrote a small C program that serves as a scheduling daemon with Redis; we have a keyspace in Redis that can be used to schedule millisecond-granular periodic or one-shot tasks with a flexible "Chronic"-like specification syntax.
It is hugely more convenient and usable than cron. Once you have it, you immediately spot lots of opportunities to factor systems into scheduled jobs that you might have avoided doing if it meant you had to deal with cron.
So a small bit of Python is enough to perform ephemeral leader election -- for the current minute -- in Zookeeper. Prefixed to otherwise stock invocations a set of machines run the same thing, one of them wins, nobody gets paged.
* * * * * my long and winding command
[becomes]
* * * * * cron-coord somename my long and winding commandThat's actually pretty interesting. Can you give a few examples that fall out of having a more fine grained (millisecond) resolution?
One advantage to doing this in Redis was that our events aren't just "run a program" (though we can do that); we can also push onto a queue that consumers BLPOP from, or send a pubsub message to a bunch of consumers, or increment counters.
later
Makes you think, it's nice that Redis is minimal, but a core feature that might work really nicely with Redis is timer support.
I'm not saying I'll never publish code, just that I've become choosy about it in my advancing years.
Chronos is built on top of frameworks, needs a few services to run.
Certainly it has value, but marketing it as a cron replacement is wrong imo. Chronos is one of these "web-server-service-tool" thingie, but no, it's not cron.
Every day, in a bizarre manifestation of Poe's Law, L33tStart seems less and less a parody.
[0] http://www.reddit.com/r/programming/comments/14ay0r/hacker_k...
Complex, interdependent systems without slack. Now add production pressures. You have all the ingredients of a "Normal Accident"[1].
You should do the opposite of this.
The benefit having something so simple and singular in function with no dependencies is that there's so little to break. While I appreciate the need for having something with more capability, calling it a "replacement" is a bit facetious.
I wish them success and call it an airplane. I'll stick to my hang glider.
[1] - http://incubator.apache.org/mesos/papers/nsdi_mesos.pdf
You still get cron, and the means to manage them across one's servers.
The crontab file has a weird syntax, that in some respects seems to look like a shell script, but isn't really one and some shell constructs work, other not. There are lots of ways you can make a mistake in writing the commands to be executed in a way that the command you intended won't run but will fail silently and you won't get any trace of an error happening. It is hard to even extract some common parts of commands and put them into a variable. I wasted hours and hours debugging weird cron errors. One case was where the crontab of one of the users was moved over to be the system crontab and strangely didn't work. Well, turns out the system wide crontab has one more field, but cron will not signal an error even in an obvious case like this, it will just fail silently (and this is the thing cron is really good at in my experience).
Of the various approaches I've had to depend on in recent memory, from perl scripts querying a central Db right through to eye wateringly expensive Control-M or Autosys in larger envs, It's plain old cron, fronted by config management (cfengine, puppet, chef, salt - it doesn't matter which) that has proven most dependable, easiest to train others on and simplest to debug.
Might be worth checking out if you are building large data flows. We probably run 10k Luigi "tasks" every day, of which the majority is Hadoop jobs. They are all organized in a large dependency graph expressed within Python, and you also get visualization, exception handling, atomic file operation, etc.
Now all that said, I'm not uptodate on that side of things and even less uptodate on open source alternative, though my quick look at this does indicate that it is a start in the right directon and can only get better. So quick look and glance over gave me a good gut feeling, which is always nice to have.
Seems like a decent idea. According to Wikipedia, Ubuntu considered it in 2006 when they were looking for a new unit system, but didn't like that it was under Apple's own license, which was shortly thereafter changed to the Apache license.
I'll remember it if I want to do something unconventional on a system.
Seriously, plists suck.
1. http://jason.the-graham.com/2013/03/06/how-to-use-systemd-ti...
Something is deeply wrong with this world.)
I suppose the definition of job scheduling is dependent on what your workload entails. Is it something you expect to complete in less than a second, a minute, or a big resource intensive HPC job you expect to run for hours? What sorts of dependencies do you have between the things you want to run? Will it run in the main event loop, separate thread, separate process, or on a separate host? I think this will reflect in the tool you choose for scheduling, or cause you to write your own.
It sounds to me like the target workload for Chronos is big slow batch jobs (ETL, data analysis). HPC job schedulers take care of things like resource management (run this job on this machine because it has the lowest load) and dependency management (don't run job b until job a completes), whereas cron handles running periodic tasks and that's about it. Chronos is nice because it looks like it is combining the two and includes a nice looking web based control panel to boot.
psubscribe __keyevent@0__:expired cron_*
psetex cron_run-me-in-1s 1000 0Currently keys don't actually really expire from the datastore until someone tries to access it after its TTL runs out. Changing redis so it will actually actively expire keys at the precise time the TTL runs down sounds expensive... but I would love that feature nonetheless.
Please open source this. Projects like these are some of the most valuable ones. It's a generalized solution to a very common problem. In fact, what you've described is one of the most elegant designs for this type of problem. It enables any program to schedule other programs just by using a standard Redis interface, for example. Almost every programming language already has a Redis library, meaning it's effectively zero additional work to use your system. Something like your project is sorely needed.
If you want help with documentation, I'm happy to offer it.
Seems like it's all downside and not much upside; it's an unambitious little C project that would mostly give people an opportunity to write blog posts about my C programming style. :)
So was Unix, originally. Those who would deride elegant design due to the choice of language are both short-sighted and mean, and aren't worth worrying about. I know how hard that can be to deal with -- people's negativity tends to bother me a lot, too -- but the world needs more production-grade solutions to real problems. It sounds like your solution has been in use for some time now, and has proven itself effective in the field.
I'm sorry you feel that way. I hope you change your mind, advancing years or no.
I'm with Sillysaurus, this is a great solution to a common problem.
Pawn off the code onto me and I'll publish it under my name if you're that worried about getting bad publicity. :P
I laughed first, then realised there is more than a ring of truth to that line :/
If you do release (pseudonymously or otherwise), then release it and please announce to those of us, who are all genuinely interested in it in the first place :)