allow a 'spot timeout' when starting spot clusters

[jtriley]

From the mailing list: http://mailman.mit.edu/pipermail/starcluster/2012-March/001122.html

It would be great if there were some kind of timeout option for spot
instances, ie if they aren't started by some deadline then shut down
everything and return an error exit code. That way a script running
starcluster could then re-try with regular ondemand instances if there
is a deadline to getting some work done.

This is useful although it could wind up wasting several instance
hours if all but one spot instance fails to come up for example. A
better approach would be for the start command to monitor spot instance
requests up until a deadline at which point the spot requests which
haven't come up yet are cancelled and on-demand instances are requested
in their place. This would avoid having to write this logic in your
script completely and also enables a mode where you 'get as much spot as
you can' up until a specified timeout...

[FinchPowers]

Business rules for this can get extremely complex. I intend to develop state machine support for the load balancer add_node operation. The loadbalance command will specify a state machine + config to use.

[derkuci]

A timeout option could be useful (and more clearly defined) for "addnode". If addnode fails to grab a spot instance within a given time, it can then cancel the spot request and exit with error. This gives a chance to a script running starcluster to try with a different spot type or on-demand instances, while not necessarily mess up the whole cluster.

My current approach is using /usr/bin/timeout with starcluster addnode, but I can't find a nice way to cancel the open spot requests when timeout happens.