The SpudFarm: converting test environments from pets into cattle

About a year ago I was trying to improve our automated deployment and testing processes but found that getting access to a functioning environment reliably just wasn't possible. At the time our test environments were pets. Each was built partially by script and then finished by hand with a great expenditure of time, effort and frustration for everyone involved. After some period of use, that varied depending on what you tested on the environment, it would break again and you'd have to make some, frequently wrong, decision about whether to just start fresh (that could take up to a week) or try to debug the environment instead (that could take even longer and often did). Here's how we went about automating the creation and management of our test environment to increase developer productivity, reduce costs and increase our ability to experiment with infrastructure configuration with reduced risk.