Ticketmaster has roughly ~3000-4000 Linux servers and my team manages about 900 of them.
We use an internal tool named provision that has recently been opensourced thanks to our legal
team.
Provision's modular design is written around plugins and actions. Each plugin is a small perl module.
The current plugins are:
- dns.pm manages bind >= 8
- Add, remove, alias names via CNAME
- Adding or removing forward and reverse zones with lots of error checking
- "deprovision" an alias and "reprovision" it as a real host
- Find, warn, and optionally fix common zone file mistake
- filer.pm manages Network Appliance Filers and Nearstore network attached storage
- Add and resize user quotas
- Add or remove qtrees (netapp's word for directories under the root of a volume)
- Update /etc/exports and export the newly created qtrees
- vm.pm manages vmware virtual machines using vm-builder
- Add new virtual machines specifying the hostname and memory
Here is a few basic workflows assuming that the provision.conf is configured and your ssh keys are on all servers:
"provision <fqdn>" Will provision <fqdn> using the default plugins (dns and filer)
"provision -a vm --vrt <hosting_system> <fqdn>" Will provision <fqdn> as a new virtual machine on <hosting_system>
The configuration file for each plugin is templatable for maximum flexibility. Writing new plugins is easy.
Phil Dibowitz wrote all of provision while at Ticketmaster before moving on to google. I wrote the
vm-builder scripts to go along with it. Everything is available under the GPLv3.
http://code.google.com/p/sysprovision/
