Cross-server deployment with servermattic
About a week ago I did a post on Tiny Projects Inspired by WordPress. Readers who actually visited the Code.WordPress Trac would have noticed a tool called servermattic, which is described modestly as “install files and applications to many servers according to their role“.
servermattic is a template configuration that allows for deployment of code and configurations across multiple servers – write once, deploy on many machines, update as much as you want with revisions.
servermattic is a template, a skeleton that has to be configured accordingly and then made to setup WordPress on a server, for example. If you landed here looking for a readily available tool for deployment of WordPress, I suggest this excellent two-part tutorial on Deploying WordPress with Capistrano.
The source code can be browsed online here. Prerequisites include any Debian-like distribution of Linux (you can make it work on other distributions with minimum reconfiguration) and the patience to dig, learn and have fun. Don’t expect servermattic to just work out of the box, it won’t.
Initialize a private and password-protected repository with the files from svn export https://code.svn.wordpress.org/servermattic/ servermattic
. This repository should be reachable by the servers that will be under control.
bin/deploy.sh
This is the only file that is ultimately shipped over to a server that is to be taken under servermattic’s control. Its task is to setup an environment suitable for applying different roles (sets of files and instructions) and setting up the necessary micro-framework for streamlined deployment control.
SVNUSER
, SVNPASS
, SVNURL
are set to the repository that will host all the bases, tags and migrations; almost certainly this is set to the one that was initialized in the first place.
This file is, at this point, probably ready to be shipped off to the new server. It will copy and launch the role.sh
script with a init
argument. This is the next file in the configuration and deployment chain.
bin/role.sh init
Again SVNUSER
, SVNPASS
, SVNURL
should be setup correctly, since the role initialization will be pulling in more bits and pieces. The init
function brings in the data from inside migrations and etc into their respective directories on the server, and creates a roles directory to store data that will be linked in the filesystem.
bin/role.sh apply base
This is how the base role
is applied. The server is identified via the etc/servers.dat file which should be taken care of as well. Identification is done by first getting the primary IP of the machine, and matching it against a line in the etc/servers.dat (this file is already on the machine from the init
step above).
The code chunk gets executed next bin/role.sh from line 85 does not make too much sense to me, and doesn’t appear to work either; so I’ve moved some bits around.
if [ "$id_meta" = "" ]; then echo "this server has no metadata!" exit 0 else # ... go on with the ID message fi
The chunk that actually applied the detected roles for a server, which is the 5th column needs to probably be moved until after a specific role is applied, like base
. So the servers in etc/servers.dat have to have an additional column along with the rest.
# (wan_ip):(dc):(host_name):(lan_ip):(role) # Server A 1.2.3.4:SOLO:one.domain.com:127.0.0.1:wordpress
So why apply base
when deploying? Roles are there for a reason, so the initial code that loops over the server roles should probably be executed during deployment. I’ve not moved the loop, since identification is done during role application, so instead I added a virtual role called all
that applies all the roles for the matched server.
## Role Work # loop over all server roles if [ "$2" = "all" ]; then id_roles=$(echo $id_meta | cut -d ':' -f 5) if [ ! "$id_roles" = "" ]; then OIFS=$IFS IFS="," for role in $id_roles; do /tmp/root/bin/role.sh apply $role done fi elif [ ! "$2" = "" ] #one specific role then # ...
Don’t forget to change the bin/deploy.sh line to deploy.sh apply all
to trigger the initialization of all roles for the machine (multiple roles are assigned by comma-separating them).
The server would probably become 1.2.3.4:SOLO:one.domain.com:127.0.0.1:base,wordpress
, where base
could be a shared role and wordpress
more specific.
The actual application of a role depends on whether a migration is available for a specific role revision. This is what goes on in relation to the repository structure (the files you’re working with, remember these may/will end up in different directories on the end server depending on how your deploy script works):
- list all
migrations/{role}/????.sh
files - detect script name, and 4-digit revision number, skip if current revision or role is larger or equal to the current one (this allows for same revision updates that can be performed as part of the current revision)
- the
$SVNURL/tags/{role}/{revision}/
directory is cloned or updated/switched to viasvn
to the main/roles/
directory on the server linkprop.sh
is run
bin/linkprop.sh
linkprop
is invoked with a role name. For each and every file that the role directory on the server contains (after having been created/updated) the script creates a symbolic link in the root of the server (/
). So, following the long chain of events, everything that is inside of tags/{role}/{revision}/
is symlinked over the filesystem.
Since I moved the base
role to a less intrusive position, I’ve also ripped out the overriding of all role files by base
, resulting in soemthing similar to this:
for i in $(find /root/roles/$1 | grep -v '/\.svn') do to=$(echo $i | sed s/"\/root\/roles\/$1"/''/1) to_dir=${to%'/'*} if [ ! "$to_dir" = "" ] then mkdir -p $to_dir fi if [ ! -d $i ] then ln -sf "$i" "$to" 2>&1 | grep -v 'are the same file' fi done
The migration script is executed next. Note how the script will have access to the server datacenter, hostname and backend. More argument can be easily added.
recap
Quick recap before moving on.
- tags/{role}/{revision}/* is fetched and symlinked over the filesystem
- migrations/{role}/{revision}.sh is executed
- revisions are read from /etc/roles/{role} file, so this has to be updated inside tags to prevent from old revisions to be applied
base + config + wordpress
Let’s see how WordPress can be deployed on a new server.
base
A base
role would probably entail the installation of a webserver, MySQL and PHP inside of a migration file. SSH keys can be added via the base tags, for example.
# migrations/base/0001.sh apt-get -qqfy install nginx php5 mysql-server mysql-client mysqladmin -u root password "__SeCuR3__$3"
config
A config
role would carry settings files for nginx for example,tags/congif/0001/etc/nginx/httpd.conf, PHP .ini files and MySQL .cnf files. The migration script would restart the services if necessary. The migration script would alter the IPs, hostnames, etc. where necessary, conditioned by the server info.
wordpress
The wordpress
role would carry the necessary directory structure and configuration files to wget
the latest WordPress version, symlink over sites-available/enabled
and further run scripts on the database to setup a new user. WordPress itself can be installed via cURL
without any further interaction. Passwords can be randomly generated and mailed; this all during deployment.
Remember, the latest role always overwrites the actual filesystem links.
Updating
The roles.sh update
script pulls all the new files in for each of the roles, including migration scripts and tags. After an update is complete a roles.sh apply all
can be performed.
Conclusion
servermattic is an awesome tool to play with and learn from, although it’s quite raw and unusable as it is, the concept and architecture is quite fascinating, lightweight and flexible. I got a pre-migration and post-migration script to execute around the filesystem linking step in just a couple of minutes without any major headaches. Also, don’t forget that migration scripts can launch scripts in other languages, like PHP, Perl or Python. Making it work with git
instead of svn
should also be easy. I’ve probably just brushed over the full potential of servermattic, and there’s much more to talk about. Props to Automattic to have made it available and the folks behind it:
Most of the code contributed by Demitrious Kelly (https://apokalyptik.com). Remaining tidbits by Barry Abrahamson (https://barry.wordpress.com).
So, have you ever tried servermattic? What do you think? Have you ever written custom deployment scripts with or without revisions, especially revolving around deployment of full application stacks (like LAMP + WordPress)?
Also, check out CloudSource, a tool that’s based on a heavily modified version of servermattic, with a lot of additional features and commands. Utmost interesting.
Wow, great post Gennady! I played around with these things a while ago, back when I wrote that Capistrano series, but I had never actually tried deploying code to more than one server. Feeling ashamed, so I’ll probably kill some time with VirtualBox over the weekend 🙂
Thanks for sharing all of this!
Thanks for dropping by 🙂 looking forward to reading on how it goes.
What are the differences between servermattic and something like Chef/Puppet?
Simon, I would say that lightweightedness, less steep learning curve, open core to implement the behavior you want directly, the insight into the tiny architecture and the fun you get from fiddling with it even when you have no more than just a couple of servers is what makes servermattic a different game. No features, plugins, setups, cookbooks, manuals and configurations to go through – you write what you need, how you need, and learn from doing so (given that you have enough time to write big configs/migrations/tags). In no way do I say that servermattic is better than Chef, Puppet or Capistrano – they’re giant, powerful systems with huge communities, have all sorts of fancy features and use cases.
I would try reading Chef’s 2000+ files (most of these are configs of course), and almost 200,000 lines of code to fully understand how it works, but I’m afraid it would take months; vs. 2000 lines of simple bash (including the readme and the license) that took a couple of hours to shape, use and understand
Thanks for dropping by, what do you prefer to use Chef, Puppet or something completely different?