Friday, 29 June 2012

How I got involved with Configuration Management

I sometimes speak to people publicly for my company, and I tell them I've been involved in Configuration Management for 13 years. Obviously Puppet has not been around for that long, and a lot of people ask me if it was cfEngine or something else I used, well - it was bespoke - and I want to talk a little bit about that.

In fact if someone was to look at my Linked in account, they would realise that the length of my administration experience is roughly that - 13 years. This is because there was never a time where I wasn't using configuration management of some kind.

When I was younger and much more foolish I started work at a company called Microplex. Microplex was a Sydney based ISP and I had managed to work my way into Engineering thanks to the kind promotion of an old friend and mentor Michael O'Reilly. At round that time Microplex was bought by Optus - one of the larger Telco's in Australia.

Michael O'Reilly used to partly own iiNet, one of the bigger ISP's in Australia today. When he started working in engineering at Microplex, he introduced a system for configuration management from the start - it was called 'conform' and it consisted of about 100 lines of Perl code.

My first experiences as an Engineer were what I would later consider to be some of the best in my life. The team was way ahead of their time, and in many cases way ahead of most technical teams even today (and this was 1999).

At the time I had no idea about how to administrate systems, but my senior colleagues were automation mad. In fact we were all called 'Perl Programmers' not administrators. This was DevOps long before the term was invented.

We had 'conform' doing our configuration management, which was awesome. In fact I think I was one of the earlier people to create a fully automated build, at the time it was the 'usrweb' system - you know - the place where you use to upload your personal 5 mb website for Optus/Microplex? That was rebuilt by me using conform - 100% automatically. In 1999 no less.

When I left that group to work in other teams I tried to follow their ways of working but always hit up against the manual error problem, being young and naive I couldn't understand how these people worked at all. Having grown up with pure CM deployment I was shocked to learn that the outside world had no idea nor concept of this. My attempts at introducing other tools always hit up against dusty engineers spouting 'we aren't developer' mantras which drove me mad.

In fact, there was a time where building systems was documented fully in large 200 page engineering manuals. Every configuration file, every Cisco router configuration. I was totally shocked by this way of working and only lasted a very short amount of time in these types of roles as my frustration with my colleagues drove me mad.

Later I tried reimplementing the things I had at my first job at other places. I tried coercing JASS and Jumpstart to do things, but the single build concept was useless in a world of changes. I tried early versions of cfEngine only to find myself doing more work then necessary and living in a sea of badly documented tooling.

I tried writing tools myself and tried forcing them on teams, only to find myself being the only one who maintained them as no one saw the benefit. Yep, my career was depressing on that level - as it seemed everyone was quite happy and job satisfied with making manual changes. Sad really knowing what I knew.

Later I came back to work nearby the team I originally worked with at Optus/Microplex (I moved a bit within Optus) and found they were still using the old tooling, 8 years later. And still succeeding with it. In fact the tooling was so ingrained into their deployment process it was mandatory to use it, and manual changes were not allowed unless under incident. Brilliant :-).

Later when I left Optus to come to England for fame and opportunity I found myself working for Alfresco. Alfresco at the time I had very little systems administration, as there infrastructure was small at that time. I immediately wanted to implement a tool, and learning the lesson from having written my own tools in the past I went looking for an Open Source tool.

And this is how I got into Puppet, I tried all the popular tools - and Puppet started working for me quickly and I understood it the best. The resource model and granular ordering was something I had petitioned for at Optus for conform, but the change was too big and hard to make. So seeing that a tool was already a generation or two ahead of our old thinking and got right into it, and couldn't stop.

I went on to automate almost 80% of what we did at Alfresco, and I believe they still use the tool.

I had a stint at another company, and did the same thing - automated the hell out of it.

Then James Turnbull from Puppetlabs reached out to me to work for them - I couldn't say no. Now I'm a developer I'm as happy as larry, as this is my favourite problem domain to work on. I'm also surrounded by extremely smart people who are passionate about this problem domain as well. What more could I want?

So that's it, I've always been into configuration management. There hasn't been a time really when I wasn't or wasn't trying to get it into a company (and being ignored mostly). So if I struggle to answer your questions when you ask me why Configuration Management - its because I don't really understand the world without it. Its like a world without a modern Linux operating system, or a world without Open Source ... I just can't comprehend the world without it.

And - when people say to me 'automation is too hard, it will take us years to automate' remember this: Microplex was doing it in 1999, without any tools on the market to help them besides Perl. If your not doing it yet, change your attitude fast, and step into it - ignorance or fear shouldn't be an excuse any more.

1 comment: