Blog Layout
The Phoenix Project — (Almost) 10 Years On
Vince King • 14 June 2022
Society, technology, and IT’s relationship with the Business have moved on a great deal since 2013; is the advice held in The Phoenix Project still as powerful as it was when I first picked it up?
I read The Phoenix Project, A Novel About IT DevOps and Helping Your Business Win (by Gene Kim, George Spafford, and Kevin Behr) shortly after its release; almost 10 years ago. Society, technology, and IT’s relationship with the Business have moved on a great deal since 2013 and I wondered if the book was still as powerful as it was when I first picked it up. So, I re-read it.
SPOILERS. This book is almost 10 years old, but I am amazed at how many IT professionals have heard of it, but never read it.
First, a quick recap. Parts Unlimited is an automotive parts manufacturer in severe trouble. Its market share is down, and it is being beaten on all sides by the competition who are outperforming the company in innovation and capability. The board is putting pressure on the CEO, Steve, to fix … well, everything. We are quickly introduced to Bill who is offered a promotion to the role of VP of IT Operations and tasked with, essentially, saving the company. Our reluctant hero accepts the challenge and is immediately dropped into a blizzard of Sev-1 incidents, understaffed teams with a lack of capacity and capability, and project pressures from the Business. Bill’s work/life balance gets pushed aside as he fights battles on all fronts while priorities shift under his feet like quicksand.
When I first read the book, I was a developer and closely identified with the Brent character, and over-worked full stack developer, and the only resource that could do anything. At the time I had been part of a small business and needed to wear many hats, however even after moving to larger organisations and teams the pressure on my time could be overwhelming. I can now see that my desire to help everyone was, well, unhelpful, but I was unprotected from the telephone calls, emails, and walk-up requests that came my way almost hourly. I was suffering from too many work items in-flight with new tasks coming from all directions without any prioritisation. Inevitably the firefighting would start when something got missed and things went wrong.
In The Phoenix Project, Bill is introduced to work pipelines, prioritisation of tasks, proper change management, the importance of limiting WIP (Work In Progress), and most importantly, the concept of constraints. A constraint: the cogs in the metaphorical machine that can’t spin fast enough to keep up. As a “developer-cog” I could relate, and later as a manager of “cogs” I used my experience, working hard to emulated Bill and took responsibility for protecting my people.
While reading the book recently, now with many more years’ experience behind me, I empathised with Patty (the change manager), Chris (the development manager), Wes (the operations lead), and most recently John (the Cyber guy). I could now relate their issues to those that I’d had during my various roles and had a newfound appreciation for their situation.
The processes, techniques, and advice provided by the book’s mentor, Erik, haven’t changed. but the world around us has. The procedures Bill implements within Parts Unlimited aren’t new, and weren’t new 10 years ago, but now they are more widely understood and accepted. There are fewer barriers to adopting all the buzz words, Agile, SCRUM, Lean, WIP, CI/CD, fail-fast, and of course DevOps. However, one issue that I see too often in companies, both large and small, is a lack of understanding of Technology’s role within the organisation, and its importance in achieving business objectives. Throughout the book the IT department is seen as a blocker, and I’d argue that Bill’s biggest achievement wasn’t better code, or faster and more reliable deployments, but his work to get IT seen as an enabler. This was a game-changer. One of the biggest takeaways from The Phoenix Project: without the Business, IT wouldn’t be needed; without IT there’d be no Business.
So, my recommendation is to read the book; it’s easy to read and may help you identify issues in your own situation. However! This is not a silver bullet for all your IT problems. There are no quick fixes and even the story gives a sense of the effort required all levels.
If unauthorised “fixes” are going into Production – look at your change management systems.
If code quality is poor – look a technical training.
If security controls aren’t in place – look at policy enforcement processes.
Whatever your issues, there is a solution, and it will most likely touch on each part of the Technology/People/Process triad. I have rarely seen an issue that could be resolved with only one of these three elements. Investing in tooling will only get you so far, but you must also change the culture around the issue and create clear processes for staff to follow.
My final piece of advice. Start small. Fix one thing. Then fix another. Then another. As confidence grows and your processes mature, you’ll see improvements, but only if you create measures to track your progress and feedback on anything that doesn’t work. Never knowingly push bad practices forward. Kicking a problem down the road will only create technical debt and come back to cause you even more trouble.
I found reading The Phoenix Project again incredibly useful, and even with all my experience and war wounds, there was practical advice that I could use.
Share Blog Post
Cloud Security Posture Management (CSPM) is a popular topic of conversation. Go to any event or expo and you'll see the growing number of solutions, and with organisations moving to the cloud at a staggering rate, this is set to continue. If you need a CSPM … and you do need a CSPM solution if you are in the cloud, not all the tools you see may match your needs. For me, when considering a CSPM solution I have a number of requirements. 47 to be exact. Here, we are going to talk about 3 of the most important. Inventory Simple things first … you can't protect what you can't see, so inventory is hugely important to Cyber Security. Typical on premise, agent-based solutions can only guarantee to report on assets that are able to make contact. However, the tooling could never be held responsible for assets that were poorly configured or failed to register with the platform. Having been faced with the issue of unknown unknowns in the past, I had to create an application to combine data from numerous, agent-based systems to allow me to identify gaps in our coverage. Now, with so many organisations moving to the cloud at speed, these issues will only intensify, especially when you include the potential need for multiple Cloud Service Providers (CSPs). So, it is imperative to know what you have to protect, but if we are using cloud correctly, we start to move away from the traditional VM-based estate. If you ever talk to me about cloud I quickly start ranting about the benefits of the cloud and the new abilities it gives us. No longer are we tied to building and configuring VMs. Applications can now be built using cloud native resources such as serverless functions, blob storage, and other managed services. So, now we must think about the health of these new types of resources, some of which, due to the ephemeral nature of the cloud, may only live for short periods of time. Intelligence Next is intelligence. You cannot use the internet without hearing reports of new vulnerabilities, zero-days, and breaches. Whether it's organised crime, nation states, or script kiddies, threat actors are constantly knocking on our door trying to find the smallest crack in our defences. Attack vectors are constantly evolving and we, as cyber professionals, need to keep up-to-date with the ever-changing landscape. The global pandemic changed many things, and cyber security activities were no exception. Historically time-to-exploit of a new vulnerability was months. More recently I've seen this come down to weeks or even days. Vulnerability data sources are freely available, from NVD, to GCHQ, and even security vendors reporting the findings of their own research. However, collecting, triaging, and interpreting all this information can be very time consuming and overwhelming. But, I can't understate the importance of data. Having this information allows us to identify the issues in our estate by being pro-active in our response and minimising our risks. Insights Finally, insights. Risks without context are meaningless. If you remember only one thing from this presentation … risks without context are meaningless! All CSPs provide the ability to view known vulnerabilities associated with a resource, however this normally highlights issues in isolation. CSPs also allow you to compare your assets against defined policies, but again this evaluation is on a resource-by-resource basis. Knowing that a VM is missing an OS patch and is outside policy is useful, but how serious an issue is it …. Really. What we need is to be able to look at the bigger picture, not quite the 35,000 feet view, but certainly something larger than individual assets. Being able to view issues on assets and see how those assets are related to other resources across our resource group or subscription is hugely important. If we follow basic cloud security we will have applied layers of protection, but the constant stream of new vulnerabilities and exploits means we must continuously evaluate the potential impact of a successful attack. Data sources, again can help, but the advice provided is not tailored to our environment, assets, or configurations. Understanding a vulnerability within the context of our environment can quickly elevate the severity of an issue and demand a response to avoid exploitation. An unpatched server may not raise an alarm, but let us play the "what if" game. A bit of threat modelling. What if the patch addresses an elevation of privilege vulnerability? And what if that server has a public IP address and open ports that expose it to the internet. And what if the server has poor SSH authentication and is connected to other servers within an application. Could a successful attacker start to move laterally through our estate? Without context the possible risk and impact of vulnerabilities may never be realised. Summary So as I mentioned, I actually have 47 requirements for a Cloud Security Posture Management solution, but these are a few of the ones I MUST have. I'd advise you to keep them in mind when considering a tool to monitor you cloud instance.
COVID has really had a devastating effect on a huge number of industries, but let’s spare a few minutes to talk about corporate events; specifically, conferences, expos, and forums. Once a great “work-related” reason to get out of the office for a day or two and nab some swag, the pandemic shut them down with lightning speed. No more wandering around large conference centres, mingling with thousands of strangers; no more catching up with old vendors and being wooed by enthusiastic salespeople; no more cramming into an auditorium for the opening keynote; and no, no more free t-shirts. As a regular attendee to conferences I would sit beforehand figuring out how to get to the venue, which vendors I wanted to talk to, and the sessions I really wanted to see but I could never see them all and some had to be missed. I’d block out the whole day, sometimes two, in my work diary and set my out-of-office to “I have limited access to e-mail”. I’d get to the venue early to register and get my pass, and then shoot through the exhibition hall to get a seat in the keynote address. Looking around at my fellow attendees I’d see some tapping away on work laptops, and others playing with their phones, killing time until kick-off. A couple of sessions later and with an armful of swag, I’d need some over-priced lunch, and hope of a seat. A few more sessions in the afternoon, a bit more swag, and then back home via planes, trains, and automobiles. In March 2020 this all went away in an instant. Conferences were cancelled en masse; organisers scrambled to “go virtual”; and my t-shirt collection stopped growing. All the big names were affected; Cybersecurity Europe; Microsoft’s Ignite; CES; even the Chelsea Flower Show. However, amongst all the cancellations new opportunities appeared. With so many events pivoting to go online and companies offering the chance to watch live or on-demand session after the event, a wide range of talks, webinars, and roundtables became available. With a lot of us working from home, travel to these events was a matter of clicking a button on our already connected computers. No longer did we need travel time, dropping into an online session about GDPR being held by the BCS Law Specialist Group in Manchester was easily achievable. I quickly embraced this new extensive library of events now free from the previously high demands on time and expense. During the first lockdown I attended information sessions by SOPHOS, training sessions by SPLUNK, and a very interesting session on Policing the Cyber World by the BCS’s own Information Security Specialist Group. Previously I would have looked at these events with interest but I’d have felt that it was too much effort, or I was too busy and couldn’t spare the time. However, suddenly taking an hour out of my day to learn something new had never been so convenient. For larger events, I would sit comfortably listening to a session on digital transformation without worrying that I’d missed other session I wanted to attend about DevSecOps; knowing I could catch the recording later. In September 2021, during the short period between lockdowns, I was given the opportunity to speak at a small conference in Central London. The event was planned to be hybrid and I’d be attending in-person. The session was well attended with 50 or so people in the room, however the online attendance was must greater with 200+ watching the live stream. Speaking to the organisers after the event we discussed the difficulty in running online or even hybrid events. These sessions were appreciated by the attendees, but vendors were less interested due to the smaller in-person audience with whom they could engage. We agreed that the future of online sessions could be a YouTube-style, un-skippable advert regularly dropped into events. I’ve always accepted that vendor-specific events will include an element of “show-and-tell” or “big announcement” of a new feature, but conferences rely on footfall to make the exhibition worthwhile. For our internal Security Awareness Month in October the organisers took the decision to make the sessions online only, even though office numbers were on the increase. Again, attendance was up, and feedback was very positive. Speaking with some colleagues we all agreed that during previous in-person events we’d often sign up, but when the time came, we wouldn’t leave our desks to travel the few floors to the session. So, this year, jumping into an online meeting proved much more convenient. So, what does the “new normal” look like for conferences? Personally, I hope we stick with the hybrid model. I still like the in-person conferences; catching up with the vendors I know; meeting new people; attending interesting sessions; and don’t forget the swag. I’ve already registered for a few this year. But I’ll also be on the lookout for virtual events, and on-demand sessions. These have proven too convenient and accessible to be missed. I’ve been given the opportunity to broaden my knowledge from the comfort of my own desk while eating lunch and keeping an eye on my e-mails. So, let’s not lose the advantages that have arisen from such a devastating period and embrace this new way of coming together; and if we must watch non-skippable adverts, it’ll be a small price to pay.