(Antarctica Investment Bank is a pseudo name. Original article at InfoQ here)

Unfreezing an Organization

Large Scale Scrum transformation at Antarctica Investment Bank

Ladies and gentlemen: the story you are about to hear is true. Only the names have been changed to protect the innocent. - Dragnet

Background and Context

Look back at your career and remember the last successful project that you were part of. What was the one thing that made it successful?

I have asked the above question to hundreds of people in my own non-scientific study. The answers have always been roughly the same:

• Someone who knew what they wanted and wasn’t afraid of transparency.
• A sense of urgency and purpose.
• A team who trusted each other and sat in a war room (aka UAT room) who knew what they were doing.

What I have never heard, not even once, was a process they used. Which makes perfect sense, because process is not all that material at small scale. Much the same way that architecture is not a major factor if you are designing an application for 6 traders on a desk, you probably don’t need microservices deployed on AWS. Excel probably does the job well.

The real difficulty comes when you scale.

The overall results clearly show that waterfall projects do not scale well, while agile projects scale much better. - 2015 Chaos Report, Standish Group

Scaling a piece of software without forethought or using unsuitable architecture resulted in many of the enterprise systems that most of our companies run on: expensive to maintain, fragile, hard to replace and loaded with decades of technical debt.

Scaling an organization without forethought or using an unsuitable assembly line process, resulted in the organizations that run our companies: expensive, fragile, slow, hard to replace, and loaded with decades of organizational debt.

Rewriting a 20 year old antiquated payments system that processes a bank’s payments or transforming an organization share the same characteristics: it takes much longer than originally promised, it’s incredibly difficult, and most senior leaders will shy away from it.

At the same time, in a world of Uber, Wealthfront, and Fintech incubators, most organizations have no choice. The brave few are doing something about it.

This is a story of a department that attempted to redesign their organization by returning to principles that work, and attempting to replicate it at scale. This story takes place in an unlikely place, the back-office of a large bank.

Pre-Transition

Back office or operations technology is a horizontal function that services multiple product lines of businesses. This group is primarily responsible for the efficient settlement of transactions as well as operational functions such as asset services, cash management, and allocations. Often, technology departments are not co-located with their users and frequently have commoditized developers in “offshore” locations. Strong development, project and program management is required to keep the whole system running.

Organizational agility has slowed down for many reasons, including but not limited to:

• Decades of short-term thinking leading to fragile architecture
• Acquisitions of other companies
• Years of lay-offs due to market pressures

In this environment, the leadership of operations technology at Antarctica Investment Bank made the decision to try to reverse the trend.

As with many large organizations, the original structure of Antarctica Investment Bank loosely mirrored the various states of a waterfall project.

• Technology project/program managers
• Change managers in between business and technology who managed change on behalf of business operations
• Architects in a separate group
• Business analysts in a separate group
• Component teams
• Testers in a “QA” group

Small changes (< 1 month) in existing systems were managed by a separate team that had a specific manager. Many banks call this type of team a ‘Run the Bank’ or ‘Business As Usual’ team. Larger changes was prioritized by a separate change manager that worked in the ‘change’ organization. For these changes, typical cycle time (concept -> production) was roughly 8 months.

Larger changes (6 month - 1 year) were managed by a technology project manager and a representative of the operation’s manager’s interests, commonly called a change manager

Strategic projects (>1 year) had the added complexity of an enterprise wide toll-gate process as well as governance structures that were created to oversee the strategic change.

Our brave group began talking about agile development sometime in 2009, almost a full decade after the Agile Manifesto was signed. A couple of the senior developers suggested we use “scrum” as a way to deliver a specific project. It was quickly shot down by the senior technology managers that couldn’t understand how planning and reporting was done in this model.

A year later, a second attempt was launched when a new Head of Architecture and Strategy was hired. He had delivered using something like Scrum before and he wanted to deliver the new post trade processing platform in the same vein. He was senior enough to make this happen.

The team did many of the Scrum ceremonies, but it was just a superficial transformation of ‘WaterScrumfall’ on top of a traditional organizational design. Individuals were not acting as multi-skilled, multi-learning “team members”. There was no business-side Product Owner, the product was not shippable nor shipped every sprint. No team was an “all competency” cross-functional team that could deliver everything end-to-end, and the people and teams were not truly self-managing.

About a year later, a new senior technology manager was hired. I will refer to him as Mr. Disruptive. He convinced the Head of Securities Technology that moving the entire department to real Scrum was the right next step.

Having worked with him in the past, Mr Disruptive brought in Craig Larman to educate and oversee the transformation. At this point in the transformation, operations thought of this as some technology “thing” but agreed to participate in some of the educational sessions.

Retrospective: A coach and a framework are just guides

The ownership and accountability for the transformation sits with the leadership of the organization. Do not shift accountability to a consultant, framework, or methodology.

Getting to Shu

The transformation began with one location and 50 people. The pattern was:

1. Educate everyone involved
2. Form your own teams & chose a ScrumMaster
3. Select your manager
4. Set up an environment for success
5. Run Sprint 1 like a “soft restaurant start”
6. Kick the tires

Retrospective: Partner with the business first to create the product organization

The agile transformation was seen as a “tech” thing by many in the Business Operations group. A story explaining the “why” was either not communicated well or not heard. Most of the coaching support focused on the feature teams and not enough on the business group from where the Product Owner should come.

I would go so far as to recommend that you do not try to set up a large scale scrum framework if you do not have a capable product organization.

Break the Ice. Educate Everyone

Everyone involved in the transformation attended training on scrum and lean thinking. The impact of the training was palpable. There were those who loved the message, those who hated the message, and those who were just going with the organizational flow.

What people took home were not the superficial rules of Scrum, but thinking tools about organizational designs and management principles along with a profound awareness that the established ways of delivering software were antiquated and simply did not work.

The rules of Scrum took only a small portion of the class. Rather, thinking tools featured most in the class. Predominantly: Lean Thinking, Systems Thinking, & organizational and management theory. Most notably:

• Seeing waste was particularly poignant. While humorous in retrospect, the fact that powerpoint documents were considered waste gave many pause. Learning to see value from waste was a lens that many had not viewed their world through before.
• No separate specialization departments. (e.g. Quality assurance, Architecture, Business analysis, etc).
• While the lean principle of “job safety but not role safety” was repeated, this was a controversial and thought provoking notion.
• The Contract Game which illustrated a typical waterfall project was familiar to everyone in the class. Through this two minute exercise, the game demonstrates the illusion of control and opacity in a typical waterfall project.
• Go see “gemba” (the real place where value is added in the eyes of the paying customer)
• Seeing the whole and avoiding local optimization.
• Theory X/Theory Y & the elimination of command and control.
• The “lake and rocks” metaphor. The metaphor illustrates that lowering the water level (short cycles that are potentially shippable, lower work in progress) will expose the rocks (systemic issues) that are hidden during long release cycles.

The Scrum framework itself is very simple: a few roles, ceremonies, and artefacts. The implication of a true Scrum transformation is profound and disruptive. The training focused not on the mechanics but how to think.

Retrospective: Do not use the LeSS framework…unless:

The LeSS framework causes deep and profound change in an organization. Some organizations prefer the illusion of change rather than real meaningful changes.

Large scale scrum is a product development framework for creating products with a high degree of variability. If you are a maintenance organization I would not suggest using LeSS. The one notable exception is if you have systemic key man risk issues.

Shatter the ice. Form your own teams & pick your own ScrumMaster & manager

The next step was for the new feature teams to be formed.

The results of the team formation exercise were five scrum teams capable of delivering across all components. The head of the department created a virtual management team that consisted of team members.

The key point was to support self-organizing teams, where the people created their own teams rather than being assigned to a team by a manager. In short, treating people like adults and trusting them.

The key take away was that this was a signal to the employees and the organization that the teams would be self-managing. It was the first of two major steps taken to move from a Theory X to Theory Y organization.

What better way to support Theory Y and self-managing than to allow the teams to self form?

Allowing teams to self form is a huge leap of faith for a traditional Theory X organization. My advice is to have “management” do a feasibility study ahead of the team formation ceremony. This will allow them to see that cross-functional teams can indeed be formed and if the teams fail to form appropriately they will have this analysis in their “back pocket”. Having conducted this ceremony many times, this “pre-work” is rarely used. As an added benefit, I have had management reconcile this list with the teams that were self-designed. Invariably the teams that are self-selected are better than those that were pre-selected.

Some of the skills that are overlooked in the formation of the teams are skills that project managers and former managers have in abundance: communication, presentation skills, coordination skills, domain knowledge, etc. These skills are overlooked and teams should be reminded of these skills during team formation. Skills such as these are critical to success in large-scale development within an investment bank. These skills can also be found in what is sometimes called the “change group”. This is why it’s important that former members of the management and change groups are in the team formation exercise. Furthermore, if the change group is not present during the team formation, they will most likely create a layer of indirection between the Product Owner and the teams.

Retrospective: Leadership, like Test Driven Development, is a skill that needs to be coached and learned

A leadership gap emerged as managers were relegated and feared being labeled ‘command and control’. Managers have many skills that are required for successful delivery of an item (e.g. communication, coordination, soft skills, etc). These skills were not discussed or recognized during the formation of the teams. Teams were now interacting with senior business leaders without the benefit of those skills. The immaturity of many of those skills caused further friction between the larger business organization.

Select your own Manager

What does a line manager mean within the context of Scrum? If a team is self-managing, what is the purpose of a line manager? What happens to the old line managers before the transition to self-managing teams?

These were some of the questions we grappled with early on. The decision was made to allow each feature team member to select their own line manager. This was a controversial move that also introduced some logistical and systemic issues.

Retrospective: Involve HR from the beginning

Invariably in a large scale scrum transformation, you will hit against organization constraints. Many of these involve HR policies and practices (e.g. compensation, role definition, rating systems). I suggest that you involve an HR stakeholder in the transformation discussions.

Breaking the command and control manager stranglehold over people was critical early on as a further signal that teams would be self-managing.

It has been my experience that not all teams take to self-management right away. A coder that has been told what to do for two decades will not become a self-managing team member overnight. Assuming so is to set themselves up for failure.

The managers that people selected were more like mentors, overseeing the education and the growth of that person. Interestingly, the majority of team members selected those more senior in the organization as line managers. Having a teaching/mentoring role allowed those former managers to become leaders in the best sense of the word.

It’s important to note that a well-functioning department still requires management activities (e.g. budgeting, HR functions, location strategy etc.). Thus, a virtual management group comprised of members of different feature teams was created. They would remove a certain percentage of their capacity from each Sprint. This group was made up from people at “gemba.” This was further augmented by a few people with full-time management roles.

Retrospective: Avoid sidelining managers. The management layer is the greatest asset or the largest obstacle.

“Managers in Toyota are critical to the system running and improving, they create the space where change can happen - it’s not the eradication of managers that is necessary, but the recalibration of them; they need to turn themselves into systems thinkers, learners, enablers and teachers - not team members. the workers help and find problems + potential solutions - but it’s the managers that facilitate the change.” - Mr Disruptive

Great ScrumMasters are rare. There is an industry-wide dearth of them. And great ones will get paid more if they are labeled Scrum coaches. This creates a dearth of talented ScrumMasters in the consulting industry and forces organizations to look internally. Of the many that either volunteered or were put forward, few actually possess the wherewithal, gravitas or knowledge to become a master of Scrum. This creates a problem as scrum puts much responsibility in their hands. For areas where I could not find enough scrum masters, my imperfect solution is to hire experienced Scrum coaches and have them span multiple teams.

Managers can make great scrum masters. Given the correct coaching, certain managers can fill the role intended for the scrum master. They have the added benefit of understand the system and having the gravitas to change it.

Preparing for Sprint 1: Setting up an Environment for Success

A line was drawn in the sand for when the first Sprint would be. It was to be three months after the team formation ceremony. There is never going to be a perfect time to start. There is always inflight work. The group needs to select the time that is least bad.

During this transition period, work would continue as normal but additional activities would need to be carried out to set up for success. While failure is a great learning tool, setting the initial transformation for success is important.

We decided to do sufficient preparation before the first Sprint to increase our chances of having a truly completely “done” shippable product increment, and then actually shipping. If we could do that, we felt it would make a powerful and positive statement about the LeSS transformation. Shipping speaks louder than words.

The preparation period allowed for a number of things to happen:

• Finish inflight work
• Setup the physical space (e.g. whiteboards, walls for visual management)
• Relocate team members to sit next to each other
• Educate/coach people on clean coding, specification by example, etc.
• Cross-train so that people started to learn secondary skills
• Setup a fast build system to support continuous integration
• Automate or eliminate manual regression tests
• Hire great talent
• Setup some communities of practice, such as an architecture CoP

Suggestion: Use volunteers for many of the activities that need to happen during this period.

By definition, cross functional teams will work on components that they have never worked on before. This will create risk to the environment. To reduce the risk, it’s critical to create automated tests that will reduce the regression risk. Also, architecture-learning workshops where the existing architecture is decomposed and explained can also further reduce the risk.

Sprint 1: Soft Start

When opening a new restaurant, there’s the practice of a “soft start” in which for a few days or longer the restaurant is privately open only to friends, but not publicly open. Why? To allow a new organization the chance to work out the rough edges before the food critics arrive.

A similar practice may be applicable in some LeSS transformations; it can depend on how open-minded, forgiving, and supportive the brand-new real business-side Product Owner is, and how messy the first Sprint will be. For example, if (1) there’s always been massive mistrust between the business and technology, and (2) the new Product Owner and other business-side people are very suspicious of getting involved in this “new Scrum thing”, and (3) the first Sprint is almost guaranteed to be a train wreck due to extremely new ways of working for all concerned, then one might want to start with a “soft start” to not lose the very limited goodwill and patience of a reticent business-side group.

In contrast, if Sprint 1 is likely to be a train wreck but the business-side people and Product Owner are open to understanding the reasons, and having patience, one may go with a “hard start.”

Because we felt a “soft start” would help avoid scaring off the real business-side Product Owner and the rest of operations, the first (and only the first) Sprint had an experienced development manager act as a “temporary fake Product Owner”. Starting in Sprint 2, the real Product Owner was engaged

During the first Sprint, there was heavy coaching support. The reality is that home grown ScrumMasters will not be masters of Scrum. They are learning the role and learning the subject. The need for outside experienced coaching is vital during the early stages of the transformation.

The Product Backlog was managed by a member of the change organization who served as a proxy product owner who controlled the overall prioritization of multiple requirements areas. The refinement of the items themselves was done by a combination of “subject matter experts” from the change organization as well as actual operations staff from the “line”.

Product Backlog refinement (PBR) was particularly painful in the early transformation phase:

• Too much time was spent on the secondary issue of so-called “correctly formatting” user stories and specification by examples. The big idea of both practices are the conversations they prompt, but that was not properly appreciated. This was later adjusted through the Retrospectives.
• The Product Owner’s time was not utilized wisely. In many cases he did not need to sit through the entire PBR session because he was not a subject expert on the particular item under refinement (though he did correctly know it was a high priority). This was seen as waste of his time.

Sprints were two weeks long and delivered a truly shippable product. However production releases were every 6 weeks due to enterprise release schedules. This was a significant reduction over the previous lead times.

Retrospective: Avoid changing too many behaviors at once.

“The teams are suffering from Post Transformation Stress Disorder”, one scrum master later told me.
Too many new practices were adopted in the beginning. The teams suffered from information overload during the early stages of the transformation: LeSS, Scrum, TDD, ATDD, XP, unfamiliar components, etc. Coaches should be wary of this and limit the new practices introduced to teams.

Similar to a technical backlog which captures the technical debt that has grown over years, organizational debt also accumulates over years. Consider creating a behaviour backlog and target certain behaviours at a time.

Six months after we started, considerable progress was made: co-located, cross function teams were created, the traditional management hierarchy had been broken, the testing department was dismantled. The organization had been unfrozen.

Retrospective: Avoid premature declaration of success

Post-it notes on the wall, people standing up, and build monitors flashing red do not spell success. Avoid declaring success at this point. I suggest defining near term success with the business/product team.

Was the large scale scrum transformation successful? It really depends on your definition of success and whom you ask. The simple answer is yes and no. More features are being delivered faster, with higher quality, by less people. So in that traditional sense it can be seen as successful.

Organizational debt, like technical debt, that has been built up over decades will take years to correct. You do not rewrite a 30 year old mainframe system in a year. The same applies for transforming an organization.

One such example was the fractured relationship between the “business” and “technology.” These issues were not magically solved by the move to Agile. In some ways the relationship was worse. Why? One major reason was the “lake and rocks” dynamic that happens when a group introduces Scrum or lean elements. Existing weaknesses, including weaknesses in interactions and culture, are painfully exposed and provoked.

With regards to the journey, the organization had reached the equivalent of “base camp” at the foot of Mt. Everest. Reaching base camp is an achievement in its own right and in doing so the organization had shown considerable maturity and courage, but the journey is far from over.

Published material:

• 2016 - Large-Scale Scrum at Antarctica Investment Bank - Ahmad Fahmy