Ericsson
Petri HaapioEricsson’s M-MGw LeSS adoption
Ericsson’s Media Gateway for Mobile Networks (M-MGw) is a market leader product deployed globally in almost 400 networks. MMGw’s job is to transcode digital media streams between different telecommunications networks.
M-MGw product has been developed since year 2000 by 350 people in Finland and Hungary.
Background
Traditionally the telecom business has been slow, characterised by standardisation and regulations. The product development has followed this business model with heavy upfront planning and big multi-year projects.
M-MGw’s product development was arranged by the areas of development expertise and functions. R&D had component teams and followed a waterfall development process. Only a few people in product management had good understanding of the entire product. The result was organisational silos with multiple handover related challenges. The lead times and feedback loops within the product development were long.
Increasing competition required Ericsson to become faster, shorten the release cycles and continuously improve their product. It was decided to improve flexibility both within Product management and R&D, and adopt Agile ways of working.
The approach
After the M-MGw’s management read the book Scaling Lean & Agile Development: Thinking & Organizational Tools for Large-Scale Scrum by Craig Larman and Bas Vodde, they were convinced to try an Agile approach and Large-Scale Scrum (LeSS). They decided to use external coaches in the change and selected Reaktor, some of whom had previous experience with LeSS. The plan was to coach the whole organization on all levels at the same time. For this four coaches joined the customer organization. Whole staff were to be trained in dedicated Scrum and LeSS trainings during the next 12 months, before they moved in.
After a couple of meetings with the management and key people, we had a consensus on the approach. The vision was a real Scrum-like organization with one Product Owner, Product Backlog, multiple Scrum Teams with their Scrum Masters. The existing line and project managers were to be transformed into coaches, Scrum Masters or team members.
However, instead of top-down directives, an inside-out practical change was put in motion. The transformation itself had to be iterative and incremental enabling organisational learning all the time.
The idea was to set up a volunteer cross functional team of known professionals, to teach them Scrum and required software development craftsmanship practices, and to make sure that they really try out the new ways of working before a possible scale up. The office layout was to be changed so that each team had their own co-working space.
We created a transformation group involving the management and other people from different parts of the organisation. This group gathered together once in a month to inspect and adapt the whole transformation.
The product management decided to give up from fixed scope releases by decoupling feature decisions from releases. Features were to be implemented in business priority and committed to releases when sufficiently certain.
Action
Few weeks after the initial meeting the actual transformation started with the pilot team.
The first team kickoff was a half day session where a bunch of professionals walked in and from which a team would walk out. It turned out to be an essential part of the psychological part of the transformation. How else would one throw away most of what he has been taught about doing professional software development, how else would people who have learned to communicate through documents and meetings suddenly replace it with just by sitting together? How could a bunch of strangers suddenly work in a close proximity and a tight schedule without some kind of agreement how to work together? This certainly would not happen by itself.
Every Friday afternoon one of the Pilot Team members volunteered to share information about the team’s progress to anyone interested, who were not part of the transformation yet. This was an easy and honest way to get to know what Agile and Scrum really was at M-MGw. People could get rid of their fears and prejudices early on.
The next few months most of the transformation happened inside the pilot team. They really knew how to self-organize and be a real team.
Rigging up for scaling
After few months two more volunteer Scrum teams were started. The focus was in cross-team collaboration. Before scaling up it was important to see how teams could and would work on same code base with the official tool set. Because of this goal the new teams worked on the same feature as the pilot team.
It turned out that the tool set did not support cross-team work. The version control system was so slow and merges so difficult that commits were made too seldom. Code tools did not support refactoring which made the code base difficult to understand. Worst of all the technology basically made unit testing really hard thus forcing the developers to trust on testers to validate that their code did what it was supposed to. All this led to weeks long development cycle.
As these tools came given from the Ericsson tool organization it was really hard to get the teams a permission to try out new tools. In the end it took almost a year before other tools were tried out. This was probably the single most slowing factor in the whole transformation (fortunately the new tools seemed to make efficient cross-team development both easier and faster).
Even with the tool difficulties slowly over many sprints the teams learned how to work together. It was time to start scaling up.
Scaling to LeSS
At the same time as the first few teams were piloting the new ways of working the most of the organization was still working in component teams. As the signals coming from the pilot teams were positive the other people were eager to also get aboard. As the current release was a few months away and unnecessary risks were to be avoided it was decided that scaling up would happen at the start next release. To allow people to get started with the new ways of working it was decided that the teams would be formed right away even if most of them would only start months later.
The teams were formed with something that would be known as The Rainbow Excercise. In LeSS this is a self-designing teams session. Everyone not yet in a team were called in a big room. On the walls were templates. Each template consisted of list of wished roles in a team, for example “1 developer with strong knowledge of component A, 2 with component B, 2 testers” and so forth. People were presented the templates and given the opportunity to discuss about them with the Product Owner and the management. Finally everyone were asked to find themselves a team. Coaches and the team members of the existing pilot teams circulated the newly forming teams and consulted them with any further questions regarding how the teams should be formed. Still in the end the teams fully decided their composition.
After only a hour or so there was fifteen new cross functional teams. Later only two of these had to be changed!
During the last two months of the ongoing release the new teams sat together in the agile team space. This had both the advantage of the people to get accustomed of each other and the team space and the disadvantage of having to sit with people who you do not directly work with and in a space that does not serve well individual work. Later it was concluded that it would probably have been better to work in the component teams and in the old office setup until the new teams really started working in the new way.
Despite a particularly challenging technological environment, Ericsson M-MGw managed to bring about a radical change in their process and thinking. The most important factor surely was a particularly constructive company culture, but the external coaches certainly helped them make the journey with less detours and pain.