TelecomAustralia
(TelecomAustralia is a fictious name as the company won’t approve a case study written from a vendor perspective)
Outsourcing with LeSS at TelecomAustralia
Introduction and Context
In 2012, a large telecom provider in Australia embarked on a journey to build a next-generation telecom billing and ordering capabilities.
The complete development was outsourced to three vendors.
As a key member working for one of the vendors, I could see this was big. Based on my past experiences implementing large-scale Agile transformations (for example, at Valtech India with LeSS co-creator Craig Larman), and since this was greenfield product development, I had the advantage and freedom to recommend an organization based on LeSS. Being quite large, the LeSS Huge framework was used, and the development at its peak was structured into six Requirement Areas.
The client provided one overall Product Owner and several Area Product Owners (all who focused on prioritization and solutions), and one Program Manager who handled all the commercial and legal aspects. All other responsibilities (development, staffing, etc.) were handled by the vendors.
The technologies used were predominantly off-the-shelf products such as Siebel CRM and a commercial billing system. There was Java use, and integration with banking vendors using APIs and web services. Some Requirement Areas used predominantly technology X, some used technology Y, and so forth.
We worked in a common three-week Sprint.
Group Structure, Positions, People
As per LeSS Huge there was one overall Product Owner driving the broad vision and an Area Product Owner for each (at its peak) of six Requirement Areas (RAs). Each RA had only two Feature Teams, working closely with their Area Product Owner. LeSS Huge groups typically have 4+ teams within each RA, however, in the context of this product development, the multiple vendors, budget constraints, and scarcity of skills all played a role in the decision to have only two teams per RA. If I could do it again, I would have wanted more teams in each, perhaps with fewer but bigger RAs.
There was one Scrum Master for each of the RAs.
So in total there was about 100 people involved.
Physically, an Area Product Owner and their two teams for that RA were co-located in one big room.
Scrum (and so Large-Scale Scrum) does not allow any single-specialist role such as being only a business analyst or only a tester. Unfortunately, in this context, there was a heavy reliance on vendors to provide the people with necessary skills, and they did not join the group being multi-skilled or with that mindset. So the teams started out as teams of single-specialists (an analyst, a tester, …). Over the course of the development some of the people grew some secondary skills to become more “T-shaped”, but single specialization remained an ongoing bottleneck and fragility. Especially because all the temporary vendor specialists did not have the long-term context and motivation to become multi-skilled, as this was a “vendor project” context with an end date after which they would not be involved in the product any more.
Inter-Team Coordination
For general coordination between teams, a multisite Scrum-of-Scrums meeting was held daily, using video conferencing equipment. It was attended by a few representatives from each team.
I would very actively encourage frequent communication (mostly, talking) during the initial days of the project.
Over a period of time, as the group got to know each other, the coordination and communication improved and the Melbourne teams also added a weekly informal meeting at a cafe to catch up and sort out issues.
There was one overall architect, and a related team representative from each Requirement Area, who formed a community of practice for architecture. They met regularly to make design decisions, align, and raise and resolve technical issues. The team representatives would communicate back decisions to the teams of their Requirement Area. This community also attended the Scrum of Scrums for daily coordination.
As with many outsourced vendor projects, the client expected “more or less” a certain pace of delivery each Sprint, and the vendors would be held accountable for “excessive” violations of delivery expectations. Therefore, every Tuesday and Thursday a meeting was held to review and adapt to the risks for the delivery that Sprint. A Wiki was also used to record and coordinate about risks and their handling. Anyone could (and did) use it.
Sprint Planning
As per LeSS Huge, Sprint Planning was done separately for each Requirement Area.
The Product Owner Team (the Area Product Owners and one overall Product Owner) met each Sprint for Product Backlog refinement, focusing on prioritization. In this way, they aligned on what each Requirement Area would be offered during Sprint Planning of the next Sprint.
Sprint Planning used to take nearly 8 hours of effort during the initial days of the program. The total effort reduced as the teams got a good grip on the domain, and built a good collaborative partnership with the rest of the group and their Area Product Owner.
Due to the multiple Requirement Areas and two sites with different time zones, total completion of Sprint Planning (across all Areas) was flexibly relaxed to only necessarily be completed by the end of the 2nd day of the one common Sprint, Melbourne time.
Product Backlog Refinement
Each Sprint, the Product Owner Team used to meet regularly in a product-wide Product Backlog refinement (PBR) meeting to discuss priorities within each Requirement Area.
Also, within each Requirement Area (of two teams), a single multi-team PBR meeting of people from all the teams along with the Area Product Owner was held; it would start mid-Sprint. This focus of this PBR meeting was more in-depth clarification of items by the teams and the Area Product Owner, rather than prioritization. Sometimes, a novel issue would be discovered and then part of the PBR meeting would be used to conduct an exploratory technical spike by some of the participants.
There was an agreement to have Product Backlog Items in a “ready” state for at least two Sprints forward. A common challenge I have encountered while working on large-scale product developments are the communication and coordination issues, with new solutions popping up in the middle of the Sprint, and with dependency issues. These issues can create a lot of anxiety if not handled well. The mid-sprint PBR acted as a good mitigation strategy by providing visibility and coordination opportunities for two-Sprint-forward Product Backlog Items.
One of the best thing I liked in this development was the overall PO’s creativity in articulating the theme for the development. He used to create a fictional character, a persona, and build a persona-story around the features we are developing. He used to have a new persona-story for every Sprint and release. For example, if the theme we are developing is around changing the billing format, he would create a story around John who has changed his address then calls the customer care number for a new invoice.
Sprint Review
The Sprint Review used to happen at the end of each Sprint in a big hall facilitated by one of the team members in a “show and tell” format.
The Review used to start with overall Product owner sharing the vision of the product and then handing over to one of the Requirement Area teams to continue.
Representative from each of the areas used to step forward and showcase their work. The facilitator used to rotate between an onshore or offshore team member on a voluntary basis.
Teams were supposed to enact the Product Owner’s previously defined persona-stories and at the same time show the features being developed. This made the Sprint Reviews not only fun-filled but also interesting for everyone.
Computers in Melbourne and Chennai were connected to a projector so that every one in the room could see the same screen. The desktop screen was shared as well. For audio, teams used to dial-in to a conference number.
Stakeholders had the opportunity to ask any questions at the end of this meeting.
Retrospective
As per LeSS, in addition to team-level Retrospectives, then there was an overall Retrospective with representatives from each of the teams. Video conferencing tools were used.
From the overall Retrospective, action items were added to a Wiki. Team members used to volunteer to do items, and were tracked through Wiki. This tracking not only provided visibility about the progress but also built confidence in the team and stakeholders about the improvements/learnings.
Definition of Done
There was a simple overall definition of done (DoD). But because of the different technologies in different Requirement Areas (Siebel, Java, …) each of the Areas had their own DoD relevant to that technology. For example, some commercial technologies don’t have good test automation support and so the Area DoD included manual testing. But others (such as Java) do, and so included automated acceptance testing.
What Worked Well
- LeSS enabled excellent transparency of the work at all levels
- Fantastic support from the senior leadership of the telecom company to implement LeSS and Agile ideas
- A highly skilled group and good inter-team coordination and bonding
- LeSS enabled the teams to work in a highly coordinated fashion to refine the features for the next two Sprints
- The simple flexibility and the principled approach provided by LeSS enabled the teams to tailor and evolve the practices and processes to suit the context.
- Every large-scale development goes through ups and downs and it is rather unpredictable and highly contextual. Trying to control and define such an unpredictable environment with a set of fully-defined processes and practices is a futile exercise. A “low-definition” simple principle-based approach like LeSS and allowing the teams to evolve and customize the practices is the secret recipe to tackle uncertain environments, especially at scale.
Challenges
The client’s budgeting policies caused a few hiccups and intense negotiations in feature selection and prioritization. For example, each of the Area Product Owners had a set funding for the financial year. Many a time, the solutions and the new ideas that emerged during the development used to impact the originally planned effort and in turn increased budget. This change used to raise eyebrows at the senior leadership level, which in turn needed approvals to proceed further.
The vendors had previously deep-rooted waterfall-style processes of integrating code at the end of a long release. Trying to change the mindset to continuously integrate was a major challenge in the beginning. Support from one good leader at the time made a lot of positive difference.
As mentioned earlier, an ongoing problem was the single-specialists coming from the vendors, who tended to be fragile bottlenecks and lone heroes. Getting them to evolve into multi-skilled T-shaped workers and helping each other out as one single team was an ongoing challenge.