Scrum Case

By Luciano Osorio – Scrum Master at Zappts

Free Translation and Adaptation of the Original Text Scrum Papers (pages 45 to 47), written by Jeff Sutherland, Ph.D.

In 2005, Jeff Sutherland (one of the creators of the Scrum Methodology) worked with Peter Deemer at Yahoo!. They wanted to present Scrum to their managers.

After management decided to adopt Scrum, a productivity analysis of its implementation at IDX Systems (now GE Healthcare) was used to calculate an annual ROI (return on investment) of 1000% over a period of three years of Scrum usage at Yahoo!.

After two years of implementation across more than 100 teams, Yahoo!’s return on investment for each internal Scrum trainer was US$1.4 based on training 10 teams per year. This equates to approximately 1000% ROI.

Teams trained by a Scrum trainer achieved 3 to 4 times the productivity gains of untrained teams.

Scrum was adopted as a Digital Transformation tool at both Yahoo! and GE Healthcare.

ROI Increase with Scrum

The internal rate of return on Scrum training investment is quite high. When averaging across all their teams, many companies doubled their software production rate.

Recently, a CMMI Level 5 company cut costs of software projects in half. Additionally, there was a 40% reduction in measured defects. Even with cost cuts, the company maintained CMMI Level 5 compliance for all projects.

Even the best companies show radical performance improvements when they start using Scrum and achieve much more than 1000% return on Scrum training investment.

This article addresses the return on investment in Scrum training for large companies. These companies have thousands of employees and hundreds or thousands of developers.

These companies established, over many years, heavy, bureaucratic, and flawed processes.

Although it may seem easy to generate substantial gains merely by eliminating the most obvious sources of inefficiency, introducing a radically new process across the entire organization can be slow and painful.

Scrum has a continuous and systematic process of quality improvement. This process identifies and prioritizes impediments to operations in the company’s progress.

IDX Systems (now GE Healthcare): Scaling Scrum for the First Time

During the summer of 1996, IDX Systems hired Jeff Sutherland as vice president of engineering and product development. IDX had more than 4,000 clients and was one of the largest health software companies in the US. There were hundreds of developers working on dozens of products.

This was an opportunity to extend Scrum to large-scale development.

The approach at IDX was to organize the entire development group into an interconnected set of Scrums.

Although this was the first large development team to attempt this approach, the strategy had already been executed many times and documented by Ken Schwaber in “Scrum in the Enterprise”.

Every part of the organization was team-based, including the management team. It had two vice presidents, a senior architect, and several directors. Front-line Scrums met daily.

A Scrum of Scrums, which included the leaders of each Scrum team in a product line, met weekly. The management Scrum met monthly.

The main lesson at IDX was that Scrum adapts to any size.

With dozens of teams in operation, the hardest problem was ensuring the quality of the Scrum process in each team. The entire organization had to learn Scrum at the same time.

IDX was large enough to bring in productivity specialists to monitor each project’s performance.

Scalability Metrics

While most teams managed to double the number of Function Points per month compared to the industry average, some teams entered a hyperproductive state. They produced four or five times more deliverable functionality than the industry average. These teams became stars in the organization and examples to follow.

One of the most productive teams at IDX was the Web Framework team. It created a front end web infrastructure for all products. The infrastructure was designed to host all IDX applications, as well as interoperate seamlessly with end-user or third-party applications.

The Web Framework was created by a distributed team, with developers in Boston, Seattle, and Vermont, who met daily via video conference.

The geographical transparency of this model produced distributed teams with performance as high as co-located teams. This became the most striking characteristic of hyperproductive distributed/outsourced Scrums at companies Xebia in the Netherlands and India, and Exigen Services in the United States and Russia.

The software quality of many hyperproductive Scrum teams can be extraordinarily high.

The IDX Web Framework was first implemented in 1997. In 2007, it was selected as the primary web technology for GE Healthcare systems.

However, very few IDX software teams achieved the hyperproductive state.

On average, based on the Function Points analysis by Capers Jones’ company, Software Productivity Research, IDX only achieved an average productivity gain of 240%. The main reason was the productivity decline caused by excessive Scrum team sizes, with up to 15 people.

Today it is common knowledge that large teams cause significant productivity loss and prevent linear scalability.

Linear scalability is one of the main characteristics of well-executed Scrum implementations.

Productivity Gains Summary at IDX

IDX’s development organization budget was nearly US$ 50 million per year. This was enough for a detailed analysis of baseline productivity before Scrum and the productivity gains generated by it.

An independent consulting firm was hired to perform Function Points analysis on all IDX softwares. This company was called Software Productivity Research (SPR). Jeff Sutherland had worked with Capers Jones, the founder of SPR, during the creation of Scrum and wanted to compare Scrum’s projected productivity goals with its actual performance.

Some IDX products had more than 12,000 Function Points. This was equivalent to more than a million lines of code in Java or C#. Smaller products had around 5,000 to 6,000 Function Points.

The applications were financial and clinical products for operating hospitals and independent medical groups at thousands of sites.

Implementation platforms ranged from Mumps to Cobol and Java, to the latest Microsoft tools and languages available.

Function Points and Story Points

Function Points were chosen as an industry-standard measure, independent of software development language and environment. This was done to ensure a realistic comparison of productivity across technologies and development teams. There was also assurance of comparability with external industry data.

External specialists were hired to calculate Function Points, in order to provide qualified data for research.

As a rule, Function Points are not easily calculated by the development team and are not recommended as an operational strategy.

Story Points emerged as the industry best practice for measuring Agile Development team velocity. They are easily calculated and useful for software release planning.

However, they are not comparable between teams or between companies. Therefore, they were not suitable for research data on the first Scrum implementation in a large company.

In each release of each product, the number of Function Points was recalculated to reflect new features through Software Productivity Research consultations.

The increase in Function Points was divided by person-months for overloaded development teams. This includes design, development, testing, administrative, and management teams.

The initial velocity of all development teams was the industry average, at 2 to 3 Function Points per team per month.

Some teams accelerated until they reached hyperproductivity using Scrum, achieving 5 to 10 times the industry average performance. These teams were about 10% of the organization and achieved an average productivity increase of 666%.

A small number of teams (less than 5%) experienced failures due to personal or technical reasons.

Team Gains

Occasionally, a team was unable to work together effectively and was reorganized or disbanded, usually during the first Sprint.

Less frequently, a high-performing team had taken a calculated risk on new technologies (always approved by management) and the technical failure of some Sprints was anticipated (even encouraged) to gain technological leadership in the market.

There were no failures in major projects, only failures in software delivery in isolated Sprints or a short series of Sprints. In the case of team failures, teams were always rebuilt. In the case of technical failures, efforts were redoubled in subsequent Sprints to overcome research and development challenges.

The remaining 85% of teams achieved an average velocity of 5 to 6 Function Points per team per month, with an average gain of 100%. The net productivity gain of all teams combined was 240%.

This was seen as a failure to achieve performance comparable to Toyota’s level. However, it was a good start for implementing Scrum across the entire company. Recently, comparable-sized development teams have consistently achieved more than 10 Times the Industry Average Velocity.

All Teams in an Entire Organization Were Hyperproductive, achieving Scrum’s original goal.

Conclusion

It is important to emphasize that these productivity gains were achieved at a sustainable pace. There was greater employee retention and greater capacity to hire the best people. This is due to the high-quality work environment provided to developers through the use of Scrum.

Hyperproductive teams were always the most enthusiastic teams, who loved their jobs and worked with great synergy, like a professional sports team.

Hyperproductivity is not achieved by working harder, but by working more effectively through intense communication, mutual support, and an “effortless” ability that makes difficult things seem easy.

Think of Michael Jordan going up for a basketball “shot.” The team worked to make the game conducive to the “shot,” which often makes it appear so smooth and easy that it brings joy to both players and spectators.