Over the past 23 years, I have experienced my fair share of Waterfall projects, both small and large. Over the past eight years or so, I have been involved with a number of Agile-based projects. I understand the critics of Agile and why many Agile teams do not succeed in obtaining their goals. However, I have also seen several companies that have successfully migrated to an Scrum process and benefited from the difficult transition. Below is my take on Agile, Scrum in particular.
Scrum is a form or practice of the Agile methodology for project management. Unlike its name suggests, Agile does not mean a loosely run project with care-free developers and a project manager who could not care less. Agile allows the software development process to change and (hopefully) become more efficient with time. Scrum, with properly planned Sprints, focuses on the productivity and quality of a team. Agile is a very formal process, even though it is often customized to the business implementing it. Agile teams are agile, the Agile process is not. If used properly, it can transform the way your organization manages projects and delivers software over years, not just months.
To the project accountants and executives out there, I have only one question: How many times have you had a project come in close to budget? By close to budget, I mean within 10% of the anticipated cost. You usually sacrifice quality, time, or money (for the most part, it's quality and time) because of the inaccuracies of Waterfall estimating. Being a business owner, several times over, I understand the fear of not being able to estimate project costs. All I can say is that software development is costly, one way or the other. Managing the cost overruns and mental pain of a poorly-run project, in my opinion, is far worse of a situation than admitting that the next six months will require six developers, a Scrum master, a product owner, a testing team, and an good idea of what you would like to accomplish (not necessarily what will be accomplished).
The simple answer is yes, for the most part. This is obviously a very simple answer to a complex question. If you have a development staff of less than three, the answer is probably no. If you have a one-off project that will last a couple of weeks, the answer is probably no. Setting up an Agile infrastructure takes some doing. Waterfall is fairly easy. All you really need are some requirements documents, a timeline, some estimates, and then someone to crack the whip when the documents, timeline, and estimates are compromised. Agile, as I mentioned previously, is not very agile when it comes to project management. It requires discipline from all parties, not just project managers.
Agile, Scrum particularly, allow your teams to hone in on their actual productivity. Unlike Waterfall, where estimates are created in a vacuum, Sprints focus on defining your teams' productivity, even as it changes over time. Estimating effort is a never-ending process. To some organizations, it is part of the game. Being Agile means not only finding ways to be more efficient and productive, but also more accurate. It means that you are focused more on the quality and productivity of your teams, not the cost. Having said that, the cost actually goes down when your team is more concerned about their work than having to report to management on a daily basis, which is eventually what happens with Waterfall projects.
It also gives your staff the tools necessary to stay on top of problems. With daily scrums, the teammates are responsible for their own work and can easily be called out for not producing. Just as Open Source projects breed better coders, Scrum teams police themselves, for the most part. Short, frequent scrums replace long, overdrawn, time-consuming status reports.
Finally, implementations are more accurate and predictable. Smaller and more frequent releases, with follow-up user acceptance testing, ensure that major changes are avoided. If the past 40 years have told us anything, it is that bigger releases mean bigger problems. Big Bang implementations mean major headaches and awful results. If there are problems with an Agile release, it is usually small and fixed in a relatively short amount of time.
It goes without saying; there is no simple project management solution. When you get into even moderately complex projects, dealing with resources, timelines, and cost are difficult. I assume that most people will say that they have or are willing to implement some form of Agile methodology. Many of you have probably had some experience with Agile, including the pain that comes with changing from Waterfall to Agile. To those of you who fall into this category, I feel your pain. I did not really understand the benefits of Agile for a number of years. What I can say, is that Agile is a great attempt at focusing on the real issues of project management, not just the cost. Feel free to contradict me with some real-life experiences.