A Magician pulls a rabbit out of his top hat. Can you repeat his trick? Yes you can, but only if you don’t believe in magic. Then you will focus on understanding what made the trick successful, and soon realize that it was the secret compartment on the table and a hidden opening in the hat placed on top of it. Then you will spend some time building the similar props, and practicing your hand at discretely holding the hat while pulling the rabbit, without destroying everything. And one day, you will be standing in front of the audience, impressing them with your trick.

On the other hand, if you do believe in magic, you will dedicate your time to getting a black cloak adorned by silver stars, flicking your wand and saying “abrakadabra” right. And nothing will happen, a rabbit would not appear out of nowhere because, well, Physics.

Significant number of Agile projects I know of choose similar path. They call their requirements “User Stories” and write them in the “As a <role> I can <capability>, so that <receive benefit>” format. They stand up for their daily meetings. They talk about Guilds, Epics and Release Trains. They do all their Agile Rituals right. Yet they miss the key element of Agile, and their only chance of success is to redefine whatever point they are going to arrive to as “success”.

Introducing Agile Magic - User Stories

Traditional Business Analysts would collect Business, Functional and Non-functional Requirements, some possibly in form of Use Cases. They will talk to a number of Subject Matter Experts, hold a number of meetings, distribute the number of drafts, and eventually produce a document.

Now you get into Agile spirit, so you collect User Stories. You ask the Subject Matter Experts to record User Stories in the format “As a <role> I can <capability>, so that <receive benefit>”. The work is done, you enter your User Stories into Excel, TFS or Jira, and get your Developers to work.

By providing a magic format, you ensured that:

• Your User Stories represent wider understood needs rather than incomplete and unrepresentative views of a particular individual. All other Stakeholders and Subject Matter experts bound to be happy with the software built according to those User Stories, even if they vehemently disagree with the author of your User Stories on everything else.
• You and your Subject Matter Experts have the same understanding of what those User Stories mean.
• You get complete coverage, nothing slipped through the cracks. The User Stories recorded over a couple of brainstorms is all the enterprise needs.
• After you build the software on those Use Stories, it will deliver the business benefits the stakeholders were promised in exchange for funding.

So what a team of experienced Business Analysts could achieve in several months, you achieved over a few hours, all because of the format in which the User Stories are written. Is it possible?

No, it is not possible, it is magic thinking.

If you check the Agile Manifesto written by the people who introduced User Stories, you will find that line: “Working software over comprehensive documentation”. User Stories are not “comprehensive documentation”. They are, as one of the Manifesto’s authors defined it, a “promise of communication”

Your SME or Stakeholder said you a few words that you scribbled on a post-it note, the words you do not fully understand. Your SME or Stakeholder put them ahead of the other post-it notes. Now these words are your task for the Iteration, and you start working with your SME on what that actually means. In 3-4 weeks you produce “working software”, people use it and like it. It is that working software that gets the approval rather than the User Stories. It is produced on the basis of the Developers’ informal communication with the SMEs and Stakeholders. The User Story was barely a conversation starter.

There is no magic. The User Stories are much easier to produce not because of some magic format, but because much less is demanded of them. The “magic” format is actually a very good tool to make it easier to produce those stories at high rate. It also helps to cut out too high-level and too low-level details. Yet it possesses no magic powers.

Is it therefore possible that you find that the User Story represents an opinion of one person everyone else disagrees with? Is it possible that shortly as you start working on a User Story, you find you that the original author meant something entirely different, something that cannot be achieved within the project constraints? Is it possible that the User Story represents existing functionality that the author didn’t understand?

Yes, all of that is possible. Yet the damage from such defect in an Agile project would not be nowhere near as severe than in a conventional project.

Rapid Iteration without Magic

For a person who doesn’t believe in magic, User Stories do not cut it as “comprehensive documentation”. What does play the role of comprehensive documentation in an Agile Project?

The Agile Manifesto have the answer - they prefer working software over comprehensive documentation. Just as a Business Analyst or a traditional waterfall Software Designer would bring a document to a meeting, an Agile development team is supposed to bring working software.

The traditional (Waterfall) Software Development approaches seek to provide all answers and eliminate all risks before coders start coding. The Agile crowd believes, correctly, that:

1. Producing detailed specification is prohibitively expensive, and takes enormous time
2. Necessarily written without understanding of most modern programming techniques, tools and resources, such specifications always request suboptimal solutions.
3. Despite all the effort, such specifications do not eliminate risks and uncertainties.

So the Agile people suggest negotiating over working program rather than over some document that could be misunderstood by the reviewers as well as the Developers.

However, unlike text documents, software code has a lot of interdependencies. Past a certain point, “sorry, we misunderstood you completely, let’s rewrite the whole thing” would become prohibitively expensive and represent unacceptable failure in exercising accountability and risk management at a project and program management level.

Hence the rapid iterations - bring working software to the table, but do it often, before too much work is done on top of the original mistake. If you refuse the [deceiving] safety of comprehensive specifications, you want to minimize possible damage by having the clients verify the next iteration of the working software you produced.

The Paradox of the First Agile Project

The First Agile method was called eXtreme Programming, and their show project was Chrysler payroll system implementation. It is where they piloted all their great ideas, including rapid iterations, User Stories, resident stakeholder etc. Yet there are two revealing facts about that project that avoid attention.

First is that the project got canned. Clearly having a resident stakeholder is not the same as securing support of executives that are not available full-time.

The other fact is that a payroll system for a heavily unionized workplace has a near complete set of Requirements. It is called Enterprise Bargaining Agreements, and they cover in detail everything about pays and entitlements. You do not need User Stories for anything except for details of User Interface, which is the minor part of the effort. For the rest, you need a good paralegal going over the Agreements, working with a good Analyst/Designer (Ideally an Ontologist, but that may be too much to ask) to define adequate abstractions and rules that represents the Agreements.

How Agile is Agile

Say, you implemented a payroll system in a very Agile way. You hold stand up meetings - or even meetings in plank. You had your User Stories, you had your interdisciplinary team.

You delivered the system. Yet the Unions demanded more, the bosses counter-offered, and the Enterprise Agreements changed. They have entitlements that didn’t exist before, some conditions changed, some decisions (like “too hot to work”) that were made by line managers are now made centrally or vice versa. So your payroll system now has to be changed. How easy is it?

Do you have the original developers available? Or there is self-explanatory administration UI? Or the system is documented well, and the is the system architecture and design documentation up to date for the current implementation and easy to read? Or the system is driven by metadata which is easy to understand?

Or none of the above is true, and you need to hire new people who would start researching the code of the system, trying to figure out how it works?

You need an answer to how exactly the business agility is achieved. To claim that Agile development process by itself delivers Agile business is to believe in magic.

Achieving Superior Results with “Freefall” Project Methodology

Please allow me to introduce the Freefall Project Methodology. It is a high-performance approach to delivering IT projects that combines the unstoppable nature of Waterfall with light-weight, dynamic culture of Agile. The key principles of the Freefall methodology are:

1. You use rapid iterations, calling them Sprints. You create a multi-disciplinary team, conduct daily stand-up meetings, and form Guilds.
2. You collect your Requirements in form of User Stories over a couple of brainstorms, then use them to guide and evaluate your progress over the length of the project.
3. You ask everyone who has something to say about your project, like Business, Data & Technology Architects, IT Operations etc, to stay out of the way and not impede your progress
4. The primary indication of success is that the software you are developing works, meaning compiles and doesn’t crash when used by your team.
5. At the end of the iteration, you report the progress to your stakeholder in terms of technical achievement, without providing them with software to investigate, or meaningful figures they can understand.

Implementing a Freefall methodology ensures that the project proceeds with little resistance, interference or control. It ensures 100% success of all iterations, even if you use mediocre analysts and developers. The performance of a Freefall project team is unmatched to any other methodology.

There is only one shortcoming of a Freefall project. Eventually it lands, and the enterprise is left with whatever is usually left after a freefall from significant height.

Agile Process Health Assessment

So is your project based on belief in magic, or on solid and logical foundations? Is it an Agile project, or a Freefall project. Two questions are all that is needed to provide a comprehensive answer.

Question 1: How often your iterations fail, at least partially, to gain stakeholder approval for the outcomes they provided?

If the answer is “never”, your project is in freefall. It is developing on a hunch, hoping that everything will eventually fall into place. The guesses (we call them assumptions and hypotheses) the project team makes are not validated at the end of each iteration.

Question 2: How many stakeholders with failing rights does the project have, and who are they?

The second most common reason an Agile project does not have any issues with the stakeholders is not having enough stakeholders. Say if you don’t treat IT Operations as your stakeholder, you don’t need to prove that your software would be easy to change.

The difference between Software and Traditional Enterprise

That is the distance that is not well understood, and likely deliberately obscured. At least, I never heard an Agile Trainer explaining the challenges of applying a method well-proven, say, in Spotify to say a bank.

A software business never stops developing their product, not until the product is outdated and they no longer need it. If you put together an Agile development team to develop a Database, an Operating System, an internet bookstore or a social network, and you managed to get some traction, you keep developing your product to make it better. The members of the initial team have ample reasons to stick around, as they are promoted to lead more developers to move past the “early adopters” users and capture the mainstream audience. The early code would be changed and expanded, again and again, mainly by the same people, in a bid to make it more stable, more feature-rich, more idiot-proof. Unless their business model fails, they would keep the expertise in-house.

When however a traditional enterprise develops a payroll system, they want to treat it as an appliance and use it as it is for twenty years with necessary changes. Your prized development team would likely be poached by other sites that have exciting greenfield projects. Then the Government change some regulations, and your union negotiate some changes, and you need to modify the system.

How many Stakeholders an Agile Project needs?

That is a question that is easy to answer. Your project needs one Stakeholder for every claim it makes. In some cases, some Stakeholders can wear several hats, but that is rather uncommon.

The primary Stakeholder, the one that is usually referred to, approves UI and the overall workflow of the system. If your project claims that the staff represented by the said Stakeholder would be happy with the system, that is all you need. However most projects makes other claims as well.

You claim that the solution would interoperate with the other system? You need an Integration Architect to see that happening. You claim that the solution would be easier to maintain? Someone has to be there to assess it. You claim alignment with Enterprise Architecture? That is your other Stakeholder. You claim that advanced Analytics would be applied to your data for extra incite? You need another Stakeholder. You claim your system will protect personally identifiable information and be secure from accidental or malicious breaches? Another Stakeholder, specifically qualified to assess the project’s delivery on that promise.

Scaling Agile without Magic

There are vendors who offer you the way to run Agile on a higher scale, for a mega-project or across enterprise. They offer you some magic incantations like “Epics”, “Release Trains” and yes “Guilds”. They demand some license fees and highly expensive training for their magic. So, is it worth it?

No, because there is no such thing as Agile Magic. The whole idea of scaling Agile is easy:

1. You identify products that are needed by the major Stakeholders, with their Agile teams.
2. You identify common parts, or parts that require specialized expertise, make them separate products, and form an Agile team for each. The downstream projects then become Stakeholders for upstream projects.
3. You identify and manage dependencies between projects with a scaled approach to communication, interaction and responding to change across the program of development

Of course, then you need to ensure that there is no political play, that the definitions of projects is based on common sense rather that fight for influence, and that the Agile teams negotiate in good faith and collaborate effectively, but that cannot be helped by any magic either.

What is next?

What should you do if you decide to try Agile without Magic?

First, have a good look at your Agile consultants. If they demand exorbitant fees for the methodology and associated training, they are likely selling you some magic. Well, it is your choice, but this article classifies as Magic any suggestion that particular vocabulary and rituals can change company culture, which in turn would result in people delivering what is required, fast.

You can consider inviting an Agile without Magic Manifesto signatory. An AwM Consultant should:

• Ensure that all component Agile projects have measurable criteria of progress, and practice learning accounting
• Ensure that all promises of individual Projects are represented by Stakeholders
• Ensure that all Stakeholders can effectively validate outcome of an Iteration
• Ensure that effective collaboration happens over the course of an Iteration
• Check for known Magic Anti-patterns, like
• Promises without Stakeholders
• Stakeholders in the dark
• User Stories treated as Specifications
• Delivery in technical terms

Alex Jouravlev, Business Abstraction. Fee free to contact me.  

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.