Rise from Ash: Avi Shaked’s notes on systems and products development
Your feedback is valued. Please send it to shavi81 @ gmail.com.
Moving Beyond Agile
Last Updated: October 21, 2017
Agile Software Development has become a notable trend since its introduction in the form of the Agile Manifesto in 2001. The Agile Manifesto provided an umbrella to a variety of software development methods by suggesting a set of twelve principles that these methods need to comply with in order to be considered “Agile.” Agile Software Development has undoubtedly brought process and methods consideration to the table when discussing software development efforts, even in small organizations.
The times they are a-changin’, though. Software is embedded everywhere and in various forms, constantly evolving to drive more and more aspects of our lives and being the heart of services, products and systems. These software based services, products and systems can further compose systems of systems, affect people’s individual and collective behavior and disrupt those of the past. Sticking to the traditional Agile Software Development principles no longer holds value, and can, in fact, be limiting instead of assisting or facilitating.
It is important to understand that the underlying principles of Agile Software Development are merely tailored versions of systems development ones, and certainly not universal ones. As mentioned, Agile made development methods a service by raising awareness to them; but once the talk had been established, and especially due to the extensive, diverse use of software, it is time to raise the bar, and understand that the pallet of development methods is greater than the one delimited by the Agile principles. Organizations that try to practice Agile, typically with only a partial attention to the principles in the first place, might miss out on the true objectives of the development process. In a way, adopting Agile principles blindly (or just because it is popular) is equivalent to a root cause analysis stopping at the first “why?” – there is more left to uncover.
In this article I review the Agile principles, and, while trying to identify the real development principles behind them, demonstrate that they form a limited, at times misleading and inconsistent, manifestation. I also discuss how some of these principles can alienate software development from the overall product, system or service development. Eventually, I propose a new set of overarching principles for systems development consideration, emphasizing that software related aspects might only be a partial factor of the overall development effort and, at the same time, encouraging these aspects to fit with the overall effort, just like software is required to be integrated into a system or a product.
Agile Principles Reviewed
In this section I go over the twelve principles of Agile software development, as published by the Agile Manifesto authors, and demonstrate their deficiencies and inconsistency.
1. “Our highest priority is to satisfy the customer through early and continuous delivery of valuable software”
This principle has several facets, and requires delving into it in order to understand the tailoring it suggests.
First, it puts customer satisfaction above everything (“highest priority”). In reality, though, a software development effort may have different goals, not always involving the customer. It may, for example, be performed as a part of a multidisciplinary development of a novel product (the original iPhone for example), or as a service to improve the software development infrastructure and the operational performance of the organization. A welcomed change should be replacing “customer” with “stakeholders,” as this will reflect a wider, more absolute systems development principle, which also implies that stakeholders of various types (including management, development teams, and regulators) should be identified and considered while planning and performing the software development effort.
Second, “early and continuous delivery of valuable software” is a very specific way – the “how” - of achieving the first clause of the principle (the “what”). It is a method almost unique to pure software development, which ignores the physical limitations of interrelated hardware (in cyber-physical systems development, for example). Moreover, it implies that “continuous delivery” goes along with “valuable,” which is not always the case: if the software was valuable in the first place, then there may not be a need to continuously deliver a new version of it to the customer. In fact, a customer may find it frustrating to receive frequent software updates, and might question the software’s maturity. This remains true even if the customer is being regarded as one of the various types of stakeholders (e.g. continuous delivery may require integration related rework of an integration engineer, who is one of the stakeholders of the software, and this might introduce waste to the process).
This principle can therefore be seen as a rather narrow reading of two basic product development paradigms: prototyping, and incremental development (e.g. product development in blocks). Some prominent goal of these paradigms is to expedite the understanding the problem as well as to prove different levels of feasibility. Once you understand this, Lean’s frontloading principle is certainly a more acceptable principle.
2. “Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.”
What this principle is saying is we “welcome changing requirements,” leaving the details of how this is approached to the specific Agile methods (the “Agile processes”). If we treat this principle as a requirement for the process, then a process needs to accommodate change in order to be considered Agile.
Systems Engineering has long been occupied with the need to accommodate change throughout development, giving birth to fields such as requirements engineering, requirements management and impact analysis. Moreover, it is a well-known systems engineering practice to encourage stakeholders involvement throughout the development (and, as a specific example, through formal reviews that are highly important in achieving customer’s commitment to the end product, and, implicitly, the costs of customer-sponsored change).
This Agile principle, however, implicitly discourages some of the systems engineering ways of accommodating change, like contract-based development (as in build to spec situations), by adding “even late in development.” This addendum does not state how we should welcome the changing requirements in late development stage: it can be achieved by asking the customer to pay for such changes, by redesigning the entire system to confirm with them, or by designing the system to accommodate changes we anticipate in advance, amongst others. Unfortunately, Agile practitioners typically interpret this principle as a waiver from practicing requirements engineering and impact analysis, and as a call to practice shortsighted development (e.g. to focus on Sprint backlog tasks).
3. “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
This is merely a scale-related tailoring of incremental development. In practice, it is more confusing than helpful, as those trying to practice Agile often ask themselves “what is a working software?”, “what is the right timescale for a software development iteration?” and “how much should we compromise the design in order to obtain a working software in time?”
As software becomes more complex, sometimes serving as a scalable infrastructure, the compromise involved in achieving a working software during a single development iteration is becoming more and more apparent to practitioners, who introduce debugging iterations, redesign iterations and infrastructure development iterations to the development process just so that they won’t break the Agile development scheme, instead of tailoring the entire process with a more holistic approach in mind.
4. “Business people and developers must work together daily throughout the project.”
Is it really necessary for business people and developers to work together daily?
It may be required for business people and developers to work close on specific events (a demonstration, a proof of concept, a product launch, a startup pitch and others), but such daily interaction does not represent the general situation.
In fact, I would say a frequent interaction of the developers with the customer (or stakeholders, remember?) is far more important than interaction with business people. Also, daily interaction of developers and business people is a rare thing to happen in large enterprises, in which they typically belong to different divisions, each being focused in its own domain to achieve not only the daily but also the strategic goals of the organization.
5. “Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.”
This principle depicts an explicit expectation from a manager, as well as bears an implicit expectation from the developer (be motivated). As it is, this principle is more of a wishful thinking than it is a process or method guideline.
But it is not my goal to dismiss. Instead, I try to get to the bottom of things. If considered correctly, this principle represents the real need to plan the development effort. This planning should, according to the law stated here, take into account the environment and resources (budget, time, facilities, and motivated people are some of these) required in order to sustain the development effort and to get results (“the job done”).
6. “The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.”
I disagree. While interpersonal communication is of utmost importance (especially in developing a deeper understanding of the problem domain, as Lean Product and Process Development sponsors), a face-to-face conversation may lead to partial requirements and premature decision making, and is therefore not an alternative to proper engineering practices.
It is my experience that better handling of information is via documented communication. Specifically, requirements engineering, including but not limited to a well written requirements specification and well defined scenarios, is a key in communicating ideas.
Also, face-to-face conversation might lead to an agreement between the development team and the stakeholder, but if that agreement is not documented, and in some cases even approved by all relevant stakeholders, this may interfere with other aspects of the product development and lifecycle (e.g. product testing, product certification by regulators, or product maintenance).
7. “Working software is the primary measure of progress.“
This is a software-specific Agile principle, and one that has not aged well. This principle takes away from the importance of planning and designing, and glorifies implementation (coding is an implementation phase task). It is a principles that corresponds with the notorious reputation of software programmers as people who most of all love to code.
If we put aside the fact that a working product has nothing to do with being valuable or attractive, having a working software also does not necessarily reflect true progress. If you take embedded software development, as an example, having a software that works on a developer’s workstation, or in a virtual sandbox, does not mean that there has been a real progress regarding its integration with the hardware platform or the non-functional/performance aspects of the software and product. Furthermore, problems with regards to these may actually require redesign of the software, bringing the development effort several steps back.
8. “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.“
This is a principle which is easy to agree with. It lives in harmony with the longstanding, general principle of involving the multiple stakeholders in and throughout the development process. Admittedly, the Agile related “Sprint Showcase” is a minor novelty, adding another method of performing reviews.
9. “Continuous attention to technical excellence and good design enhances agility.”
This one is surprising. Obviously, technical excellence and good design are the basics of a successful engineering effort. Any serious research of development efforts supports the need to invest in the requirements and design prior to the implementation, as the ability to influence the quality and cost of systems decreases as you move forward in the development lifecycle.
The surprise, therefore, has nothing to do with the content of this principle yet with its consistency with some of the other ones; or rather its inconsistency. Specifically, this principle stands in opposition to principle #7, which encourages implementation over design!
In practice, mind you, principle #7 is the frequent winner, as people tend to credit progress to implementation instead of crediting design artifacts as progress indicators. Systems Engineering, on the other hand, has long been encouraging design as crucial in demonstrating progress. Even on Waterfall-styled efforts, a good development process typically leads the development through 3 design gates (System Requirements Review, Preliminary Design Review and Critical Design Review) and only one implementation gate (System Acceptance Review), putting the emphasis in the right place.
10. “Simplicity - the art of maximizing the amount of work not done - is essential.”
Now, how do you maximize the amount of work not done? One of the major tools in achieving this – as explained above – is to dwell on requirements and design. But reconsider principle #3 and/or principle #7: does supplying a working software every two weeks in a project that last for 2 years maximize the amount of work not done, or does it just maximize the amount of work? There is no definite answer, but it is something to consider before blindly adopting the Agile-related paradigm of continuous integration.
11. “The best architectures, requirements, and designs emerge from self-organizing teams.”
Do you really think pair programming (a technique associated with some Agile flavors) results in “the best architecture, requirements and design”? It might be the case provided that the two individuals exhibit certain behavioral characteristics, individually and collective; and provided that they are highly educated, highly trained and highly experienced. This is a very specific case, and not the common one.
Personally, I can tell you that my requirements, architectures and designs were bettered by the mentoring I had received from my professional managers (whose important role Agile seems to wrongfully disrespect). Their experience and their familiarity with various solutions, design situations and – even more importantly – their clever queries, helped me in developing a better understanding of not only possible solutions but also of the problems to be solved.
Furthermore, this Agile principle does not encourage the development and use of a collective body of knowledge, but in fact limits the playground of the developers, and confines them to look within their own self-organizing team. By doing so, Agile establishes itself as “old-fashioned,” as it misses out on the benefits of the Web 2.0 age. Returning to the pair programming example: it might be preferable that instead of designing in a pair, each individual takes the time to search the Internet for related problems and design patterns, and draw design principles from the abundant body of knowledge available online.
As for requirements, they typically emerge from a deep understanding of both the problem and the product domains. Systems Engineering has been putting the emphasis on this for a long time, including the development of architecture frameworks that incorporate operational views with technical ones; and even the aforementioned Lean Product and Process Development (LPPD) system is preferable to Agile with regards to this, as LPPD puts great emphasis on the learning stage (“Kentou”).
12. “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
Reflection and debriefing, of both failures and success has always been the basis for improvement. No argument here.
A New Set Of Development Principles
After discussing the Agile principles and their shortcomings, I would like to suggest a new set of overarching principles for software systems development. While these principles are not novel in any way, I believe that their composition as a cohesive and clear set of principles will encourage better development processes and methods tailoring. It is a set firm enough with regards to universal development objectives, yet open and expandible. It is with purpose that I do not detail how to achieve these, as I believe tailoring is a key in establishing the right tools for every specific development situation.
The development effort principles are:
1. Define requirements, involving all relevant stakeholders.
2. Plan the development process, including but not limited to:
a. Development iterations (including entry and exit criteria)
c. Reflection events, methods and triggers
d. Methods to be used
e. Required Resources
3. Work in a disciplined manner to develop a system which satisfies the requirements, according to the plan.
Agile has helped in maturing both the discussion and the practice of development techniques in the software development domain, and has even spawned methods that have enriched the arsenal of development methods.
Still, the demonstrated fact that Agile principles offer a specific, narrow take on methods and process engineering (disregarding time, scale and stakeholders objectives issues amongst others), discourages situation-specific tailoring using the wider, richer and continuously evolving set of available methods and body of knowledge.
It is time to acknowledge that there is a greater world out there, with high diversity of design situations. By gaining better understanding of the development situation’s characteristics as well as of development methods and their applicability, and in the light of the set of overarching development principles introduced here, we can use existing knowledge to compose a development process fit for the situation.
Your feedback is valued. Please send it to shavi81 @ gmail.com (omit blank spaces to form the address).