What’s a Methodology For?

As I work on Implementing Agile Values and Principles, the follow-up book to Understanding Agile Values and Principles (UAVP) (https://www.infoq.com/minibooks/agile-values-principles/), I have been editing out material that, while of some interest, does not really advance the point of this new book. I’ve decided to post this material, bit by bit, here on Medium. This is the third of these.

— — — — — — — — — — — — — — — — — — — — — —

Over the course of several decades of software development experience, there have been many discussions published regarding the challenges faced and the problems encountered. Understanding what is really needed to achieve an organization’s business goals and properly translating that understanding into software systems which support those ends are two major challenges. (During my consulting and training over the past 5 years, people have identified over 300 such challenges in their work.) The nature of the relationship between development/IT and stakeholders, both internal and external, is what leads to either overcoming or succumbing to these challenges.

From a product development methodology perspective, there are two forces at work. Either can lead to succumbing to the second of the main challenges. Making it even harder is the fact that these forces seem opposing, even contradictory. On the one hand, inconsistency in following whatever methodology is prescribed for an organization is identified as a cause of project failures and quality problems. On the other hand, it is just as true that inability of methodologies to be flexible enough to handle the inevitable changes in business needs is just as likely to lead to problems and project failure.

Traditional formal quality concepts and software lifecycles began by emulating a manufacturing model. Classic quality principles applied to physical products exist, I believe, because of two key aspects of manufacturing:

(1) the impact to manufacturing cost if changes occur or design mistakes are discovered after tooling and assembly-line processes are prepared and

(2) the repetitive nature of most manufacturing where the product is produced over and over through consistent, reliable repetition of a process.

Production engineering concerns, therefore, derive from these considerations. A process is desired that counts on little (or no) change and highly predictable behavior with quality being assessed by sampling techniques to determine the consistency and repeatability of that process.

But new products are initiated using design engineering, not manufacturing, concepts. Software is a process of iterative design, just like the design of physical products. The only “manufacturing” in software is when a set of software components are “built” into a full system and, eventually, get copied to some media for distribution. Indeed, among the first things to be automated as development tools (after compilers) were systems to manage the repetitive aspects of managing and building software, i.e., source code control and configuration build systems. Software has evolved, by its very nature, to be based on change and addressing “unknowns” (or, at least, needs that are difficult to formally express). This calls into question how manufacturing inspired attempts to get it all “right” up front and freeze changes for relatively long periods of time can be ultimately satisfactory for software development.

Taking a manufacturing approach for creating software, while based in the sincere desire to pursue the same due diligence done in manufacturing, often results in methods implicitly targeted at control mechanisms. That is, methodologies are viewed as mainly existing to prevent incorrect results from being produced, just as quality systems in manufacturing exist to guarantee requirements are met, rework is minimized, and schedules are made predictable. However, methodologies should be viewed as being there to help do the job well not (just) prevent people from doing it badly. Certainly, a methodology should be easy to use correctly and be an integral part of “real work.” (One of the problems in software development, of course, is getting agreement on what the “real work” is.)