Implementing Agile Values and Principles: Chapter 3 (2nd post)

A few years ago I wrote Understanding Agile Values and Principles (https://www.infoq.com/minibooks/agile-values-principles/). I’m working on a follow-up book about implementing them entitled Implementing Agile Values and Principles. I will post draft parts of the book on Medium.com as I finish them. I would certainly be interested in hearing people’s opinions about the draft material which I will certainly consider in making revisions as I work toward the final version.

This second post is the draft of the second (and last) part of the 3rd chapter on Working Software.

Documentation

As noted above, part of being done is to have completed all the necessary documentation both internal and customer focused. The latter should be considered part of the value delivered, i.e., the left side of this value statement. User guides, installation procedures, help-system entries, and training materials are all part of customer-focused documentation needed to make effective use of other aspects of the delivered value. It can be argued that the other parts of that value don’t “work” if needed documentation is absent.

Comprehensive Documentation

“I have only made this letter longer because I have not had the time to make it shorter.” — Blaise Pascal

The right side of this value statement says, “comprehensive documentation.” Early concerns expressed by both traditional software engineering and developers thought this encouraged no documentation. The former found that idea terrifying and threatening to decades of effort to improve quality while the latter found it freeing as they never liked writing documentation anyway.

This value never intended to reject documentation. The idea was to pursue an approach to documentation that made it “sustainable.” As mentioned in UAVP, this meant documentation considered so valuable people willingly created it, referred to it, and kept it updated.

Where possible, executable work products (e.g., tests) are preferred to static ones (e.g., traditional text documents). However, if you successfully employ artifacts such as Use Cases, UML Diagrams, E-R Models, State-Transition Diagrams, Prototypes, etc. and they fit well with ideas noted below, certainly don’t abandon them. For some technologies that implement such forms of documentation there may be an “executable” component to them.

Why Document?

Typically, when asked, “Why do we document things?” people often answer by saying something like “So we/they know what we/they are doing.” This refers, not to customer-focused documentation such as noted above, but process related documentation including requirements, analysis and design specifications, V&V plans and specifications, and other types of plans for what should/will happen and records of what did/does happen. The problem with “So we/they know what we/they are doing” is that it can degrade into a lowest common denominator approach to work. The goal becomes to control the unpredictable them on the project.

Perhaps more generous views of the purpose of documentation are for it to:

“explain what should/will happen,”

“clarify what is happening,”

“record what did happen,” and

“support future enhancement and maintenance.”

As understandable as these may be approaches to doing work this often comes back to “so somebody knows what somebody (else) is doing.”

(Use of documentation as audit records raises an interesting observation about auditing: auditing is about someone coming into an organization to try to find out if they did the “right” things when that person wasn’t around to see what happened. Reliance upon “comprehensive” documentation has been a classic way to presumably prove that.)

Unfortunately, comprehensiveness of documentation can offer a potentially false sense of confidence by suggesting it is:

· complete and authoritative because it must be so if we are to assume appropriate due diligence and responsibility have been achieved.

· precise and agreed upon by everyone involved because it must be so if we are to accept the quality of the content and the effectiveness of understanding about what is written without other forms of confirmation.

The Agile belief that direct, real-time communication matters is based on being able to confirm, in real-time, the confidence desired above. It is about clarifying potential differences in understanding and interpretation, working out the effective resolution to those problems, and resulting in the shared understanding needed for a successful outcome. Verbal “documentation” is often appropriate and sufficient to avoid wasted effort due to rework because of misunderstandings. (The diagram describes Alistair Cockburn’s view of forms of documentation/communication based on his study of successful work groups.)

One aspect of why things are documented as noted above is so people gain a level of confidence in what is, or should be, happening. Stakeholders say they want some sort of documentation but may not understand the costs involved yet still require you to produce the document. It can be difficult to discuss with them if they truly need it. Need may not relate to usefulness. It may have to do with other aspects are not always beneficial:

· The request for documentation represents some form of control being exercised as the requester wants to be seen as, or be, in control of the work. This may be an aspect of the requestor justifying their role in the work. Consider asking them how they will make use of the documentation they are requesting and ask if interaction with the product as it is created will allow them to get what they need and

· The requester is used to such documentation and has not had the Agile approach explained to them. This is reinforced when the process for doing the work says the document should be created because it always has been. An aspect of this is that such documentation provides reassurance about the sufficiency of the work. Again, offering them the opportunity to interact with the work as it is produced might be a reasonable way to provide that confidence.

· The requestor represents compliance expectations for producing the documentation. This might be due to government regulations, process certification/registration standards, or vendor contractual requirements. As necessary as these might be, sometimes the requestor believes comprehensiveness is required by such compliance when it often isn’t.

A final consideration is related to the extensiveness of the functionality expected in the product. The Standish Group has conducted many industry surveys to examine things like process effectiveness, project success patterns, and, related to the discussion of documentation, the frequency of use of the functionality required.

The diagram indicates that much of the functionality in a product is not used frequently, or at all. From an Agile development perspective, this reinforces incrementally developing a product focusing on the highest valued, and presumably most frequently useful, functionality. From a documentation perspective, documenting with the same incremental, value-driven approach makes sense. It will make the best use of everyone’s time, effort, and costs in pursuit of high-value results throughout the work. It may avoid developing and documenting the 45% of Never used functionality and even the 19% of the Rarely used parts. At least, it will get the 36% of Always, Often, and Sometimes used functionality available as soon as possible which can provide the feedback needed to know of the remaining 64% needs to be created or when.

Some Things to Consider about Documentation

The following ideas and suggestions are intended to offer useful thoughts regarding documentation and approaches for creating and using documentation:

1. The goal should always be to effectively communicate, not document.
2. Being “comprehensive” does not guarantee project success but may actually increase the likelihood of failure because of the opportunities to misunderstand/misinterpret what such comprehensive documentation tries to convey. The goal should be the lean idea of “just enough” documentation to be helpful but not so much to be discouraging.
3. Document if that’s the best way to achieve results, but consider there may be better ways to do so than writing documents.
4. If you are going to write documentation, accept the total cost of ownership and explicitly decide to make that investment believing the benefit is greater than the cost of both creating and maintaining such documentation.
5. People, especially development teams often distrust (particularly detailed) documentation because it is not kept up to date and believing it could be, at the least, wasteful and even dangerous.
6. Different work has different documentation needs. One size does not fit every situation. Documentation expectations must be flexible to allow for this or you’ll get documentation for documentation’s sake not truly valuable information.
7. Always consider why you believe you need any particular documentation as opposed to why it’s always been wanted. (In UAVP I noted hearing Barry Boehm speak about “memorial libraries” filled with shelves full of binders where only auditors ever looked at what was inside them.)
8. Wait until things stabilize before you (too extensively) document them, seeking and acting on feedback regularly.
9. Where possible, prefer executable work products over static ones.

Michele Sliger and Stacia Broderick in their 2008 book, The Software Project Manager’s Bridge to Agility, said

“In business terms, spending lots of time up front to capture every design detail in a specification can be a waste of time. Most agile teams say that design changes as the system is built, which results in outdated documentation; therefore, why waste time documenting ideas that will most likely change as implementation begins, and as customer feedback is received?”

and

“…in traditional projects when the scope is defined up front, the scope is protected in order to keep the project ‘on schedule and within budget,’ even when the features defined up front need to change or even be dropped based on changes in the customer’s environment. Therefore, why should we develop something that will not be used by our customers? In fact, why even write about it?”

What Documentation Needs to Be

You will be writing some documentation since a minimal recording of what was done does have future benefits when returning to what was produced when you need to enhance and/or maintain it in some way. Being able to offer people new to the product some light, but useful, documentation allows them to learn more effectively about the product with which they are going to interact.

While communication with people already knowledgeable in the product can be very effective, it is not effective for all knowledge to come from “Harry…he’s been here a long time and knows about it.” Indeed, this latter approach can make reasonably competent people feel far less competent if they cannot learn useful things on their own without being dependent on “Harry” all the time.

A couple key properties of effective documentation are that:

· It is unambiguous, preventing misinterpretation. Some form of peer (or customer) review can confirm this. If people have (too many) questions after such review, it likely isn’t clear enough. (Consistency of terminology contributes to clarity.)

· It is complete enough that relevant aspects of what the customer wants are present. It explains, for example, key features, how the product is intended to be used, who it is intended to be using it, and any constraints that exist. (Being kept up to date contributes to such relevance.)

In order to get these important aspects of documentation there are some approaches that can be helpful.

First, development staff may not be the people with the skill to produce truly effective documentation. However, they have important knowledge about the product that needs to be captured. If development staff can work collaboratively with technical writers, they will learn from one another and produce the best documentation possible. Being on the same development team will foster this relationship.

Second, what development teams need is different from what people might need after a given development period is completed. The development staff, with their more intimate knowledge of what is developed, may not need the same level of information that people after development require. Rough (but sufficient) forms of documentation may work during development where some greater formality is needed after that.

Third, and related to the last point, is when documentation gets produced. Writing it as the product is being created may seem to make sense, but that may slow down the delivery of product value to the customer. It may also mean creating documentation that will have to be reworked as the increments of product are completed. Rough documentation along the way, as noted above, may not be sufficient afterwards. But rough documentation can be used to produce the final documentation needed.

If you wait until everything is done and stabilized, this has disadvantages. People may simply forget rationale and reasons by the time documentation is created. People with aspects of the knowledge may end up being moved to other work and not available to do more for their prior project. And people may frankly no longer have the commitment to do such documentation with other “real” work facing them. (Interestingly, these are also problems with doing lessons learned sessions at the end of a project rather than regularly during the work.)

An Agile approach to documentation, therefore, recognizes the purpose for creating documents which is to document things that are important to know. This means focusing on what may not be obvious such as the “why” (rationale) associated with the work. It also means taking a lean, just in time approach to providing the needed details so others can benefit from that knowledge.