• Bytesmiths Editions -- large, archival, fine-art photography on unusual materials
• Bytesmiths Press -- artists' services: web design/hosting, jury slides, giclee reproductions, opening announcements, brochures, etc.
• Champagne Beadworks -- handcrafted jewelry and beadwork
• Crafted By Carol -- handcrafted jewelry and beadwork
• Dharm Atma Yoga -- Kundalini yoga instruction
• EcoReality, an organization devoted to establishing a sustainable ecovillage
• Ecovillage Newsletter -- Diana Leafe Christian's news of her travels.
• Environmental Education Outreach -- providing environmental education worldwide.
• Gemini Gypsy -- Carole Good-Hanson's fused glass frames
• Green Chipper -- light forestry and environmental services.
• Salt Spring Island Society for Community Education -- community education on our island of 10,000.
• Veggie Van Gogh -- two artists' mobile warehouse and living quarters, petroleum-free!
• Veggiemog -- life and times of Kelly O'Toole's Unimog, running on biodiesel

Preface

26 January 1999, Jan Steinman

This was the initial column in a series that continued up until the final issue of The Smalltalk Report, in October, 1997.

Column Introduction

Smalltalk is growing up. It's rapidly moving out of the research-lab "onesy-twosy" environment into production environments involving large teams, with the requisite procedures, standards, conventions, and bureaucracy. Through years of introducing Smalltalk into organizations, we've noticed that cultural and procedural issues have more impact on success than technical issues.

In this column, we'll be bringing you tools and techniques we've found useful in the broadest sense of "managing objects." If you are involved with Smalltalk (or hope to be involved), and are a manager or technical leader (or hope to be one!), we hope to be addressing many of your project-related concerns.

Success at last! It's been tough -- you managed to put a good team together, train them, fight the "why aren't you using C++?" kinds of battles, and still get your project done in record time. It has amazingly few problems for a new project, and the alpha-users love it, so you take it to your department head who schedules a meeting for final release approval.

Waiting at the release meeting is an old political enemy, who at first opportunity says, "So, the project is documented in accordance with MegaCorp Standards and Procedures Manual section 32, subsection C?"

"Well, not really, you see, because it's, well, new technology, and things have to be a bit different, and..." The meeting falls into disarray, you overhear someone whisper to your boss, "Right, if I didn't have to follow the rules, I could be a hero too!" and the consensus finally emerges that it wasn't a fair race. Release is delayed; you are sent back in disgrace to "make things right."

Despite this setback, you eventually deploy, and senior management is impressed enough to try again. "This time, by the book," you mutter as you start your second Smalltalk project.

As the weeks go by, you notice that things aren't as smooth as before. Half the team isn't documenting to standards, and the other half are whining about having to get out of Smalltalk to run WordBlaster 6.0, or they're doing massive copy-and-paste documenting which is quickly out-of-date and never revisited. This time, you meet the corporate standards, but at a heavy toll in productivity.

Over 20 years ago, Donald Knuth had a similar problem. He noticed that software was not as "obvious" as the original author thought it was when writing it, even to the author himself after a few weeks! On the other hand, switching back and forth between coding and documenting was tedious and disruptive. So he invented "literate programming," in which the documentation is tightly bound to the code.

About the same time, Ted Nelson was dreaming about "hypertext," interconnecting all the information in the world in such a way that simply referring to something would take you deeper into its meaning.

As with many things, Knuth and Nelson were ahead of their time. The technology for literate programming was necessarily at a concrete level and batch-oriented, and hypertextual documents were more easily read than written.

A bit later, Adele Goldberg and a group of Xerox researchers were working on a programming system that, among other things, would greatly simplify both abstract expression and programming. The stage is set for hyper-literate programming!

A Process Of Continuous Documentation

In a perfect world, there would be no programs. A computer user would describe a problem and a proposed solution in natural, but precise language, and feed it to the computer. This is a long way off -- natural language is not precise enough, and computer language is not natural enough!

Keeping as close to that ideal as possible, we can set down some principles for documenting software "things"; a description of a thing must

1. be on the same conceptual level as that thing,
2. constantly and accurately describe that thing,
3. be accessible; by creators, their peers, re-users, reviewers, end-user documentors, and the merely curious, and
4. be measurable, quantitatively and especially qualitatively.

Principle 1: Conceptual Integrity

Driven largely by the limited abstraction available in traditional languages, most organizations have a limited bevy of documentation levels. These often follow the physical organization of the code: a function specification describes a single function, a module specification describes the functions in a file or directory, a system specification describes what you get when you type "make", or the result of some other build script.

Your documentation has conceptual integrity when it describes a software component at the same conceptual level as that component.

We specify Smalltalk software components at many levels, and add new ones as needed. Specifications we've found necessary are at the level of method, class, class extension, variable, nestable module, and configuration. (We'll explain each of these shortly.)

In addition, we've added other useful documentation components, such as gating checklists, requirements maintenance and tracing, and meeting minutes.

Principle 2: Constant Accuracy

In the old days, it was simple. You opened both your ".c" file and your doc file in emacs or vi, and you worked on them simultaneously. Unfortunately, this only really works at a single conceptual level, thus violating Principle 1. Also, the popularity of WYSIWYG editors and specialized coding tools has weakened this binding, since many developers lack the memory and processing resources needed to have both their coding and documenting environments running at the same time!

We've adopted a simple strategy that we think Knuth would approve: the documentation for a thing resides with the thing it describes. This has always been the case for methods, but is present in varying degrees for other components, depending on Smalltalk dialect and additional tools used.

At a more subtle level, we never "comment" our code, we "specify" it. A comment sounds optional, while even "cowboy coders" can appreciate the need for specifications -- especially when they need them from someone else!

Principle 3: Accessibility

Much of the move to WYSIWYG tools for documentation has been driven by accessibility. A nicely formatted bit of paper can reach a much broader audience than can the programmer-accessible file C:\PROJALPH\INPUTSYS\MOD01334.DOC, for example!

Simply adopting the Principle 2 tactic of co-residence for docs and software components vastly improves programmer accessibility, which makes continuous documentation practical, but this does little for the non-coding audience. Also needed is a way to "roll up" the conceptual levels in a hyper-access way that Nelson would find appealing. Although it should be discouraged, there will also remain the need to print a serialized version of an entire portion of the documentation tree.

Principle 4: Measurability

Traditional projects rarely measure project documentation, or they may only take gross physical line/file counts. What is needed is a qualitative measure that is significant in evaluating the overall project. In this sense, document reviews are much more important than code reviews, and are aided by compliance with the other three principles.

Simply having a project tollgate or milestone associated with documentation quality is not enough, however. To ensure compliance with Principle 2, it must be measured continuously. This does not mean daily, time-consuming review meetings; it means developing a team culture in which developers continuously refer to each other's latest documentation, and work together to correct inaccuracies on the spot.

Because of various documentation impediments noted above, the first thing you try when you want to use something in a traditional C project is often "grep the source." By having accurate, accessible documentation at the appropriate level of abstraction, the new ethic must be to first look at the documentation, and to immediately fix things if it is not what you need.

Component Documentation Needs

Not surprisingly, different levels of abstraction have different documentation needs. Here's how we handle the different components.

Method Specifications

Every method must have a specification. Period. In fact, period at the end, capital at the beginning, and grammar throughout -- remember that any method specification might get "rolled up" into some serialized document that a VP will read! It only takes a moment -- do the right thing.

As mentioned, we prefer the term "specification" instead of "comment". What does the method do? How are its arguments used? What objects are the arguments expected to be? What are the error conditions? What does the method answer?

As soon as you decide to create a method, capture in writing what you intend the method to do. (Of course, naming the method properly is vital, as Kent Beck has discussed in his column.) Developers often pay lip service to this rule, and in practice may only comment their methods when some process checkpoint demands that all methods have comments. It is much more difficult to comment a method after the fact, sometimes weeks or months after you wrote it. Or even worse, having to comment a method someone else wrote!

Put yourself in the shoes of a fellow team member who must take over the enhancement or maintenance of your code, or the member of a different team that is a client of your class. What should you tell the enhancer, the client, or the maintainer about this method so they can do their job well? (What should you write to keep clients from misusing your code and reporting "false" bugs against it!)

Tools that automatically generate accessing methods produce comments of little value. VisualWorks will generate instance variables and "getter" methods if you ask it to. The getter method comment simply states that it was auto-generated.

In general "getter" and "setter" methods should specify the variable being gotten or set. What kind of object should go in here? Is it lazily initialized, and guaranteed to never be nil? How does changing it affect the containing object?

We won't go into accessor method philosophy, except to say that they are not always appropriate. If you use tools that auto-generate such methods, realize that the tool cannot specify the meaning of those methods, and document them accordingly.

Class Specifications

As soon as a developer decides to create a class, they must write their justification for the class. Why does the class exist; what does it do? Even during rapid prototyping, a minimum specification for the class is in order -- it is a good work habit to have.

If you cannot yet describe the class at a high level, what sort of behavior are you about to implement for the class? The first specification you write for a class can be a rough draft, but it has to be there.

Throughout your further development of the class you return to the class specification and add details, bring it up to date, and polish it. By the end of the current development cycle the specification will be accurate and complete. The spelling and grammar will be correct. The class owner should feel comfortable having their manager read it. (Or having their manager's manager read it!)

Class specifications are not written once and forgotten. In each subsequent development cycle, the developer will review the specification and update it as required. The only time it is finished is when the class is no longer being changed.

Variables

Each class specification must have a section that documents all variables associated with the class: instance, class, class instance, pool variables, and (ouch!) globals. For each variable, its acceptable objects are listed, and a description is provided about how the variable is used. This is a good place to mention if the value is internally derived or externally settable, and whether it is public or private state. If the variable must be non-nil, the time and place where it is initialized should be spelled out. For example, if this is state that is provided at instance creation, point that out in the variable's comment.

We developed a technique for separating documentation of variables, instead of embedding them in the class specification. [951223-JWS See also SmallDoc, part of the Bytesmiths toolkit.] This separates the specification of a class's state from that of its behavior, thus increasing conceptual integrity. It also allows superclass state documentation to be merged when printing or browsing.

Class Extensions

Most Smalltalk code management systems distinguish between defining a class and adding behavior. Behavior is added in class extensions. Unfortunately, there is no built-in support for documenting class extensions. If you find it necessary to add a suite of methods to an existing class to support your work, shouldn't that need be explained somewhere? We find this necessary, and added class extension support to ENVY/Developer®.; The class extension comment summarizes the behavior added by the extension. Each method in the extension also has a complete specification.

Modules

Smalltalk source code management environments such as Team/V® and ENVY/Developer contain software components that collect classes and/or class extensions. These components are called Packages in Team/V, and Applications or SubApplications in ENVY (which we'll simply call apps). The ability to support code modules larger than classes is essential in even moderate-sized projects! We're more familiar with ENVY, but the following discussion applies equally to Team/V.

We developed "smart" specification templates for apps that generate much of an app specification at the time the documentation is viewed or printed. This follows Principle 1 by letting the developer concentrate on subsystem documentation, Principle 2 by dynamically showing the state of included code modules as of the time of access, and Principle 3 by conditionally including class specifications and other detailed documentation chunks.

The most important part of the app specification is an abstract that explains its purpose and goals. The next section explains its component relationships. These two specification parts are maintained at this level; all other information is available via link, and maintained at a more appropriate place or automatically generated.

An important feature of the smart template approach is that organization-specific data is readily handled. We have sections for document control numbers, cost-center information, and other data specifically required by the company. Paying attention to these things keep the "by the book" crowd happy, or at least tolerant!

Also in the app specification are links to automatically generated information, which is only feasible in a tightly-integrated documentation system. For example, the app version and time stamp, prerequisites, class hierarchy, and system event method specifications are among the automatically linked information.

Other information that we have included (via link) in the app specification are references, glossary, declared external interfaces, [951223-JWS: See also our marketing blurb on SmallDoc.] design decisions in the form of meeting minutes, requirements, use cases, test cases, and test results.

Linking information dynamically is more important in app specifications than elsewhere. The developer doesn't want the clutter of multiple "boilerplate" items that are not important to him, yet others may want to see everything. Good examples of linked information that is necessary, but should be hidden most of the time are specifications of contained subapps and contained classes.

Configurations

ENVY and Team/V have ways of collecting modules into "load builds" of some kind. In ENVY, they are called configuration maps, or just maps. The specification for a map provides an overview of what it will load, and any other maps that should already be loaded. If for some reason the map is not unloadable, or is compatible with only certain versions of prerequisite maps, it needs to be documented.

An overview of what is different in one version of a map from the previous version is a good idea, but a better convention is generating and editing release notes at the end of each development cycle.

Release Notes

ENVY and Team/V each have facilities for finding differences between two versions of components. We extended ENVY's facility to produce a smart template that captures all the changes in a textual form. Of course, ENVY isn't smart enough to say why something changed, but having a template to complete jogs the developer's memory, and guarantees coverage of all changes. We place these in the ENVY "notes" field of each changed application.

Diagrams

Documents without drawings are as unacceptable as monospaced, 80 column computer displays. Luckily, the publicly available HotDraw drawing framework is available. What it lacks in sophistication, it more than makes up for by being easily adapted to arbitrary object structures.

For example, the HotDraw diagramming inspector is suitable "out of the box" for documenting complex instance relationships that would be difficult to explain in words. Using it as an example, you can easily craft your own boxes-and-lines documenting aids.

We added a simple facility to ENVY for associating hot-drawings with arbitrary software components, for linking those drawings with appropriate browsers, and for embedding those drawings in hardcopy at the appropriate place.

Style Guide

Just as a magazine needs a consistent (or at least non-conflicting) style, an organization needs a guide for its code and documentation. Most Smalltalk projects start by searching for published style guides, adopting them, and modifying them as their needs evolve. We've found less attention is given to the style guide after the early stages of the project -- typically, new hires are given the style guide to read. Hopefully, the developers have internalized the style guide, because they don't use it as a reference. It might come out of the bookcase again at code inspection time, or when the company is being audited for certification of the software development process.

Regardless of frequency of use, it is important to have one. There should be some agreement on what must be documented, and how it should be documented. Typically style guides cover many areas in addition to documentation. Novice Smalltalkers should refer to the style guide, but hopefully, they are also seeing good examples of documentation by fellow team members.

The need for the style guide is less important when there are good templates and tool support for documentation.

Documentation Measuring

Beyond conformance to style and periodic peer review, the quality of documentation is difficult to measure. We've implemented existence checks, but they can't tell the difference between a random characters and a line from Shakespeare. The best guarantor of quality is a group culture that encourages use.

Beyond mere existence checks, we've found a few tools that helped ensure quality documentation. Meeting minutes are linked into documentation to link important design decisions with the components impacted by those decisions, and smart check-lists enable a developer to quickly asses their state of "doneness" for a given development cycle. Finally, we added a document-centered browser, so one could browse component specifications without being distracted by code.

Conclusion

"Well, that all sounds great, but what do I do now?" Everything we've discussed here can be implemented fairly easily, depending on what is available for re-use in your environment. Although your schedule and resource may be such that "the cobbler's children have no shoes," it is also fairly easy to justify spending time building tool support for a continuous documentation regime. We've found it not only increased the quality of project documentation, but resulted in a savings of about eight percent of total project time, which means a team of seven people can justify a half-time toolsmith. In the next issue, we'll present some concrete examples and source code.

Postscript

23 November 1995, the authors

Since writing this column, we provided some implementation detail in another column. (The next column in the series actually got diverted onto a different topic.) Some of these tools are also summarized in our marketing blurb -- blatant commercial advertising you might not be interested in.