3

software for networked information systems 63

ously might have traveled independently, the information now can be transmitted and accessed together.

In building an NIS for an HMO, management is likely to have chosen a "Web-centric" implementation using the popular protocols and facilities of the World Wide Web and the Internet. Such a decision would be sensible for the following reasons:

• The basic elements of the system, such as Web servers and browsers, can now be commercial off-the-shelf (COTS) components and, therefore, are available at low cost.

• A large, growing pool of technical personnel is familiar with the Web-centric approach, so the project will not become dependent on a small number of individuals with detailed knowledge of locally written software.

• The technology holds promise for extensions into consumer telemedicine, whereby patients and health care providers interact by using the same techniques as are commonly used on the rest of the Internet.

Clearly, the HMO's NIS must exhibit trustworthiness: it must engender feelings of confidence and trust in those whose lives it affects. Physicians must be confident that the system will display the medical record of the patient they are seeing when it is needed and will not lose information; patients must be confident that physician-entered prescriptions will be properly transmitted and executed; and all must be confident that the privacy of records will not be compromised. Achieving this trustworthiness, however, is not easy.

NIS trustworthiness mechanisms basically concern events that are not supposed to happen. Nonmalicious users living in a benign and fault-free world would be largely unaffected were such mechanisms removed from a system. But some users may be malicious, and the world is not fault free. Consequently, reliability, availability, security and all other facets of trustworthiness require mechanisms to foster the necessary trust on the part of users and other affected parties. Only with their failure or absence do trustworthiness mechanisms assume importance to a system's users. Users seem unable to evaluate the costs of not having trustworthiness mechanisms except when they experience actual damage from incidents (see Chapter 6 for an extended discussion). So, while market forces can help foster the deployment of trustworthiness mechanisms, these forces are unlikely to do so in advance of directly experienced or highly publicized violations of trustworthiness properties.

Although the construction of trustworthy NISs is today in its infancy, lessons can be learned from experience in building full-authority and other freestanding, high-consequence computing systems for applications

such as industrial process control and medical instrumentation. In such systems, one or more computers directly control processes or devices whose malfunction could lead to significant loss of property or life. Even systems in which human intervention is required for initiating potentially dangerous events can become high-consequence systems when human users or operators place too much trust in the information being displayed by the computing system.¹ To be sure, there are differences between NISs and traditional high-consequence computing systems. An intent of this chapter is to identify those differences and to point out lessons from high-consequence systems that can be applied to NISs, as well as unique attributes of NISs that will require new research.

The Role of Software

Software plays a major role in achieving the trustworthiness of an NIS, because it is software that integrates and customizes general-purpose components for some task at hand. In fact, the role of software in an NIS is typically so pervasive that the responsibilities of a software engineer differ little from those of a systems engineer. NIS software developers must therefore possess a systems viewpoint,² and systems engineers must be intimately familiar with the strengths (and, more importantly, the limitations) of software technology.

With software playing such a pervasive role, defects can have far-reaching consequences. It is notoriously difficult to write defect-free software, as the list of incidents in, for example, Leveson (1987) or Neumann (1995) confirms. Beyond the intrinsic difficulty of writing defect-free software, there are constraints that result from the nature of NISs. These constraints derive from schedule and budget; they mean that a software developer has only limited freedom in selecting the elements of the software system and in choosing a development process:

• An NIS is likely to employ commercial operating systems, purchased "middleware," and other applications, as well as special-purpose code developed specifically for the NIS. The total source code size for the system could range from tens to hundreds of millions of lines. In this setting, it is infeasible to start from scratch in order to support trustworthiness.

¹This is a particularly dangerous state of affairs, since designers may assume that system operation is being monitored, when in fact it is not (Leveson, 1995).

²Once succinctly stated as, "You are not in this alone." That is, that you need to consider not only the narrow functioning of your component but also how it interacts with other components, users, and the physical world in achieving system-level goals. Another aspect of the "systems viewpoint" is a healthy respect for the potential of unexpected side effects.

software for networked information systems 65

• Future NISs will, of necessity, evolve from the current ones. There is no alternative, given the size of the systems, their complexity, and the need to include existing services in new systems. Techniques for supporting trustworthiness must take this diversity of origin into account. It cannot be assumed that NISs will be conceived and developed without any reuse of existing artifacts. Moreover, components reused in NISs include legacy components that were not designed with such reuse in mind; they tend to be large systems or subsystems having nonstandard and often inconvenient interfaces. In the HMO example, clinical laboratories and pharmacies are likely to have freestanding computerized information systems that exemplify such legacy systems.

• Commercial off-the-shelf software components must be used to control development cost, development time, and project risk. A commercial operating system with a variety of features can be purchased for a few hundred dollars, so development of specialized operating systems is uneconomical in almost all circumstances. But the implication is that achieving and assessing the trustworthiness of a networked information system necessarily occur in an environment including COTS software components (operating systems, database systems, networks, compilers, and other system tools) with only limited access to internals or control over their design.

• Finally, the design of NIS software is likely to be dictated—at least, in part—by outside influences such as regulations, standards, organizational structure, and organizational culture. These outside influences can lead to system architectures that aggravate the problems of providing trustworthiness. For example, in a medical information system, good security practices require that publicly accessible terminals be logged off from the system after relatively short periods of inactivity so that an unauthorized individual who happens upon an unattended terminal cannot use it. But in emergency rooms, expecting a practitioner to log in periodically is inconsistent with the urgency of emergency care that should be supported by an NIS in this setting.

Fortunately, success in building an NIS does not depend on writing software that is completely free of defects. Systems can be designed so that only certain core functionality must be defect free; defects in other parts of the system, although perhaps annoying, become tolerable because their impact is limited by the defect-free core functionality. It now is feasible to contemplate a system having millions of lines of source code and embracing COTS and legacy components, since only a fraction of the code has to be defect free. Of course, that approach to design does depend on being able to determine or control how the effects of defects propagate. Various approaches to software design can be seen as provid

ing artillery for attacking the problem, but none has proved a panacea. There is still no substitute for talented and experienced designers.

Development of a Networked Information System

The development of an NIS proceeds in phases that are similar to the phases of development for other computerized information systems:

• Decide on the structure or architecture of the system.

• Build and acquire components.

• Integrate the components into a working and trustworthy whole.

The level of detail at which the development team works forms a Vshaped curve. Effort starts at the higher, systems level, then dips down into details as individual software components are implemented and tested, and finally returns to the system level as the system is integrated into a cohesive whole.

Of the three phases, the last is the most problematic. Development teams often find themselves in the integration phase with components that work separately but not together. Theoretically, an NIS can grow by accretion, with service nodes and client nodes being added at will. The problem is that (as illustrated by the Internet) it is difficult to ensure that the system as a whole will exhibit desired global properties and, in particular, trustworthiness properties. On the one hand, achieving a level of connectivity and other basic services is relatively easy. These are the services that general-purpose components, such as routers, servers, and browsers, are designed to provide. And even though loads on networks and demands on servers are hard to predict, adverse outcomes are readily overcome by the addition or upgrade of general-purpose components. On the other hand, the consequences of failures or security breaches propagating through the system are hard to predict, to prevent, and to analyze when they do occur. Thus, basic services are relatively simple to provide, whereas global and specialized services and properties—especially those supporting trustworthiness—are difficult to provide.

System Planning, Requirements,
and Top-Level Design

Planning and Program Management

A common first step in any development project is to produce a planning and a requirements document. The planning document contains information about budget and schedules. Cost estimation and scheduling

software for networked information systems 67

are hard to do accurately, so producing a planning document is not a straightforward exercise. Just how much time a large project will require, how many staff members it will need (and when), and how much it will cost cannot today be estimated with precision. The techniques that exist, such as the constructive cost model (COCOMO) (Boehm, 1981), are only as good as the data given them and the suitability of their models for a given project. Estimation is further complicated if novel designs and the implementation of novel features are attempted, practices common in software development and especially common in leading-edge applications such as an NIS.

Although every attempt might be made to employ standard components (e.g., operating system, network, Web browsers, database management systems, and user-interface generators) in building an NIS, the ways in which the components are used are likely to be sufficiently novel that generalizing from past experiences with the components may be useless for estimating project costs and schedules. For example, it is not hard to connect browsers through a network to a server and then display what is on the server, but the result does not begin to be a medical records system, with its varied and often subtle trustworthiness requirements concerning patient privacy and data integrity. The basic services are even farther from a complete telemedicine system, which must be trusted to correctly convey patient data to experts and their diagnoses back to paramedical personnel. All in all, confidence in budget and schedule estimates for an NIS, as for any engineering artifact, can be high only when the new system is similar to systems that already have been built. Such similarity is rare in the software world and is likely to be even rarer in the nascent field of NIS development.

The difficulties of cost estimation and scheduling explain why some projects are initiated with unrealistic schedules and assignments of staff and equipment. The problem is compounded in commercial product development (as opposed to specialized, one-of-a-kind system development) by marketing concerns. For software-intensive products, early arrival in the marketplace is often critical to success in that marketplace. This means that software development practice becomes distorted to maximize functionality and minimize development time, with little attention paid to other qualities. Thus, functionality takes precedence over trustworthiness.

A major difficulty in project management is coping with ambiguous and changing requirements. It is unrealistic to expect correct and complete knowledge of requirements at the start of a project. Requirements change as system development proceeds and the system, and its environment, become better understood. Moreover, software frequently is regarded (incorrectly) as something that can be changed easily at any point

during development, and software change requests then become routine. The effect of the changes, however, can be traumatic and lead to design compromises that affect trustworthiness.

Another difficulty in project management is selecting, tailoring, and implementing the development process that will be used. The Waterfall development process (Pressman, 1986), in which each phase of the life cycle is completed before the next begins, oversimplifies. So, when the Waterfall process is used, engineers must deviate from it in ad hoc ways. Nevertheless, organizations ignore better processes, such as the Spiral model (Boehm, 1988; Boehm and DeMarco, 1997), which incorporates control and feedback mechanisms to deal with interaction of the life-cycle phases.

Also contributing to difficulties in project management and planning is the high variance in capabilities and productivity that has been documented for different software engineers (Curtis, 1981). An order-of-magnitude variation in productivity is not uncommon between the most and the least productive programmers. Estimating schedules, assigning manpower, and managing a project under such circumstances are obviously difficult tasks.

Finally, the schedule and cost for a project can be affected by unanticipated defects or limitations in the software tools being employed. For example, a flawed compiler might not implement certain language features correctly or might not implement certain combinations of language features correctly. Configuration management tools (e.g., Rochkind, 1975) provide other opportunities for unanticipated schedule and cost perturbation. For use in an NIS, a configuration management tool not only must track changes in locally developed software components but also must keep track of vendor updates to COTS components.

None of the difficulties are new revelations. Brooks, in his classic work The Mythical Man Month (Brooks, 1975), noted similar problems more than two decades ago. It is both significant and a cause for concern that this book remains relevant today as evidenced by the recent publication of a special 20th anniversary edition. The difficulties, however, become even more problematic within the context of large and complex NISs.

Requirements at the Systems Level

Background

There is ample evidence that the careful use of established techniques in the development of large software systems can improve their quality. Yet many development organizations do not employ techniques that have

software for networked information systems 69

been known for years to contribute to success. Nowhere is this refusal to learn the lessons of history more pronounced than with respect to requirements documents.

Whether an NIS or a simple computer game is being implemented, a requirements document is useful. In special-purpose systems, it forms a contract between the customer and the developer by stating what the customer wants and thereby what the developer must build. In projects aimed at producing commercial products, it converts marketing and business objectives into technical terms. In the development of large systems, it serves as a vehicle for communication among the various engineering disciplines involved. And it also serves as a vehicle for communication between different software engineers responsible for developing software, as well as between the software engineers and those responsible for presenting the software to the outside world, such as a marketing team.

It is all too common, however, to proceed with system development without first analyzing and documenting requirements. In fact, requirements analysis and documentation are sometimes viewed as unnecessary or misdirected activities, since they do not involve creating executable code and are thought to increase time to market. Can system requirements not be learned by inspecting the system itself? Requirements derived by such a posteriori inspections, however, run the risk of being incomplete and inaccurate. It is not always possible to determine a posteriori which elements of an interface are integral and which are incidental to a particular implementation. In the absence of a requirements document, project staff must maintain a mental picture of the requirements in order to respond to questions about what should or could be implemented. Each putative requirements change must still be analyzed and negotiated, only now the debate occurs out of context and risks overlooking relevant information. Such an approach might be adequate for small systems, but it breaks down for systems having the size and complexity of an NIS.

The System Requirements Document

The system requirements document states in as much detail as possible what the system should (and should not) do. To be useful for designers and implementers, a requirements document should be organized as a reference work. That is, it should be arranged so that one can quickly find the answer to a detailed question (e.g., What should go into an admissions form?). Such a structure, more like a dictionary than a textbook, makes it difficult for persons unfamiliar with the project to grasp how the NIS is supposed to work. As a consequence, requirements documents are supplemented (and often supplanted) with a concept of operations

(Conops) that describes, usually in the form of scenarios (so-called "use cases"), the operation of the NIS. A Conops for the example HMO system might, for example, trace the computer operations that support a patient from visiting a doctor at a neighborhood clinic, through diagnosis of a condition requiring hospitalization, admission and treatment at the hospital, discharge, and follow-up visits to the original clinic. Other scenarios in the Conops might include home monitoring of chronic conditions, emergency room visits, and so forth. The existence of two documents covering the same ground raises the possibility of inconsistencies. When they occur, it is usually the Conops that governs, because the Conops is the document typically read (and understood) by the sponsors of the project.

Review and approval of system requirements documents may involve substantial organizational interaction and compromise when once-independent systems are networked and required to support overall organizational (as opposed to specific departmental) objectives. The compromises can be driven more by organizational dynamics than by technical factors, a situation that may lead to a failure to meet basic objectives later on. That risk is heightened in the case of the trustworthiness requirements, owing (as is discussed below) to the difficulty of expressing such requirements and compounded by the difficulty of predicting the consequences of requiring certain features. In the case of the HMO system, for example, advocates for consumer telemedicine might insist on home computer access to the network in ways that are incompatible with maintaining even minimal medical records secrecy in the face of typical hackers. Anticipating and dealing with such a problem require predicting what sorts of attacks could be mounted, what defenses might be available in COTS products, and how attacks will propagate through an NIS whose detailed design might not be known for several years. Making the worst-case assumption (i.e., all COTS products are completely vulnerable and all defenses must be mounted through the locally developed software of the NIS) will likely lead to unacceptable development costs. Similar situations arise for other dimensions of trustworthiness, such as data integrity or availability.

Notation and Style

Requirements documents are written first in ordinary English, which is notorious for imprecision and ambiguity. Most industrial developers do not use even semiformal specification notations, such as the SCR/A7 tabular technique (Heninger, 1980). The principal reason for using natural language (in addition to the cynical observation that without ambiguity there can be no consensus) is that, despite significant R&D investment

software for networked information systems 71

in the 1970s (Ross, 1977), no notation for system-level requirements has shown sufficiently commanding advantages to achieve dominant acceptance.

Finally, many—if not most—software developers are forced to lead "unexamined lives." The demand for their services is so great that they must move from one project to the next without an opportunity for reflection or consideration of alternatives to the approaches they used before. The paradoxical result of this situation is that the process of developing software, which has had revolutionary impact on many aspects of society and technology, is itself quite slow to change.

One common strategy for coping with the problems inherent in natural language is to divide the requirements into two classes: criteria for success (often called "objectives" or "goals") and criteria for failure (sometimes called "absolute requirements"). The criteria for success can be a matter of degree: situations where "more is better" without clear cutoff points. The criteria for failure are absolute—conditions, such as causing a fatality, that render success in other areas irrelevant. In the HMO example, a criterion for success might be the time needed to transfer a medical record from the hospital to an outpatient facility—quicker is better, but unless some very unlikely delays are experienced, the system is acceptable. A criterion for failure might be inaccessibility of information about a patient's drug allergies. If the patient dies from an allergic reaction that could have been prevented by the timely delivery of drug allergy data, then nothing else the system has done right (such as the smoothness of admission, proper assignment of diagnostic codes, or the correct interfacing with the insurance carrier) really matters.

It is often posited that requirements should state what a particular criterion is but not how that criterion should be achieved. In real-world systems development, this dictum can lead to unnecessarily convoluted and indirect formulations of requirements. The issue is illustrated by turning to building codes, which are a kind of requirements document. Building codes distinguish between performance specifications and design specifications. A performance specification states, "Interior walls should resist heat of x degrees for y minutes." A design specification states, "Interior walls should use 5/8-inch Type X sheetrock." Performance specifications leave more room for innovation, but determining whether they have been satisfied is more difficult. Design specifications tend to freeze the development of technology by closing the market to innovations, but it is a simple matter to determine whether any given design specification has been fulfilled. More realistic guidance for what belongs in a requirements document is the following: If it defines either failure or success, it belongs in the requirements document, no matter how specific or detailed it is.

A distinction is sometimes made between functional requirements and nonfunctional requirements. When this distinction is made, functional requirements are concerned with services that the system should provide and are usually stated in terms of the system's interfaces; nonfunctional requirements define constraints on the development process, the structure of the system, or resources used during execution (Sommerville, 1996). For example, a description of expected system outputs in response to various inputs would be considered a functional requirement. Stipulations that structured design be employed during system development, that average system response time be bounded by some value, or that the system be safe or secure exemplify nonfunctional requirements.

Nonfunctional requirements concerning execution theoretically can be translated into functional requirements. Doing that translation requires knowledge of system structure and internals. The resulting inferred functional requirements may concern internal system interfaces that not only are unmentioned in the original functional requirements but also may not yet be known. Moreover, performing the translation invariably will involve transforming informal notions, such as "secure," "reliable," or "safe," into precise requirements that can be imposed on the internals and interfaces of individual modules. Formalizing informal properties at all and decomposing systemwide global properties into properties that must be satisfied by individual components are technically very challenging tasks—tasks often beyond the state of the art (Abadi and Lamport, 1993; McLean, 1994).

Where to Focus Effort in Requirements
Analysis and Documentation

The process of requirements analysis is complicated by the fact that any NIS is part of some larger system with which it interacts. An understanding of the application domain itself and mastery of a variety of engineering disciplines other than software engineering may be necessary to perform requirements analysis for an NIS. Identification of system vulnerabilities is one process for which a broad understanding of the larger system context (including users, operators, and the physical environment) is particularly important. Techniques have been developed to deal with some of these issues. Modeling techniques, such as structured analysis (Constantine and Yourdon, 1979), have been developed for constructing system descriptions that can be analyzed and reviewed by customers. Rapid prototyping tools (Tanik et al., 1989) offer a means to answer specific questions about the requirements for a new system, and prototyping is today a popular way to determine user interface require

software for networked information systems 73

ments. Systematic techniques have been developed for determining application requirements by either interviewing application experts or observing the actions of potential users of the system (Potts et al., 1994).

Interviews conducted in the 1970s with experienced project managers revealed their skepticism about making significant investments in system-level requirements documents (Honeywell Corporation, 1975). Those veterans of large-scale aerospace and defense projects believed that any significant efforts regarding requirements should be directed to the level of subsystems or components. They argued that system-level requirements documents were seldom consulted after detailed component-level requirements were written. Change—sometimes significant change—in system-level requirements was quite common and rendered obsolete a system-level requirements document.

Changes in requirements originate from a variety of sources:

• The outside environment may change—the example HMO could merge, restructure, or be affected by new statutory or regulatory forces.

• The advent of new technology could generate a desire for the enhanced capability that the technology provides. This factor would be amplified for the HMO's NIS by the current rapid development pace of Internet-related technology (so-called Internet time) and the false perception that components and features can be added to an NIS with relative ease.

Requirements errors are the most expensive to fix, because they typically are not found until significant resources have been invested in system design, implementation, and, in some cases, testing and deployment. The high cost of repairing such errors would then justify expending additional resources on systems requirements analysis and documentation. But that argument is incomplete, for it presumes that the additional expenditures could prevent such errors. Published (Glass, 1981³) and unpublished (Honeywell Corporation, 1975) studies of requirements errors indicate that errors of omission are the most common. Experienced program managers, who have internalized the experience of unpleasant surprises resulting from combinations of inputs and internal states (or other phenomena that were thought to be impossible), understand that no amount of effort is likely to produce a complete requirements document.

Resources expended in requirements analysis and documentation are, nevertheless, usually well spent. The activity helps a system's developers to better understand the problem they are attacking. Design and coding

³This reference contains the classic "Reason for Error" entry in a trouble report: "Insufficient brain power applied during design."

decisions are thus delayed until a clearer picture of needs and constraints has emerged. It is not the documentation but the insight that is the important work product. Conceivably, other techniques could be developed for acquiring this insight. However, systems requirements documents serve also for communication within a project team as well as with customers and suppliers; any alternative technique would have to address this need as well.

Doing a bad job at requirements analysis actually can have harmful long-term repercussions for a development effort. Requirements analysis invariably goes astray when analysts are insufficiently familiar with the anticipated uses of the system being contemplated or with the intended implementation technology. It also can go astray when analysts become grandiose and formulate requirements far in excess of what is actually needed. Finally, inevitable changes in context and technology mean that requirements analysis and documentation should be an ongoing activity. To the extent possible, requirements should be determined at the outset of development and updated as changes occur during development. In practice, requirements analysis and documentation mostly occur early in the process.

Top-level Design

The trustworthiness of a system depends critically on its design. Once the system's requirements and (optionally) the Conops are approved, the next step is development of a top-level design. This document is often called an "architecture" to emphasize just how much detail is being omitted. During development of the top-level design, basic types of technology are selected, the system is divided into components and subsystems, and requirements for each component are defined. This process has been called "programming in the large," to distinguish it from writing code, or "programming in the small" (DeRemer and Kron, 1976).

Components are building blocks for integration, and subsystems are clusters of components that are integrated first as a group and then the assemblage integrated into the whole. For software that is being developed (as opposed to purchased), the size of a component or subsystem is determined by the number of lines of code, the programming language used, and the complexity of the algorithms involved. A rough rule of thumb is that a component (or "module") is a body of software that can be fully grasped⁴ by one or two programmers. Using the same principle, a

⁴That is, some member of the team can answer any question about the subsystem; it is not necessary (or even desirable) that every member of the team be able to answer every question.

software for networked information systems 75

subsystem is a body of code that can be fully grasped by a team of three to five programmers, which happens also to be the maximum size group that can be supervised effectively by a team leader.

There exist no generally accepted notations for top-level design. Most designs are described using diagrams. Such diagrams rarely have precisely defined semantics, so they are not always helpful for determining whether a top-level design includes all the necessary functions or satisfies all of its requirements.

A dependency analysis (Parnas, 1974) should be performed on the top-level design, where a dependency is defined to exist between components A and B if the correct operation of A depends on the correct operation of B. The results of a dependency analysis are captured in a dependency diagram.⁵ Experienced designers attempt to move functions among components to eliminate cycles in the dependency diagram. In a cycle, the correctness of one component depends directly or indirectly on the correctness of another, and the correctness of the second depends directly or indirectly on the correctness of the first, thereby forming a circular relationship. Where a cycle exists, all components in the cycle must be integrated and tested as a unit. In the extreme case—so-called "big bang" integration—all components are integrated at one time; that process seldom has a positive outcome. At present there is no scientific foundation for determining, analyzing, or changing dependency relationships among components in large-scale systems.

Many would argue that interface determination and design are the essence of system design (Lampson, 1983). Therefore, an important output of the top-level design activity is precise specifications for the system's interfaces. These specifications define the formats and protocols for interactions between components and subsystems. A rigorous interface description is particularly important when the interface being defined is between subsystems implemented by different teams.⁶ The definition of interfaces and the determination of which interfaces are sufficiently important to warrant control by project management are, like the rest of top-level design, more an art than a science.

⁵As with the top-level design itself, there exist no generally accepted notations for such diagrams, nor do there exist widely used tools to support the development of dependency diagrams.

⁶There is an element of program management lore called Conways's Law whose essence is that the human organization of a software project and the technical organization of the software being produced will be congruent. The law was originally stated as, "If you have four teams working on a compiler, you get a four-pass compiler." A more general formulation is that "a system's structure resembles the organization that produces it" (Raymond and Steele, 1991).

Despite the innovative design concepts that have appeared in the literature in areas such as object-oriented design (Meyer, 1988) and architectural description languages (Garland and Shaw, 1996), still no comprehensive approach to the design and analysis of NISs exists. Important challenges remain in design visualization, design verification, design techniques (that accommodate long-term evolution), COTS, and legacy components, as well as tool support for the creation and analysis of designs. Among the most critical issues are design verification and design evolution, since assuring that a design will continue to implement the necessary trustworthiness properties—even as the system evolves—is central to building an NIS. Moreover, because top-level design occurs relatively early in the life cycle, detection of defects during the top-level design stage has great leverage.

Perhaps the greatest design challenges concern techniques to compose subsystems in ways that contribute directly to trustworthiness. NISs are typically large and, therefore, they must be developed and deployed incrementally. Significant features are added even after an NIS is first deployed. Thus, there is a need for methods to identify feature interactions, performance bottlenecks, omitted functionality, and critical components in an NIS that is being developed by composition or by accretion.

There exists a widening gap between the needs of software practitioners and the ability to evaluate software technologies for developing moderate- to large-scale systems. The expense of building such systems renders infeasible the traditional form of controlled scientific experiment, where the same system is built repeatedly under controlled conditions but using differing approaches. Benefits and costs must be documented, risks enumerated and assessed, and necessary enhancements or modifications identified and carried out. One might, instead, attempt to generalize from the experiences gained in different projects. But to do so and reach a sound conclusion requires understanding what aspects of a system interact with the technology under investigation. Some advantages would probably accrue if only software developers documented their practices and experiences. This activity, however, is one that few programmers find appealing and few managers have the resources to support.

Critical Components

A critical component is one whose failure would result in an undetected and irrecoverable failure to satisfy a trustworthiness requirement. Experienced designers attempt to produce top-level designs for which the number of components that depend on critical components is not constrained but the critical components themselves depend on as few other

software for networked information systems 77

components as possible. This strategy achieves two things: it enables developers to use freestanding tests and analyses to build trust in the critical components, and it permits an orderly integration process in which trusted components become available early. Unless the critical components come from vendors with impeccable credentials, development teams generally prefer, wherever feasible, to implement the critical components themselves. That way, all aspects of the design, implementation, and verification of critical components can be strictly controlled. There are two risks in pursuing this approach. One is that the criticality of a component has been overlooked—a danger that is increased by the lack of a scientific basis to assess the criticality of components. A second is that it may not be feasible to implement a critical component in-house or, for a vendor-provided critical component, it may not be possible to obtain sufficient information to be convinced of that component's trustworthiness.⁷

The Integration Plan

Once the basic structure of the system has been established, the integration plan is produced. Ideally, the plan involves two activities:

1. Integration of components into subsystems that reside on single network nodes; and

2. Connection of network nodes into subsystems that perform definable functions and whose behavior can be observed and evaluated, followed by the connection of the subsystems into the final NIS.

The essence of the integration process is progress toward a completely operational system on a step-by-step basis. Observed defects can be localized to the last increment that was integrated—if one build passes its tests and the next build fails its tests, then the most likely sources of difficulty are those components that turned the first build into the second. Working in this manner, the integration team should not have to revisit previously integrated components or subsystems during the integration process. And this process avoids a cycle of "fix and test and fix again" that could continue until time, money, or management patience runs out. Note that for the integration process to be successful, the top-level design must exhibit proper dependency relationships between components. An inte

⁷In the case of a browser, which might well be a critical component in an NIS, this situation is ameliorated by Netscape's recent decision to release the Netscape Navigator source code. A development team now can examine the code and possibly eliminate unwanted functionality.

gration plan thus can serve another purpose: to force the detailed analysis of a top-level design. Top-level designs lacking straightforward integration plans are likely to be ambiguous, incomplete, or just plain wrong.

Integration skills today are developed only through experience. There is essentially no theoretical basis for deciding what should constitute a build, nor has the problem received serious scientific examination. System integration continues to be practiced as a craft that is passed along through apprenticeship. The drift of university computer science research from emphasizing large experimental systems projects (such as Multics, c.mmp, and Berkeley UNIX) toward undertaking smaller engineering efforts is of particular concern. Looking back at the master's and Ph.D. thesis topics at the Massachusetts Institute of Technology (as an example) during the Multics era, it is striking how many concern software that had to be integrated into the larger system in a planned and disciplined manner. The shrinking of this skills base in orderly integration is further exacerbated by the reward system of the personal computer market. Financial benefits flow principally to authors of the freestanding application or component (the so-called "killer app") that attracts large numbers of consumers or is selected for use in information systems assembled from COTS components. This latter case involves a different set of skills from those required to design, implement, and integrate a large system from scratch.

Project Structure, Standards, and Process

Other branches of engineering rely heavily on controlling the development process to ensure the quality of engineering artifacts. The Software Engineering Institute's Capability Maturity Model (CMM) is a step in that direction for software design and development (see Box 3.1). As with requirements definition and analysis, there is considerable anecdotal evidence and some experimental evidence that having a systematic process in place contributes to the quality of software systems that an organization develops. There is, however, little evidence that any one process can be distinguished from another, nor is there evidence that different characteristics of development processes are correlated with product quality.

Rigorous, repeatable processes are sometimes thought to result when software development standards are imposed on organizations. Such standards typically prescribe overall process structure, documents to be produced, the order of events, techniques to be used, and so on. A recent study found 250 different standards that apply to the engineering of software, yet the authors of the study found that the standards were largely ineffective and concluded that software technology is too immature to standardize (Pfleeger et al., 1994).

software for networked information systems 79

BOX 3.1

The SEI Capability Maturity Model for Software

The Software Engineering Institute's (SEI) Capability Maturity Model (CMM) for software was first introduced in the late 1980s. The current version, version 1.1, was introduced in 1993.¹ According to the SEI: "The Capability Maturity Model for Software (SW-CMM or CMM) is a model for judging the maturity of the software processes of an organization and for identifying the key practices that are required to increase the maturity of the processes. The SW-CMM is developed by the software community with stewardship by the SEI" (Paulk et al., 1993).

The CMM defines a maturity framework that has five levels: (1) initial, (2) repeatable, (3) defined, (4) managed, and (5) optimized. The five levels are carefully defined and based on key process areas (KPAs). The KPAs are, as the name suggests, the most important aspects of software processes. At CMM level 2, for example, requirements management is a KPA.

It is important to understand that the CMM is intended only to measure maturity. It is not a software development process standard or a mechanism for assessing specific software development techniques. It also is not a means of achieving high levels of either productivity or software quality (although some users report that both tend to improve after higher CMM levels have been achieved). Rather, the CMM aims to assess the ability of an organization to develop software in a repeatable and predictable way. Thus, an organization possessing a high CMM level will not necessarily develop software more quickly or of better quality than an organization having a lower level. The higher-ranked organization will, however, develop software in a more predictable way and will be able to do so repeatedly.

After a careful analysis, an assessed organization is rated at one of the five levels of the CMM framework. Attainment of some specified minimum CMM level is sometimes required to bid on certain government contracts. (The practice seems to be becoming more common within the Department of Defense.) Whether having such a minimum CMM level ensures higher-quality work is not clear, but it has succeeded in making corporate management aware of the importance of software development processes.

A second benefit of the CMM has been reported by organizations seeking to improve their ratings. The staff of such organizations become more conscious of the software technology they are using and how it can be improved. Esprit de corps tends to be generated when the entire staff is involved in a single process-improvement goal.

Although there is no specific intention that higher CMM rankings will be associated with higher quality or productivity, there is some evidence that more mature processes do yield those advantages. Watts and his colleagues document a variety of benefits and important lessons they observed at Hughes Aircraft after moving from

CMM level 2 to level 3 (Watts et al., 1991). Dion reports increased productivity and

continues on next page

Box 3.1 continued

large cost savings at Raytheon after it moved from level 2 to level 3 (Dion, 1993). And Motorola, which observed the development performance of 34 different projects with roughly equal numbers of projects rated at each CMM level (Diaz and Sligo, 1997), has reported reduced cycle time, reduced defect rates, and improved productivity as CMM level increased.

However, a recent paper by McGarry, Burke, and Decker (1997) is less favorable in discussing the correlation between CMM level and software development metrics based on data from more than 90 projects within one organization (a part of Computer Sciences Corporation). The results of the study were mixed, and in most cases improvements were not correlated with CMM level.

Impacts of process improvement have also been surveyed. Brodman and Johnson (1996) report survey data in the form of return on investment to industry. Their results document a wide variety of benefits associated with achieving higher CMM levels. Lawlis, Flowe, and Thordahl (1995) investigated the effect of CMM level on software development cost and schedule. They found a positive correlation between CMM level and cost and schedule performance. Another survey reporting positive results of using the CMM has been published by Herbsleb and Goldenson (1996).

The actual CMM assessment process has also been studied. Kitson and Masters (1993) identify which KPAs are major factors affecting CMM ratings, thereby suggesting areas of weakness in industrial software practice.

Although many successes of the CMM have been reported, the CMM itself has also been criticized. Bollinger and McGowan (1991) raised a number of important questions about the practical benefits of an initial version of the CMM in the context of government contracting. Their concerns were mainly with the relative simplicity of the assessment process and the fact that CMM levels would be used for rating government contractors. The criticisms of Bollinger and McGowan were addressed by the developers of the CMM in Watts and Curtis (1991). More recently, Fayad and Laitinen (1997) criticized aspects of the CMM ranging from the cost of assessment to the fact that a single assessment scheme is used for organizations of all sizes. Although these criticisms have merit, they do not appear to be fundamental flaws in the CMM concept.

1The CMM of the Software Engineering Institute is available online at <http://www.sei.cmu.edu/technology/cmm.html>.

software for networked information systems 81

Barriers to Acceptance of New Software Technologies

The high costs associated with adopting new software technologies make managers less likely to do so. The concern is that, despite claimed benefits, problems might arise in using the new technology and these problems might lead to missed deadlines or budget overruns. Sticking with technology that has been used before—the conservative course—reduces the risks.

Managers' fears are well founded in many cases, as many new software technologies do not work when tried on industrial-scale problems. Things that work well in the laboratory are not guaranteed to work well in practice. All too often, laboratory assessments of software technology are based on experiences with a few small examples. The need to investigate the scaling of a new technology is common to all branches of engineering but, as already discussed, the expense of performing large-scale software experiments makes such experiments infrequent. To assess a new software technology, the technology should be observed in full-scale development efforts. Any research program that aspires to relevance should include plans for compelling demonstrations that the resultant technology is applicable to industrial-scale problems and that its benefits justify the costs of learning and applying it.

Many new software technologies are also tool-intensive. They try to improve software development practices by replacing or supplementing human effort. Testing an interactive application that employs a graphic user interface, for example, requires the manipulation of complex software structures, the management of extensive detail, and the application of sophisticated algorithms. It all could be undertaken by hand, but having computers perform as much of the work as possible is preferable. Yet, software tools are notoriously expensive to develop because, although the essence of a new idea might be relatively simple to implement, providing all the basic services that are needed for practical use is neither simple nor inexpensive. In addition, learning to use new software tools takes time. The result is one more barrier to the success of any new software technology.

Findings

1. Although achieving connectivity and providing basic services are relatively easy, providing specialized services—especially trustworthy ones—is much more difficult and is complicated by the decentralized and asynchronous nature of NISs.

2. Project management, a long-standing challenge in software development, becomes even more problematic in the context of NISs because of

their large and complex nature and the continual software changes that can erode trustworthiness.

3. Whereas a large software system cannot be developed defect free, it is possible to improve the trustworthiness of such a system by anticipating and targeting vulnerabilities. But to determine, analyze, and, most importantly, prioritize these vulnerabilities requires a good understanding of how the software interacts with the other elements of the larger system.

4. It seems clear from anecdotal evidence that using any methodical and tested technique for the capture and documentation of requirements—no matter what its shortcomings—is better than launching directly into design and implementation.

5. No notation for system-level requirements has shown sufficiently commanding advantages to become dominant.

6. System-level trustworthiness requirements typically are first characterized informally. The transformation of the informal notions into precise requirements that can be imposed on system components is difficult and often beyond the current state of the art.

7. NISs generally are developed and deployed incrementally. Thus, techniques are needed to compose subsystems in ways that contribute directly to trustworthiness.

8. There exists a widening gap between the needs of software practitioners and the problems that are being attacked by the academic research community. In most academic computer science research today, researchers are not confronting problems related to large-scale integration and students do not develop the skills and intuition necessary to develop software that not only works but also works in the context of software written by others.

9. Although systematic processes may contribute to the quality of software systems, specific processes or standards that accomplish this goal have not been demonstrated.

10. Since the investment of resources needed for a large software development project is substantial, managers are reluctant to embrace new software technologies because they entail greater risks.

Building and Acquiring Components

Component-level Requirements

It is useful to distinguish between two kinds of component-level requirements: allocated or traceable requirements, which devolve directly from system requirements, and derived requirements, which are consequences of the system architecture. In the HMO system, for example,

software for networked information systems 83

there might be an overall trustworthiness requirement that medical records must be available 24 hours a day, 7 days a week. One way to meet that need would be to replicate records on two different servers; the data management software then has the derived requirement of ensuring the consistency of the data on the two servers. The requirement is "derived" because it results not so much from an interpretation or clarification of the original trustworthiness requirement but rather from the architectural strategy—replication—being used to satisfy the trustworthiness requirement.

A common practice is to insist that all requirements at the component level be testable. That is, each requirement must be accompanied by some experiment for assessing whether that requirement is satisfied. These tests must be chosen with care because, in actual practice, cost and schedule pressures drive a development team toward making sure their component passes the tests as a first priority. If a test is not chosen carefully and described unambiguously, then a component that does not satisfy the spirit or even the letter of the actual requirements statement might be deemed acceptable.

The relationship between the requirements, which capture intent, and a test, which determines acceptance, is especially problematic for nonfunctional requirements in support of trustworthiness concerns. Continuing with the HMO medical record example, the test may check that the two copies of the medical record are synchronized within so many seconds of a change having been made, that the failure of the primary server is detected by the switchover logic within so many seconds, that switchover is accomplished in so many seconds, and so on. The problem is that the list of tests is not equivalent to the requirement being tested (i.e., availability 24 hours a day, 7 days a week). For example, the tests do not take into account simultaneous or cascading failures (e.g., primary fails while secondary is running backup, secondary fails immediately after switchover, synchronization request comes in at just the wrong time as switchover is being initiated, and so on). There are thus circumstances in which the component or subsystem will pass its tests but fail to satisfy the intent of the requirement.

Detailed, component-level requirements for user interfaces are difficult to write. So-called storyboards, which show display configurations for various inputs, outputs, and states of the system, can be hard to follow. However, the popularity of graphical user interfaces has led to the development of tools that enable designers to rapidly prototype user interfaces. Generally speaking, prototyping is sensible in requirements analysis and can even serve as an executable requirements document. But the cost of building prototypes can be high, thereby preempting other higher-payoff forms of requirements analysis. For example, devoting too

much effort to prototyping a user interface can lead to software in which an elaborate user interface surrounds a poorly thought-out core.

Component Design and Implementation

To project managers, component design and implementation are the least visible of the phases. A large number of activities are proceeding in parallel, the staff are focused on their individual tasks (perhaps ignoring the global view), and the tasks themselves are highly technical. All conspire to make measuring progress or even anecdotal observations of status extremely difficult. While there is an extensive literature on the problem of demonstrating that a component satisfies its specification, there is considerably less literature devoted to determining whether a component-level specification properly reflects or contributes toward satisfying system requirements.

For code written in traditional languages (such as C) running on a single node, and interacting in limited and controlled ways with users and other software, the craft of programming has evolved into a generally accepted process. As practiced within the aerospace, defense, and other large-scale computing system development communities (but not necessarily in commercial practice) over the last two decades, that process consists of roughly the following steps:

• Review the component requirements document for sanity.

• Prepare a component design in some notation, often called "pseudocode." (Pseudocode is usually a mixture of programming language statements and some less detailed notation, not excluding natural language.)

• Conduct an organized inspection of the component design ("structured walkthrough") with an emphasis on the logic flow.

• Write component test scripts or test drivers to exercise the component after it has been written.

• Write the component in some appropriate higher-level language ("source code").⁸

• Conduct a structured walkthrough of the source code.

• Compile the component into executable form.

⁸This and the preceding step are often reversed, and the test drivers are not written until after the component is. The order given in the text is preferable because the detailed design and coding of a test driver force implementers to rigorously analyze and understand component-level requirements.

software for networked information systems 85

• Exercise the component ("unit test" or "level 1 testing") using the test scripts or drivers.

• Release the component to the integration process.

This process, and ones like it, have been synthesized from the wreckage of expensive failures, and a significant percentage, if not a majority, of experienced practitioners would caution that any of these steps are omitted at one's peril. One variation is to repeat the cycle frequently, making very small changes at each iteration. This approach was used successfully in the Multics project (Clingen and van Vleck, 1978) and has long been part of the program management lore in high-consequence real-time systems.

Today's turnover rate among software personnel somewhat reduces the effectiveness of the component-development process just described. Software development is still typically learned through apprenticeship. Yet personnel shortages, the potential financial rewards and short life cycles of start-up companies, and the deterioration of corporate loyalty as a result of downsizing and restructuring make it less likely that a junior practitioner will witness a complete project life cycle, much less several projects conducted in the same organization. Ultimately, this will impede the development of an adequate skill base in critical areas, like synthesis and analysis of design, integration, or structuring of development organizations.

The above component development process is predicated on starting with a modular design. Achieving modularity is intellectually challenging and costly; it requires management and design discipline. In addition, modular systems often are larger and slower. So there is a tension between system modularity and cost (along a variety of cost dimensions); it can be hard to know when system modularity is needed and when it is not worth the cost. Moreover, certain NIS building blocks—mobile code and Web browsers with helper applications, for example—compromise the advantages of modular design by permitting unrestricted interactions between different software components.

Programming Languages

Modern programming languages, such as C++, Java, and Ada, include compile-time checks to detect a wide range of possible errors. The checks are based on declaring or inferring a type for each object (i.e., variables and procedures) and analyzing the program to establish that objects are used in ways consistent with their types. This kind of automated support is especially helpful for detecting the kinds of errors (such as passing arguments that overflow a corresponding parameter) so successfully used

by attackers of operating system and network software. Ever more expressive type systems are a continuing theme in programming language research, with considerable attention being directed recently at the representation of security properties using types (Digital Equipment Corporation, 1997). Success would mean that compile-time checks could play an even bigger role in supporting trustworthiness properties.

Modern programming languages also contain features to support modularity and component integration. Ada, for example, provides type checking across separate compilations; Ada also integrates component linking with compilation, so that statements whose validity depends on the order in which compilation occurs can be checked. Other modern languages provide equivalent features. At the other end of the spectrum, scripting languages (Ousterhout, 1998) (such as Visual Basic and TCL) are today attracting ever-larger user communities. These languages are typically typeless and designed to facilitate gluing together software components. The preponderance of COTS and legacy components in a typical networked information system assures the relevance of scripting languages to the enterprise.

Also of interest to NIS developers are very-high-level languages and domain-specific languages, which provide far-higher-level programming abstractions than traditional programming languages do. The presence of the higher-level abstractions enables rapid development of smaller, albeit often less efficient, programs. Moreover, programming with abstractions that have rich semantics and powerful operations reduces the opportunity for programming errors and permits more sophisticated compile-time checking.

There is much anecdotal and little hard, experimental evidence concerning whether the choice of programming language can enhance trustworthiness. One report (CSTB, 1997) looked for hard evidence but found essentially none. Further study is needed and, if undertaken, could be used to inform research directions in the programming language community.

Systematic Reuse

Systematic reuse refers to the design and implementation of components specifically intended for instantiation in differing systems. It is one of the most sought-after goals in software research, because it offers the potential for substantial software productivity improvements.⁹ More

⁹It is worth noting that the infamous year 2000 problem would be far easier to address if a small number of date packages had been reused in date-sensitive applications. There would still be the problem of database conversion, though, once the date format is changed.

software for networked information systems 87

over, components intended for reuse can be more intensely scrutinized, since the higher cost of analysis can be amortized over multiple uses. The current economic emphasis on short-term results, however, serves to inhibit the acceptance of any method of systematic reuse that requires (as appears inevitable) up-front investment.

Certain commercial vendors, such as SAP, whose R/3 enterprise-applications software (Hernandez, 1997) has captured one-third of the worldwide client-server market for business systems, claim to have solved the systematic reuse problem in a cost-effective manner for large classes of applications. R/3 is an integrated software package that includes interwoven reusable components for all the major functions of a commercial enterprise, from order entry and accounting through manufacturing and human resources. In addition, R/3 is built to use a COTS operating system along with COTS database management systems, browsers, and user-interface software. Other commercially driven attempts at providing components or infrastructure for systematic reuse include the C++ standard template library (STL) (Musser and Saini, 1996), common object request broker architecture (CORBA),¹⁰ common object model (COM) (Microsoft Corporation and Digital Equipment Corporation, 1995), distributed common object model (DCOM) (Brown and Kindel, 1998), and JavaBeans (Hamilton, 1997).

There is always a tension between the pressure to innovate and the stability associated with components intended for reuse. That tension is particularly acute for COTS components, for which the addition of new features and time to market are such strong forces. New features are usually accompanied by new bugs; careful analysis of components enhances stability but delays product release. Moreover, when bugs in COTS components do get fixed, the fixes are often bundled in a release that also introduces new features. The COTS component user must then choose between living with a bug and migrating to a release that may be less stable due to new bugs.

Commercial Off-the-Shelf Software

The Changing Role of COTS Software

Success for a COTS software component often leads to deployment in settings never intended. A component might start as an interesting piece of software at the periphery of trustworthiness concerns and ultimately

¹⁰COBRA 3.0 was introduced by the Object Management Group (OMG) in December, 1994. Additional information is available online at <http://www.omg.org/>.

become a critical component in some NIS. In 1994, it would have been absurd to suggest that a bug in a Web browser could kill someone. Yet in the HMO system we are using as an example, a Web-based telemedicine application could allow precisely that outcome. That software can be used for tasks not envisioned by its developers is a double-edged sword, especially if COTS development practices cause developers to compromise trustworthiness for other requirements.

COTS software development practices in the personal computer (PC) era arose in a technical and economic environment that tended to ignore trustworthiness. PC operating systems and applications ran on isolated desktops; the consequences of failure were limited to destruction of perhaps valuable, but certainly not life-critical, data. Failures had no way of propagating to other machines. Therefore, an organizational and programming culture arose that was very accepting of errors and malfunctions, epitomized by the notorious shrink-wrap license whose primary feature is a total disclaimer of responsibility by the developer.

This climate was amplified by economic conditions of the early PC era. Software was purchased separately rather than being bundled with a leased computer, as in the mainframe era. Consequently, there was less financial leverage for dissatisfied customers to affect vendor, and therefore developer, attitudes. A customer's financial leverage was limited to consuming vendor resources in calls to telephone help-lines, which could be ignored by inept or uncaring vendors,¹¹ and refusing to purchase other software or the next revision of the malfunctioning product from that vendor. The latter option is reduced by the diminishing diversity of the marketplace, the need to exchange data with other users, and the investment the customer may have in data that can be processed only by the product in question.

As the PC market exploded, visionary entrepreneurs realized that market share was the dominant factor in corporate survival and personal financial success. Market share is heavily influenced by market entry time. Specifically, the first product to reach a market has the greatest opportunity both to gain market share and to establish the de facto standard upon which the software industry currently operates. Another influence on market share is the richness of features and user interface, which impresses users and reviewers in the technical press. Something must be sacrificed, and it has been trustworthiness aspects such as robustness and security.

One way to reduce time to market is to reduce the time spent in

¹¹This situation is changing. A vendor, albeit of hardware, has recently settled a class action suit requiring an increase in warranty and support coverage (Manes, 1998). Similar actions against software vendors are likely to follow from this precedent.

software for networked information systems 89

testing. By making early releases (beta test versions) available to interested users and by freely distributing incremental updates to production software, vendors enlist the help of the user community in finding errors. From a societal perspective, the PC software industry's attitude toward errors was relatively unimportant, since the worst consequence of PC software errors was the time lost by individuals trying to reconstruct destroyed work or otherwise get their PCs to do their bidding. But today, COTS software is moving toward being a business of providing components—and possibly critical components—for NISs that can be high consequence, either because they were explicitly designed that way or because people assign to them a level of trust that their designers never intended.

General Problems with COTS Components

The use of COTS components presents special problems for the responsible developer of an NIS. COTS software typically is full of features that vary in quality and are a source of complexity. The complexity, in turn, means that specifications for COTS components are likely to be incomplete, and users of those components will discover features by experimentation. Being conservative in exploiting these discoveries is prudent—semantics not documented in an accompanying written specification may or may not have been intended and consequently may or may not persist across releases. Moreover, wise developers learn to avoid the more complex features of COTS components because these are the most likely to exhibit surprising behavior and their behavior is least likely to remain stable across releases. When these features cannot be avoided, encapsulating components with wrappers, effectively narrowing their interfaces, can protect against undesirable behaviors.

The COTS developer's reliance on customer feedback¹² as a significant, or even primary, quality assurance mechanism can lead to uneven quality levels in different subsystems or functionality in a single COTS product. Press coverage is not guaranteed to be accurate and may not convey the implications of the problem being reported.¹³ For example, security vulnerabilities in components such as Web browsers, which are

¹²Handling calls to customer-support telephone help-lines is sometimes claimed to be a significant portion of COTS software costs. The committee was unable to explore the veracity of this claim. However, the use of customer feedback in place of other quality control mechanisms does allow a software producer to externalize costs associated with product testing.

¹³See, for example, the February 1997 coverage of the Chaos Computer Club demonstration of a supposed security flaw in Microsoft's Internet Explorer.

used directly by the public, receive widespread coverage, as do ultimately inconsequential (and unsurprising) exploits, such as the use of large numbers of machines on the Internet to "break" cryptographic algorithms by brute-force searches. Feedback from customers and the press, by its very nature, occurs only after a product has been distributed. And experience with distribution of bug fixes clearly indicates that many sites do not, for a variety of reasons, install such upgrades, thereby leaving themselves vulnerable to attack through the now highly publicized methods.¹⁴ Reliance on market forces to select what gets examined and what gets fixed is haphazard at best and is surely not equivalent to performing a methodical search for vulnerabilities prior to distribution.

Finally, using COTS software in an NIS has the advantages and disadvantages that accompany any form of "outsourcing." COTS components can offer rich functionality and may be better engineered and tested than would be cost-effective for components developed from scratch for a relatively smaller user community. But an NIS that uses COTS components becomes dependent on a third party for decisions about a component's evolution and the engineering processes used in its construction (notably regarding assurance). In addition, the NIS developer must track new releases of those COTS components and may be forced to make periodic changes to the NIS in response to those new releases. It all comes down to a trade-off between cost and risk: the price of COTS components can be attractive, especially if the functionality they provide is a good match for what is needed, but the risk of ceding control may or may not be sensible for any given piece of an NIS.

Interfacing Legacy Software

Legacy software refers to existing components or subsystems that must be retained and integrated more or less unchanged into a system. Legacy software is used when developing an NIS because reusing an existing system is cheaper and less risky than completely reimplementing it, especially given the migration costs (training, rebuilding online records) associated with deploying a replacement system. In the HMO example, it would be very likely that the Clinical Laboratory or Pathology departments had been operating for decades with freestanding computerized systems. Incorporating such a freestanding system into an NIS poses special problems:

¹⁴Even when administrators diligently apply security-bug fixes, the fixes can then be lost when a crashed system is restored from backup media. Since such restorations are often done in a crisis atmosphere, the need to perform the additional update step is easily overlooked in the rush to restore service.

software for networked information systems 91

• NIS designers must recognize that they might be dealing with an operational subsystem that performs critical functions and cannot be rendered inactive for days or even hours.

• The legacy subsystem might not have been designed with networking in mind, and conversion to one that supports networking might not be feasible.

• The system may be an "orphan" product whose vendor no longer exists or supports this version. Or, if the system was developed locally, documentation and expertise about its internals might have evaporated over time.

The general approach to dealing with these problems is to fool some interface of the legacy system into thinking it is operating in isolation when, in fact, it is connected to a network. Often, an existing interface of the legacy system can be wrapped in a new layer of software (called a wrapper) that hides the network, perhaps by making the network look to the legacy software like an existing user interface (e.g., a keyboard and display). And a legacy system might be adapted to use a new communications protocol in place of an old one by writing software that uses the old protocol to simulate the functionality of the new one; this is called tunneling. The risk with such schemes is that the legacy system's interface, designed to serve one type of client, might not be able to handle the characteristics of the new load. For example, the volume of transactions arriving over the network might overwhelm an interface that was written to serve a single human user typing at a terminal. Inadequate or incomplete documentation for a legacy system's interfaces also can complicate employing the approach.

Findings

1. It is difficult to devise component-level acceptance tests that fully capture the intent of requirements statements. This is particularly true for nonfunctional and user interface requirements.

2. High turnover of programming staff is impeding the development of an adequate skill base in critical areas, such as NIS synthesis and analysis of design, integration, or structuring.

3. There are some accepted processes for component design and implementation. However, the performance needs of NISs can be inconsistent with modular design, and this fact can limit the applicability of an effective design tool to NIS design.

4. Modern programming languages include features, such as compile-time checks and support for modularity and component integration, that promote trustworthiness. The potential may exist for further gains

by developing even more expressive type systems and other compile-time analysis techniques.

5. There is inadequate experimental evidence to justify the utility of any specific programming language or language feature with respect to improving trustworthiness.

6. Despite theoretical concerns,¹⁵ as a practical matter the use of higher-level languages increases trustworthiness to a degree that outweighs the risks.

7. Basing the development of an NIS on libraries of reusable trusted components and using those components in critical areas of the system can provide a cost-effective way to implement component-level dimensions of trustworthiness.

8. New commercial software that includes usable components or infrastructure for systematic reuse is increasingly available, but it is too early to know how successful it will be.

9. COTS software originally evolved in a stand-alone environment in which trustworthiness was not a primary concern. Furthermore, market pressures contribute to reducing time spent on testing before releasing software to users, while emphasizing features that add to complexity but are useful for only a minority of applications.

10. COTS software offers both advantages and disadvantages to an NIS. COTS components may be less expensive, have greater functionality, and be better engineered and tested than is feasible for customized components. Yet the use of COTS makes developers dependent on outside vendors for the design and enhancement of important components; specifications may be incomplete and may compel users to discover features by experimentation.

11. Incorporating legacy software into an NIS poses risks for trustworthiness because problems may arise as a result of including a previously freestanding system into a networked environment for which it was unintended.

Integrating Components into
a Trustworthy System

System Integration

Subsystem integration is the orderly aggregation of unit-tested components into a subsystem, along with incremental testing to increase confidence in the subsystem's correctness. There are three basic approaches:

¹⁵For example, a theoretically effective attack based on a maliciously modified compiler was described over a decade ago in Thompson (1984).

software for networked information systems 93

bottom-up integration, top-down integration, and thread integration. To illustrate, consider the Clinical Laboratory subsystem for our HMO's NIS. Lower-level components in the subsystem would control the keyboard and display, maintain local data files, and control interactions with test instruments; upper-level management components would select which of the lower-level ones are activated and in what order.

In bottom-up integration, a series of programs (called test drivers) is written that simulates the upper-level components of the subsystem. The lower-level components (e.g., the ones that control test instruments in the Clinical Laboratory subsystem) are aggregated first, and only when their correct interactions have been observed are the upper-level components added. The origin of the name "bottom up" should be clear. The approach was popular in the early days of real-time control systems. Computer memory was a scarce resource then, and the integration team obtained an early warning of excessive software size by proceeding from the bottom up. Bottom-up integration carries with it the significant disadvantage that the overall logical operation of the subsystem is observed only relatively late in the process, when limited time and resources are available to deal with incorrect behavior.

The opposite of bottom-up integration is top-down integration. In this approach, upper-level components are integrated first. The components are tested using routines (called stubs) that simulate the behavior of the lower-level components. The stubs are then replaced one by one with the components that they are simulating. With top-down integration, logical correctness of the subsystem is established first, but the actual size of the entire system is not determined until relatively late in the integration process. Thus, if system size is not an issue, top-down integration is superior to bottom-up integration; if size is an issue, then with top-down integration, failure would likely be due to size problems rather than incorrect logical operation of the system.

In both top-down and bottom-up integration, confidence in correct behavior is gained through the use of simulated rather than actual components; stubs are used in top-down integration, and test drivers are used in bottom-up integration. Clearly, the use of the actual components would be preferable, so software developers devised a more sophisticated approach known as thread integration or thread testing.

In thread integration, the components being joined are selected subsets of the overall subsystem, and test cases are carefully defined to activate only the subset of components under test. There are two ways to select a subset of components to integrate. One is to select a subset of the system-level requirements. This works when the requirements map onto the top-level design in a straightforward manner. The second and more common approach is to select subsets of components according to the

top-level design and the sequence of component activations (the call tree).¹⁶

As an example, a single build in a thread integration of our Clinical Laboratory subsystem might combine the keyboard and display component, the management component, and an interface to a single test instrument (say, for blood sugar). A thread test of this build would involve an operator sitting at the console and initiating a blood sugar test; the fact that, say, the hepatitis antibody test components are not yet integrated does not matter, since these components would not be activated by the test.¹⁷ When all the builds are complete, confidence is increased that the components not only work properly in isolation (which is the concern of unit testing) but also work together.

In traditional software development, the word "subsystem" in the preceding discussion could be replaced by the word "system." Once the integration of a single node was complete, the job was done. However, the structure of an NIS adds another level to the integration process. Disparate nodes in a network must interact to perform a single, coordinated task. Relatively little is known about approaches to performing this additional level of integration compared with what is known about subsystem integration. By their very nature, networks pose special problems to an integration team. For one thing, inputs may have to be submitted miles from where corresponding outputs must be observed. For another, system behavior might be load dependent, but operational loads are very hard to simulate (notwithstanding various efforts over many years). In fact, when public networks are being used, various aspects of network behavior become uncontrollable, which means certain tests might not be possible and others might not be repeatable.

System Assurance

Review and Inspection

One commonly used technique for improving software quality is to undertake technical reviews, sometimes known as inspections (Fagan, 1986), in which objective critics examine a design or artifact in detail. A subsequent meeting of the critics allows discussion of specific defects that their examinations have revealed; the meeting also facilitates brainstorming about more systemic flaws that were observed. A great deal of effort

¹⁶The "thread of control"—hence the name of the technique.

¹⁷In actual use, stubs are incorporated to raise alarms if the decision-making component activates the wrong thing.

software for networked information systems 95

has gone into studying various types of technical reviews and various ways of organizing them, and much is known about the benefits of the approaches (Porter et al., 1997), yet their utility in security is not well documented. For example, no evidence could be identified to confirm whether traditional forms of technical reviews could facilitate the detection of security vulnerabilities in an implementation.¹⁸ A simple checklist-based review might be helpful for eliminating well-known vulnerabilities, such as failure to validate arguments, but the overall impact of this activity on trustworthiness properties has not been determined and should be studied. It might also be possible to employ technical reviews in order to identify assumptions being made by designers of a system__assumptions that can become vulnerabilities should an attacker cause them to be violated.

Formal Methods

Formal methods is the name given to a broad class of mathematically based techniques for the description and analysis of hardware, software, and entire computing systems. The descriptions may range, on the one hand, from general statements about desirable system properties, as might be found in a requirements document or high-level specification, to, on the other hand, detailed depictions of intended behavior for specific pieces of software or hardware. The analyses enable developers to derive and check whether specific properties are implied by the formal descriptions.

A system developer, for example, might employ a formal method to check whether a description of requirements is sensible (i.e., not contradictory, unambiguous, and complete) or simply implies some specific property of interest, like (for the HMO system example) "at any time, at most one surgery is scheduled for a given operating theater." Or, for a program text or a more abstract description of an algorithm (viz., any detailed description of behaviors), a formal method could be used to establish that some general condition on execution holds, like "variables and arguments declared with type integer are only assigned values that are integers," or that some specific characterization of behavior is entailed, like "messages sent using the network are delivered uncorrupted and are not reordered."

Formal methods attempt to extend the capabilities of developers by eliminating the need for exhaustive case analyses and/or by facilitating

¹⁸The emphasis of "red teaming," "vulnerability assessment," and "penetration testing" is to focus on selected areas in which intuition, experience, or other evidence indicates that problems may arise (Weissman, 1995). This contrasts with technical reviews as discussed in Fagan (1986), which seek to examine all logical paths in a component.

the construction of long and intricate arguments, so that some property of interest can be certified for a given (formal) description. They are most effective when the property of interest is subtle but can be rigorously defined and when either the description of the object being analyzed is relatively small or the formal method being used supports analyses that can be automated.

Formal methods, however, are useful only when the developer can pose the right questions. For example, establishing that a system implements multilevel security using mandatory access control, whether by formal methods or any other means, does not imply the absence of security vulnerabilities in that system, nor does it imply that the resulting system is capable of performing useful computation. Moreover, some properties (e.g., "the absence of security vulnerabilities") have no system-independent formalization and, therefore, are not amenable to direct analysis using formal methods.¹⁹

Growth in cost-effective desktop computing power continues to move the field of formal methods toward computer-aided and fully mechanized formal methods from more manual ones. A second significant force has been the need to build confidence when programming ever richer system behaviors (involving time, other physical processes, fault-tolerance, security) as well as when using complex programming constructs (for parallel and distributed systems, object orientation, and so on).

Early work in formal methods emphasized logics and theorem proving. A practitioner constructed proofs largely by hand, with automated assistance limited to proof checking and the synthesis of low-level inferences. The inability to construct a proof could signify a flaw in the implementation being analyzed, but it could equally well reflect insufficient creativity by the person attempting the proof. More recently, with model checking, raw computing cycles have replaced the manual construction of proofs. Model checking always terminates, reporting that the implementation satisfies the given specification or giving a scenario that shows inconsistency of the implementation with the specification. Inherently limited to systems having finite-sized state spaces, today it is possible to apply model checking to systems having upwards of 200 state variables and 10¹²⁰ states (making the approach powerful enough for industrial use in hardware design); ongoing research into abstraction techniques continues to push the limits ever higher.

¹⁹For any given system, there will exist properties that together imply "the absence of security vulnerabilities." But careful thought by a system developer is required to identify these constituents, and there is no formal way to ever establish that the system developer has listed them all.

software for networked information systems 97

Formal methods are being used increasingly in commercial and industrial settings.²⁰ Hardware efforts have provided the most visible
successes so far, perhaps because specifications for hardware tend to be relatively stable, the specifications are short relative to the size of implementations, there is agreement on the choice of languages for writing specifications, and the cost of design flaws in chips is very high. Examples of successes include the following:

• Intel has used formal methods in the development of its P5 processor (Pentium processor) and P6 processor (Pentium Pro processor) to prove that the hardware implements the required functionality specified at the register transfer level and to prove that hardware realizations of some of the more complex protocols correctly implement higher-level specifications.²¹

• A tool called Verity has been used widely within IBM for designing processors, including the PowerPC and System/390 (Kuehlmann et al., 1995).

• Model checkers have enabled bugs to be found in the IEEE Futurebus+ standard (Clarke et al., 1993) and the IEEE scalable coherent interface (SCI) standard (Dill et al., 1992).

• The ACL2 theorem prover was used to find bugs in the floating-point square-root microcode for the AMD5K86 processor as well as to find pipeline hazards in the Motorola complex arithmetic processor (CAP), a digital signal processor intended for use in a secure multimode joint-service programmable radio (Brock et al., 1996).

• The microarchitecture and fragments of the microcode for the Collins AAMP5 (Srivas and Miller, 1995) and AAMP-FV avionics processors were analyzed using SRI's PVS theorem prover (Owre et al., 1995).

Commercial and industrial software efforts have also benefited from formal methods. Formal methods applied to requirements analysis has led to some of the more visible of these industrial successes. By formulating requirements in a language having unambiguous semantics, developers can better understand requirements and can use automated tools to discover ambiguity, inconsistency, and incompleteness. The entire set of requirements need not be formalized to enjoy the benefits—often, the most cost-effective course is to treat a carefully chosen subset (with only those elements of concern present). The intricate or novel aspects of the

²⁰See Clarke and Wing (1996), Dill and Rushby (1996), Rushby (1995), and Craigen et al. (1993) or its summary (Craigen et al., 1995) for the many more examples and details than can be given here.

²¹Based on a telephone interview with Gadi Singer, General Manager of Design Technology, Intel Corporation, on June 8, 1998.

requirements are thereby checked without formalizing an entire set of requirements, which, as observed above in the section on system-level requirements, is likely to be neither complete nor stable. Some of the better-known successful industrial uses of formal methods for analyzing requirements include these:²²

• With the Software Cost Reduction (SCR) program's tool suite, engineers at Rockwell were able to detect 24 errors—many of them significant—in the requirements specification for a commercial flight guidance system (Miller, 1998). Also using the SCR tool suite, Lockheed engineers formalized the operational flight program for the C-130J Hercules aircraft and found six errors in nondeterminism and numerous type errors.²³

• An informal English specification for the widely deployed aircraft collision avoidance system TCAS II was abandoned for a formal version written in requirements state machine language (RSML) after the English specification was deemed too complex and unwieldy. That formal specification has since been mechanically checked for completeness and consistency (Heimdahl and Leveson, 1996).

Formal methods were originally developed as an alternative to exhaustive testing for increasing one's confidence that a piece of software satisfies a detailed behavioral specification. To date, this use for formal methods has been applied outside the laboratory only for relatively small safety-critical or high-consequence computing systems, for which development cost is not really a concern but flaws are. Examples include the verification of safety-critical software used in the Hercules Cl30 aircraft (Croxford and Sutton, 1995), parts of the next-generation command-and-control ground system for the ARIANE rocket launcher (Devauchelle et al., 1997), and highly secure operating systems (Saydjari et al., 1989).

Constructing extremely large proofs is infeasible today and for the foreseeable future, so formal methods requiring the construction of proofs for an entire system are not practical when developing an NIS having tens to hundreds of millions of lines of code. Even if size were not an issue, COTS components are rarely accompanied by the formal specifications necessary for doing formal verification of an NIS built from COTS components. It would be wrong, however, to conclude that formal verification cannot contribute to the construction of an NIS.

²²In addition to Clarke and Wing (1996) and Craigen et al. (1993), further examples of this use of formal methods appear in Easterbrok et al. (1998).

²³As Connie Heitmeyer, U.S. Naval Research Laboratory, described at the NRC's Information Systems Trustworthiness Committee workshop, Irvine, CA, February 5-6, 1997.

software for networked information systems 99

For one thing, critical components of an NIS can be subject to formal verification, thereby reducing the number of flaws having system-disabling impact.²⁴ The aircraft hand-off protocol (Marzullo et al., 1994) in the Advanced Automation Systems air-traffic control system built by IBM Federal Systems Division illustrates such an application of formal methods. Second, entire (large) systems can be subject to formal verification of properties that are checkable mechanically. This is the impetus for recent interest by the software engineering community in so-called lightweight formal methods, like the LCLint tool, which is able to check C programs for a variety of variable type and use errors (Detlefs, 1996), and Eraser, a tool for detecting data races in lock-based multithreaded programs (Savage et al., 1997).

Size problems can be circumvented by subjecting a model of the NIS to analysis instead of analyzing the entire NIS. The model might be smaller than the original in some key dimension, as when confidence is built in a memory cache-controller by analyzing a version that handles only a small number of cache-lines. Alternatively, a model might be smaller than the original by virtue of the details it ignores—checking a high-level description of an algorithm or architecture rather than checking its implementation in a real programming language. Illustrative of this latter approach are the various logics and tools for checking high-level descriptions of cryptographic protocols (Burrows et al., 1990; Lowe and Roscoe, 1997; Meadows, 1992). For instance, with a logic of authentication (Burrows et al., 1990), successive drafts of the CCITT X.509 standard were analyzed and bugs were found, including a vulnerability to replay attacks even when keys have not been compromised.

Observe that a great deal of benefit can be derived from formal methods without committing a project to the use of formal notations either for baseline specifications or throughout. Some argue that formal methods analyses are more effective when performed later, to shake out those last few bugs, rather than earlier, when less costly techniques can still bear fruit.

A well-documented example of industrial use of formal methods in building an NIS was the development by Praxis of the CCF display information system (CDIS) component of the central control function (CCF) air traffic management subsystem in the United Kingdom (Hall, 1996).²⁵ Here, various formal methods were used at different stages of the development

²⁴At least for those properties that can be described formally.

²⁵This system involved 100 processors linked by using dual local area networks and consisted of approximately 197,000 lines of C code (excluding comments), a specification document of approximately 1,200 pages, and a design document of approximately 3,000 pages.

process: VDM (Jones, 1986) was used during requirements analysis, VVSL (Middleburg, 1989) was used for writing a formal specification for the system, and CSP (Hoare, 1985) was used for describing concurrency in CDIS and its environment. With automated assistance, proofs of correctness were constructed for a few critical protocols. And Hall (1996) reports that productivity for the project was the same or better than has been measured on comparable projects that used only informal methods. Moreover, the defect rate for the delivered software was between two and ten times better than has been reported for comparable software in air traffic control applications that did not use formal methods.

Beyond the successful industrial uses of formal methods discussed above and in the work cited, there are other indications that formal methods have come of age. Today, companies are marketing formal verification tools for use in hardware design and synthesis.²⁶ And there are anecdotal reports that the number of doctoral graduates in mechanized formal methods is now insufficient to fill the current demands of industry.²⁷

Although once there was a belief that the deployment of formal methods required educating the entire development team, most actual deployments have simply augmented a development team with formal methods experts. The job of these experts was beautifully characterized by J S. Moore:²⁸

Like a police SWAT team, members are trained in the use of "special weapons," in particular, mathematical analysis tools. But they are also extremely good at listening, reading between the lines, filling in gaps, generalizing, expressing precisely the ideas of other people, explaining formalisms, etc. Their role is not to bully or take credit, but to formalize a computing system at an appropriate level of abstraction so that certain behaviors can be analyzed.

Here, the absence of shared application assumptions with the development team actually benefits the formal methods expert by facilitating the discovery of unstated assumptions.

Formal methods are gaining acceptance and producing results for industry. What are the impediments to getting broader use and even

²⁶Examples include Formal Check from Lucent Technologies, RuleBase from IBM Corporation, VFormal from Compass, and Checkoff from View Logic.

²⁷As John Rushby described at the NRC's Information Systems Trustworthiness Committee workshop, Irvine, CA, February 5-6, 1997.

²⁸Position statement on the state of formal methods technology submitted for the committee's workshop held on February 5-6, 1997, in Irvine, CA. Moore credits Carl Pixley of Motorola with the SWAT-team simile.

software for networked information systems 101

further leverage from formal methods? With minor exceptions (Taylor, 1989), the formal methods and testing communities have worked independently of each other, to the advantage of neither. Also, the need for better-integrated tools has been articulated by researchers and formal methods practitioners alike (Craigen et al., 1993), and research efforts are now being directed toward combining, for example, model checkers and proof checkers. Another trend is the development of editors and library support for managing larger proofs and for facilitating development of reusable models and theories.

Over the last decade, formal methods researchers survived only by devoting a significant fraction of their effort to performing realistic demonstration exercises (and these have helped to move formal methods from the research laboratory into industrial settings). More-fundamental research should be a priority. Significant classes of properties remain difficult or impossible to analyze, with fault-tolerance and security high on the list. Methods for decomposing a global property into local ones (which could then be checked more easily) would provide a basis for attacking limitations that bar some uses of formal methods today.

Finally, there is a growing collection of pragmatic questions about the use of formal methods. A key to building usable models of NISs is knowing what dimensions can be safely ignored. Answering that question will require a better understanding about the role of approximation and of simplifying assumptions in formal reasoning. Frictionless planes have served mechanical engineers well—what are the analogous abstractions for computing systems in general and NISs in particular? Idealized models of arithmetic, for example, can give misleading results about real computations, which have access only to finite-precision fixed or floating-point arithmetic. And any assumption that might be invalidated constitutes a system vulnerability, so analysis predicated on assumptions will be blind to certain system vulnerabilities.

There are also questions about the application of formal methods: Where can they give the greatest leverage during system development? When does adding details to a model become an exercise in diminishing returns, given that most errors in requirements and specification are errors of omission (and therefore are likely to be caught only as details are added)? And—a question that is intimately linked to the problem of identifying and characterizing threats—How does one gain confidence that a formal specification is accurate?

Testing

Testing is a highly visible process; it provides confidence that a system will operate correctly, because the system is seen to be operating

correctly during testing. And industry today relies heavily on testing. Unfortunately, most real systems have inputs that can take on large numbers of possible values. Testing all combinations of the input values is impossible. (This is especially problematic for systems employing graphical user interfaces, where the number of possible point-and-click combinations is unworkably large.) So, in practice, only a subset of all possible test cases is checked, and testing rarely yields any quantifiable information about the trustworthiness of a program. The characteristics of networked information systems—geographic distribution of inputs and outputs, uncontrollable and unmonitorable subsystems (e.g., networks and legacy systems), and large numbers of inputs—make this class of system especially sensitive to the inadequacy of testing only subsets of the input space.

Much of the research in testing has been directed at dealing with problems of scale. The goal has been to maximize the knowledge gained about a component or subsystem while minimizing the number of test cases required. Approaches based on statistical sampling of the input space have been shown to be infeasible if the goal is to demonstrate ultrahigh levels of dependability (Butler and Finelli, 1993), and approaches based on coverage measures do not provide quantification of useful metrics such as mean time to failure. The result is that, in industry, testing is all too often defined to be complete when budget limits are reached, arbitrary milestones are passed, or defect detection rates drop below some threshold. There is clearly room for research—especially to deal with the new complications that NISs bring to the problem: uncontrollable and unobservable subsystems.

System Evolution

Software systems typically are modified after their initial deployment to correct defects, to permit the use of new hardware, and to provide new services. Accommodating such evolution is difficult. Unless great care is taken, the changes can cause the system structure to degenerate. That, in turn, can lead to new defects being introduced with each subsequent change, since a poorly structured system is both difficult to understand and difficult to modify. In addition, coping with system evolution requires managing the operational transition to new versions of that system. System upgrade, as this is called, frequently leads to unexpected difficulties, despite extensive testing of the new version before the upgrade. In some cases, withdrawal of the new system once it has been introduced is a formidable problem, because data formats and file con

software for networked information systems 103

tents have already changed. The popular press is full of incidents in which system failures are attributed to system upgrades gone awry.

New facilities can be added to an NIS, and especially a Web-based NIS, with deceptive ease: a new server that provides the desired service is connected to the network. However, such action can affect performance and reliability. The dispersed nature of an NIS user community can make it difficult to gauge the impact of new features. And the lack of quality-of-service controls can make one NIS a hostage to changes in the load or features in another.

Another potential area of difficulty for NIS evolution is having critical COTS components change or be rendered obsolete. The advent of so-called "push" technology, in which commercial off-the-shelf software is silently and automatically updated when the user visits the vendor's Web site, can cause COTS components to drift away from the configuration that existed during test and acceptance; the situation leads to obscure and difficult-to-locate errors.

Findings

1. Very little is known about the integration of subsystems into an NIS. Yet methods for network integration are critical for building an NIS. NISs pose new challenges for integration because of their distributed nature and the variability of network behavior.

2. Even though technical reviews are generally considered by the practitioner community to be effective, the utility of technical reviews for establishing trustworthiness properties is not well documented.

3. Formal methods are most effective when the property of interest is subtle but can be rigorously defined, and when either the description of the object being analyzed is relatively small or the formal method being used supports analyses that can be automated.

4. Formal methods are moving from more manual methods toward computer-aided and fully mechanized approaches.

5. Formal methods are being used with success in commercial and industrial settings for hardware development and requirements analysis and with some success for software development.

6. Formal methods should be regarded as but one piece of technology for eliminating design errors in hardware and software. Formal methods are particularly well suited for identifying errors that become apparent only in scenarios not likely to be tested or testable.

7. Fundamental research problems in formal methods should not be neglected in favor of demonstration exercises. Research progress in core

areas will provide a basis for making significant advances in the capabilities of the technology.

8. Although the large size of an NIS and the use of COTS limit the use of formal methods for analyzing the entire system, formal verification can still contribute to the development process.

9. Testing subsets of a system does not adequately establish confidence in an NIS given its distributed nature and uncontrollable and unobservable subsystems.

10. Research in testing that addresses issues of scale and concurrency is needed.

11. Postdeployment modification of software can have a significant negative impact on NIS trustworthiness and security.

12. Research directed at better integration of testing and formal methods is likely to have payoffs for increasing assurance in trustworthy NISs.

References

Abadi, Martin, and Leslie Lamport. 1993. "Composing Specifications," ACM Transactions on Programming Languages and Systems, 15(1):73-132.

Boehm, B. 1981. Software Engineering Economics. Englewood Cliffs, NJ: Prentice-Hall International.

Boehm, B. 1988. "A Spiral Model of Software Development and Enhancement," IEEE Computer, 21(5):61-72.

Boehm, B., and T. DeMarco. 1997. "Software Risk Management," IEEE Software, 14(3):17-19.

Bollinger, Terry, and Clement McGowan. 1991. "A Critical Look at Software Capability Evaluations," IEEE Software, 8(4):25-41.

Brock, Bishop, Matt Koffman, and J Strother Moore. 1996. "ACL2 Theorems About Commercial Microprocessors," pp. 275-293 in Proceedings of Formal Methods in Computer-aided Design. Berlin: Springer-Verlag.

Brodman, Judith G., and Donna L. Johnson. 1996. "Return on Investment from Software Process Improvement as Measured by U.S. Industry," Crosstalk: The Journal of Defense Software Engineering, 9(4). Reprint available online at <http://www.stsc.hill.af.mil/crosstalk/1996/apr/>.

Brooks, Frederick P., Jr. 1975. The Mythical Man-Month. Essays on Software Engineering. Reading, MA: Addison-Wesley.

Brown, Nat, and Charlie Kindel. 1998. Distributed Component Object Model Protocol—DCOM/1.0. Microsoft Corporation, January. Available online at <http://www.microsoft.com/oledev/olecom/draft-brown-dcom-v1-spec-02.txt>.

Burrows, Michael, Martin Abadi, and Roger Needham. 1990. "A Logic of Authentication," ACM Transactions on Computer Systems, 8(1):18-36.

Butler, R., and G. Finelli. 1993. "The Infeasibility of Quantifying the Reliability of Life-critical Real-time Software," IEEE Transactions on Software Engineering, 19(1):3-12.

Clarke, Edmund M., O. Grumberg, H.S. Jha, D.E. Long, K.L. McMillan, and L.A. Ness. 1993. "Verification of the Futurebus+ Cache Coherence Protocol," Transactions A (Computer Science and Technology), A32:15-30.

Clarke, Edmund M., and Jeanette M. Wing. 1996. "Formal Methods: State of the Art and Future Directions," ACM Computing Surveys, 28(4):626-643.

software for networked information systems 105

Clingen, C.T., and T.H. van Vleck. 1978. "The Multics System Programming Process," pp. 278-280 in Proceedings of the 3rd International Conference on Software Engineering. New York: IEEE Press.

Computer Science and Telecommunications Board (CSTB), National Research Council. 1997. ADA and Beyond: Software Policies for the Department of Defense. Washington, DC: National Academy Press.

Constantine, L.L., and E. Yourdon. 1979. Structured Design. Englewood Cliffs, NJ: Prentice-Hall.

Craigen, Dan, Susan Gerhart, and Ted Ralston. 1993. An International Survey of Industrial Applications of Formal Methods. Gaithersburg, MD: National Institute of Standards and Technology, Computer Systems Laboratory, March.

Craigen, Dan, Susan Gerhart, and Ted Ralston. 1995. "Formal Methods Reality Check: Industrial Usage," IEEE Transactions on Software Engineering, 21(2):90-98.

Croxford, M., and J. Sutton. 1995. "Breaking through the V and V Bottleneck," pp. 334-354 in Proceedings of Ada in Europe, in Frankfurt/Main, Germany. New York: Springer.

Curtis, Bill. 1981. "Substantiating Programmer Variability," Proceedings of the IEEE, 69(7):846.

DeRemer, F., and H.H. Kron. 1976. "Programming-in-the-Large Versus Programming-in-the-Small," IEEE Transactions on Software Engineering, 2(3):80-86.

Detlefs, D. 1996. "An Overview of the Extended Static Checking System," pp. 1-9 in Proceedings of the First Workshop on Formal Methods in Software Practice. New York: ACM Press.

Devauchelle, L., P.G. Larsen, and H. Voss. 1997. "PICGAL: Lessons Learnt from a Practical Use of Formal Specification to Develop a High-Reliability Software," European Space Agency SP 199, 409:159-164.

Diaz, Michael, and Joseph Sligo. 1997. "How Software Process Improvement Helped Motorola," IEEE Software, 14(5):75-81.

Digital Equipment Corporation. 1997. Workshop on Security and Languages. Palo Alto, CA: Digital Equipment Corporation, Systems Research Center, October 30-31. Available online at <http://www.research.digital.com/SRC/personal/Martin_Abadi/sal/home.html>.

Dill, David L., A.J. Drexler, A.J. Hu, and C.H. Yang. 1992. "Protocol Verification as a Hardware Design Aid," pp. 522-525 in Proceedings of the IEEE International Conference on Computer Design: VLSI in Computers and Processors. Los Alamitos, CA: IEEE Computer Society Press.

Dill, David L., and John Rushby. 1996. "Acceptance of Formal Methods: Lessons from Hardware Design," IEEE Computer, 29(4):16-30.

Dion, Raymond. 1993. "Process Improvement and the Corporate Balance Sheet," IEEE Software, 10(4):28-35.

Easterbrok, Steve, Robyn Lutz, Richard Covington, John Kelly, and Yoko Ampo. 1998. "Experiences Using Lightweight Formal Methods for Requirements Modeling," IEEE Transactions on Software Engineering, 24(7):4-13.

Fagan, M.E. 1986. "Advances in Software Inspections," IEEE Transactions on Software Engineering, 12(7):744-751.

Fayad, Mohamed, and Mauri Laitinen. 1997. "Process Assessment Considered Harmful," Communications of the ACM, 40(11):125-128.

Garland, David, and Mary Shaw. 1996. Software Architecture: Perspectives on an Emerging Discipline. Englewood Cliffs, N.J.: Prentice-Hall.

Glass, R.L. 1981. "Persistent Software Errors," IEEE Transactions on Software Engineering, 7(2):162-168.

Hall, Anthony. 1996. "Using Formal Methods to Develop an ATC Information System," IEEE Software, 13(6):66-76.

Hamilton, Graham, ed. 1997. JavaBeans. Palo Alto, CA: Sun Microsystems.

Heimdahl, M., and Nancy G. Leveson. 1996. "Completeness and Consistency in Hierarchical State-based Requirements," IEEE Transactions on Software Engineering, 22(6):363-377.

Heninger, K. 1980. "Specifying Software Requirements for Complex Systems: New Techniques and Their Application," IEEE Transactions on Software Engineering, 6(1):2-13.

Herbsleb, James, and Dennis Goldenson. 1996. "A Systematic Survey of CMM Experience and Results," pp. 323-330 in Proceedings of the 18th International Conference on Software Engineering (ICSE). Los Alamitos, CA: IEEE Computer Society Press.

Hernandez, J.A. 1997. The SAP R/3 Handbook. New York: McGraw-Hill.

Hoare, C.A.R. 1985. Communicating Sequential Processes. Englewood Cliffs, NJ: Prentice-Hall.

Honeywell Corporation. 1975. Aerospace and Defense Group Software Program, Final Report. Waltham, MA: Honeywell Corporation, Systems and Research Center.

Jones, C.B. 1986. Systematic Software Development Using VDM. Englewood Cliffs, NJ: Prentice-Hall.

Kitson, David, and Stephen Masters. 1993. "An Analysis of SEI Software Process Assessment Results: 1987-1991," pp. 68-77 in Proceedings of the 15th International Conference on Software Engineering (ICSE-15). Los Alamitos, CA: IEEE Computer Society Press.

Kuehlmann, A., A. Srinivasan, and D.P. LaPotin. 1995. "Verity—A Formal Verification Program for Custom CMOS Circuits," IBM Journal of Research and Development,
39(1/2):149-165.

Lampson, Butler W. 1983. "Hints for Computer System Design," Operating Systems Review, 17(5):33-48.

Lawlis, Patricia K., Robert M. Flowe, and James B. Thordahl. 1995. "A Correlational Study of the CMM and Software Development Performance," Crosstalk: The Journal of Defense Software Engineering, 8(9):21-25.

Leveson, Nancy. 1995. Safeware. Reading, MA: Addison-Wesley.

Leveson, Nancy G. 1987. Software Safety. Pittsburgh, PA: Carnegie Mellon University, Software Engineering Institute, July.

Lowe, Gavin, and Bill Roscoe. 1997. "Using CSP to Detect Errors in the TMN Protocol," IEEE Transactions on Software Engineering, 23(10):659-669.

Manes, Stephen. 1998. "Settlement Near in Technical Help-line Suit," New York Times, March 3, p. F2.

Marzullo, K., Fred B. Schneider, and J. Dehn. 1994. "Refinement for Fault Tolerance: An Aircraft Hand-off Protocol," pp. 39-54 in Foundations of Ultradependable Parallel and Distributed Computing, Paradigms for Dependable Applications. Amsterdam, The Netherlands: Kluwer.

McGarry, Frank, Steve Burke, and Bill Decker. 1997. "Measuring Impacts of Software Process Maturity in a Production Environment," Proceedings of the 21st Goddard Software Engineering Laboratory Software Engineering Workshop. Greenbelt, MD: Goddard Space Flight Center.

McLean, John. 1994. "A General Theory of Composition for Trace Sets Closed Under Selective Interleaving Functions," pp. 79-93 in Proceedings of the IEEE Computer Society Symposium on Research in Security and Privacy. Los Alamitos, CA: IEEE Computer Society Press.

Meadows, Catherine. 1992. "Applying Formal Methods to the Analysis of a Key Management Protocol," Journal of Computer Security, 1(1):5-36.

Meyer, Bertrand. 1988. Object-oriented Software Construction. Englewood Cliffs, NJ: Prentice-Hall.

software for networked information systems 107

Microsoft Corporation and Digital Equipment Corporation. 1995. The Component Object Model Specification (COM). Microsoft Corporation and Digital Equipment Corporation, October. Available online at <http://www.microsoft.com/oledev/olecom/title.htm>.

Middleburg, C.A. 1989. "VVSL: A Language for Structured VDM Specifications," Formal Aspects of Computing, 1(1):115-135.

Miller, Steven P. 1998. "Specifying the Mode Logic of a Flight Guidance System in CoRE and SCR," pp. 44-53 in Proceedings of the 2nd Workshop on Formal Methods in Software Practice. New York: ACM Press.

Musser, David R., and Atul Saini. 1996. STL Tutorial and Reference Guide: C++ Programming with the Standard Template Library. Reading, MA: Addison-Wesley.

Neumann, Peter G. 1995. Computer Related Risks. New York: ACM Press.

Ousterhout, John K. 1998. "Scripting: Higher-level Programming for the 21st Century," IEEE Computer, 31(3):23-30.

Owre, Sam, John Rushby, Natarajan Shankar, and Frederich von Henke. 1995. "Formal Verification for Fault-tolerant Architectures: Prolegomena to the Design of PVS," IEEE Transactions on Software Engineering, 21(2):107-125.

Parnas, D.L. 1974. "On a `Buzzword': Hierarchical Structure," pp. 335-342 in Programming Methodology. A Collection of Articles by Members of the IFIP Congress, D. Gries, ed. Berlin: Springer-Verlag.

Paulk, Mark C., Bill Curtis, Mary Beth Chrissis, and Charles V. Weber. 1993. "Capability Maturity Model for Software Version 1.1," IEEE Software, 10(4):18-27.

Pfleeger, S.L., N. Fenton, and S. Page. 1994. "Evaluating Software Engineering Standards," IEEE Computer, 27(9):71-79.

Porter, A.A., H.P. Siy, C.O. Toman, and L.G. Votta. 1997. "An Experiment to Assess the Cost-Benefits of Code Inspections in Large Scale Software Development," IEEE Transactions on Software Engineering, 23(6):329-346.

Potts, Colin, Kenji Takahashi, and Annie I. Anton. 1994. "Inquiry-based Requirements Analysis," IEEE Software, 11(2):21-32.

Pressman, Roger S. 1986. Software Engineering: A Practitioner's Approach. New York: McGraw-Hill.

Raymond, Eric, and Guy L. Steele. 1991. The New Hacker's Dictionary. Cambridge, MA: MIT Press.

Rochkind, Marc J. 1975. "The Source Code Control System," IEEE Transactions on Software Engineering, 1(4):364-370.

Ross, Douglas T. 1977. "Guest Editorial—Reflections on Requirements," IEEE Transactions on Software Engineering, 3(1):2-5.

Rushby, J. 1995. Formal Methods and Their Role in Certification of Critical Systems. Menlo Park, CA: SRI International, March.

Savage, Stefan, Michael Burrows, Greg Nelson, Patrick Sobalvarro, and Thomas E. Anderson. 1997. "Eraser: A Dynamic Data Race Detector for Multi-threaded Programs," Operating Systems Review, 31(5):27-37.

Saydjari, O. Sami, J.M. Beckman, and J.R. Leaman. 1989. "LOCK Trek: Navigating Uncharted Space," pp. 167-175 in Proceedings of the IEEE Symposium on Security and Privacy. Los Alamitos, CA: IEEE Computer Society Press.

Sommerville, Ian. 1996. Software Engineering. 5th Ed. Reading, MA: Addison-Wesley.

Srivas, Mandayam K., and Steven P. Miller. 1995. "Formal Verification of the AAMP5 Microprocessor," Applications of Formal Methods, Michael G. Hinchey and Jonathan P. Bowden, eds. Englewood Cliffs, NJ: Prentice-Hall.

Tanik, Murat M., Raymond T. Ye, and guest editors. 1989. "Rapid Prototyping in Software Development," IEEE Computer Magazine, Vol. 22, Special Issue (5).

Taylor, T. 1989. "FTLS-based Security Testing for LOCK," Proceedings of the 12th National Computer Security Conference. Washington, DC: U.S. Government Printing Office.

Thompson, Kenneth. 1984. "Reflections on Trusting Trust," Communications of the ACM, 27(8):761-763.

Watts, Humphrey, and Bill Curtis. 1991. "Comment on `A Critical Look,'" IEEE Software, 8(4):42-46.

Watts, Humphrey, Terry Synder, and Ronald Willis. 1991. "Software Process Improvement at Hughes Aircraft," IEEE Software, 8(4):11-23.

Weissman, Clark. 1995. "Penetration Testing," Information Security, M.D. Abrams, S. Jajodia, and H.J. Podell, eds. Los Alamitos, CA: IEEE Computer Society Press.

Trust in Cyberspace
Committee on Information Systems Trustworthiness, National Research Council (1999) 352 pages   6 x 9