As the leader of one of your organization's systems development groups, you have just been given an especially difficult assignment. It seems that all projects have tight deadlines, but this one has some special problems. The application is unusually large and response time is critical. If the system cannot meet its performance goals, it will have to be scrapped.

The members of your group begin by browsing through the library of domain models that have been accumulated since your reuse program began in earnest several years ago. The hypertext browsing system locates several model fragments that may help to construct a specification of the new product.

An online conference determines what problems will arise in integrating these models and identifies parts of the application for which no models exist. Integrated conferencing software displays the graphical models to each team member on a high-resolution workstation. The team identifies and resolves issues, and the contents of the conference are captured for latter use in the documenting system.

You assign tasks to members of the group. The process management system captures this information for latter use in tracking the project.

 Once the new model fragments are constructed and all the parts are integrated, the specification is ready for review. You schedule a review with the end users. Since the models are captured in a formal graphical language, they are "executable." A similation of the product's behavior is presented using an analysis/design toolkit that provides graphical animation. The users identify some minor changes in requirements based on the simulation and you make a note to incorporate them into the specification.

Before beginning design, members of your team do a rough analysis of the performance of the proposed system. A performance engineering tool parses the specification model and constructs a performance model. There are still many details to be resolved in design and implementation, but this high-level analysis identifies some areas of risk. You assign one developer to investigate these areas more closely, possibly via a prototype.

The target environment is a distributed system, and several ways to partition the design appear to be possible. Analysts construct design models representing the alternatives using the analysis/design toolkit and pass the results to the performance engineering tool. With more information available (including task structure and processor allocations), the results of the performance analysis are more detailed. As a result, additional areas of risk are identified.

Ultimately, developers select a design capable of producing the required performance using a significant number of components from your reuse library. Code for some of the new components can be automatically generated from the design models. Other components must be hand coded to meet performance requirements. As the code is produced, its performance is measured and necessary changes are made.

After testing (also supported by automated tools), the application is delivered to the end users. They are delighted with both its functionality and its performance. Was it delivered on time? Well, since this is clearly a fictional scenario, why not?

The scenario described here, and the development environment that supports, it, are obviously fictional, but all of the required capabilities exist. Some are research prototypes, many years from production quality; others are commercial products available in today's computer-aided software engineering (Case) market.

The major barrier to more effective software engineering today is not a lack of automated (Case) tools to support development tasks. While tool capabilities will continue to expand and new tools will appear, the principal problem to be solved is integration of methods and tools across the software development cycle.

In the scenario, software analysis and design techniques were fully integrated with performance engineering techniques. The specification and design models, produced with Case analysis and design tools, provide input for performance engineering tools. The results of the performance studies provided feedback that was used to modify the design and identify possible areas of risk. The result was a product that met both its functional and performance requirements while preserving a coherent, well-documented design.

To realize the benefits suggested by this scenario, automated support for analysis/design (i.e., Case) must be tightly integrated with automated support for performance engineering. Yet, current analysis/design tools are poorly integrated with software performance engineering (SPE) tools, if at all.

Computer-aided software (or systems) engineering (Case) is a family of technologies based on the construction of models, at various levels of detail, of the system under development. Traditionally, Case products have focused on the analysis and design phase of the development process (so-called upper-Case capabilities). For the remainder of this article, the term "Case" will refer to these analysis/design toolkits.
 Case methods provide rules and suggest strategies for constructing models that are typically expressed using graphical notations. Analysis and design methods used with Case include functional decomposition and object-oriented development. Graphical notations supporting Case methods include entity-relationship diagrams, Youdon/DeMarco or Gane/Sarson-style dataflow diagrams, and, for realtime or reactive systems, the Ward/Mellor and Hatley notations.

Case products typically support the use of one or more of these notations to construct and analyze models of the proposed system. Model analysis can include: syntax checking, completeness and consistency checking, and data model normalization. The graphical notations used for modeling realtime systems are sufficiently formal that dynamic model analysis can include: reachability, deadlock and non-termination. Newer-generation Case products, such as Statemate from iLogix, Burlington, Mass., and foresight from Athena Systems, Inc., Sunnyvale, Calif., provide early versions of these capabilities.

These models also contain enough information to construct performance models. Current Case tools have not, however, exploited this fact.
 Some Case tools, such as CardTools from Ready Systems, Sunnyvale, Calif., provide performance analysis capabilities. Here, however, analysis is limited to a particular language (Ada) and target environment (VRTX/ARTX). Performance estimations also require information that is not available until the latter stages of design or implementation. Thus, early assessments of performance are not possible. While these capabilities are a step in the right direction, there is much more to be done.

MEETING PERFORMANCE GOALS

Software performance engineering is a method for constructing software systems that meet performance goals. [C.U. Smith, "Software Performance Engineering," Proceedings of Computer Measurement Group Conference XII, New Orleans, La., Dec. 1981, pp. 5-14.] It provides a framework for modeling software requirements and designs to predict performance. The predict performance is then evaluated to determine whether the goals can be met. If not, alternatives are developed and assessed, and corrective action is taken before proceeding.

This process continues from the requirements analysis phase through detailed design, with the development of increasingly more precise models of the software and analysis of its predicted performance. As code is produced and testing begins, SPE continues with performance measurement and management of the evolving software.

SPE includes techniques for gathering data, compensating for uncertainty, constructing and evaluating performance models, evaluating alternatives and verifying and validating results. It also includes strategies for the effective use of these techniques. With SPE, the metrics obtained from performance models can be related to the software architecture, the characteristics of the workload and the characteristics of the target (hardware and software) environment. This allows the construction and evaluation of alternative models.
 SPE methods and the tools that support them emerged from the technical areas of performance evaluation and capacity planning [C.U. Smith, "Evolution of Software Performance Engineering: A Survey," Proceedings of the Fall Joint Computer Conference,, Dallas, Tex., Nov. 1986, pp. 778-783.] For example, capacity planners create models of their current configuration, use measurement data to characterize workloads and calibrate the model to match existing performance metrics. They examine future needs by using forecasts of future workload volumes and resource requirements, and study hardware configurations that will satisfy those needs.

Initial applications of SPE supplemented capacity planners' measurements of existing workloads with models of new software in early development stages. This was done to forecast the new software's transaction volumes and resource requirements. The models then predicted hardware requirements of systems early enough to enable timely procurement.

There are many performance modeling tools available that provide capacity planning decision support, such as Best/1 from BGS Systems Inc., Waltham, Mass., Map from Amdahl Corp., Sunnyvale, Calif., and SES/Workbench from Scientific and Engineering Software, Austin, Tex.

 Mature SPE efforts use model predictions to evaluate trade-off in software functions versus hardware cost. Model predictions are also used to control resource requirements by selecting requirements and designs with acceptable performance characteristics. They track and manage performance throughout development to prevent problems that can threaten successful and timely project completion from surfacing late in the development process; for example, integration testing.

Case and SPE methods, notations and tools evolved independently. As a result, there are several barriers to integration. The notations used for constructing analysis/design models and SPE models are very different. In addition, the information maintained in the upper-Case tool repositories contain some, but not all, of the data needed by SPE tools.

There is, however, no reason why Case and SPE cannot be more closely integrated. As the fictional scenario suggested, the methods are complementary. The results of SPE analysis can provide valuable feedback in the analysis and design stages. Despite notational differences, the models the two approaches use are very similar. Integration of Case and SPE tools would eliminate the need to construct and maintain consistency between redundant sets of models and make it possible to receive this feedback in a more timely fashion.

Several organizations are already combining Case and SPE on and ad hoc basis. MCI Telecommunications Corporation, Arlington, Va., regularly uses SPE techniques. Alfonso Rodriguez, a senior software performance analyst, described recent efforts on a new Network Provisioning System.

"The system maintains a very large database of circuit activity, network connections and circuits available, and processes orders for new analog and digital services generated by other remote, automated systems," he said. "The database consists of approximately 150 DB2 tables with sizes ranging from one million to 95 million rows. The software consists of 983 programs. An average program makes 100 database accesses."

 According to Rodriguez, developers implemented the system within 10 months using the Method/1 and Design/1 Case tools from Chicago-based Arthur Andersen.

"Our SPE team involvement began in the project initiation phase," Rodriguez said. "In the early stages we created and evaluated Crystal [from BGS Systems Inc.] and Paws [from Scientific & Engineering Software] models to identify and alleviate potential bottlenecks, and to determine the best mechanisms for transmission of data between remote systems.

"Subsequent efforts concentrated on the critical components. During detailed design and coding we used performance reviews to monitor the DB2 database structure and the SQL within programs. We found that some inexperienced programmers created 'fancy' queries that would have excessive run times. With strong management backing, we instituted early corrective action," Rodriguez said.

During integration testing, MCI used IBM's TeleProcessing Network Simulator (TPNS) with a suite of performance measurement tools for functional, stress and volume tests. They tuned DB2, CICS, and Vsam to reduce system overhead, and conducted studies that identified programs causing the highest DB2 activity. They then bactracked to the queries tables and indices causing inefficient performance, and analyzed and implemented alternatives to reduce the total access path, I/O and sort costs.

 The combined efforts of the software developers and performance engineers resulted in a system which was deployed on time, functioned correctly and had exceptional performance. "On the first day of operation, we were able to process a peak load of 50 online transactions per second, with an average end-to-end response time of 0.33 seconds," Rodriguez said. "The system also ran a concurrent batch workload of 1,500 jobs. Everyone was both amazed and delighted with the system's initial performance."

Developers at Baltimore Gas and Electric have also experimented with Case and SPE integration. Marc Cherbonnier, systems project leader for the Customer Information System (CIS) project, said, "We began with performance benchmarks to study the feasibility of using DB2 for our new CIS. Then, developers used Arthur Andersen's Method/1 and Design/1 to document their design, and Install/1 to assist in implementation. Our performance engineering team used the Design/1 reports in performance walk-through to identify potential problems.

"We combined the performance data to predict the computer capacity that the new system will require. We used spreadsheet models to analyze logical and physical I/Os for key processes in the system. Early efforts focused on transaction response times. Later, we also found a key batch process that requires 1.8 million physical data table I/Os and approximately the same number of index I/Os.

"Obviously, this process cannot complete within our batch window. We know that if we can structure our tables and break the processing into subsets that can run concurrently, we can prevent a serious performance problem" Cherbonnier said.
 "Now we are using TPNS to conduct performance and stress tests to ensure that we will meet performance goals upon delivery. Although we have not deployed the system, we are confident that our early studies led to a system correctly sized to handle the load. We can now process 22 user transactions per second (56 CICS transactions) with 90% of the end-to-end response time under two seconds.c

Success was not always easy. "While Case tools helped in our early SPE studies, as we got closer to the project deadline, the performance testers have found inconsistencies between design documentation, the performance-tested version of code and the latest version. After we prepare performance recommendations we have to double check to make sure that they still apply," Cherbonnier said. "We also found that the early design specifications were overly optimistic."

An ideal unified Case product would alleviate many of their manual SPE tasks, provide early warning of changes that cause new performance problems and minimize design and code inconsistencies. Cherbonnier added, "An ideal product should incorporate measurement data as the software evolves for easier performance validation. It should also provide performance feedback to users so they can distinguish between a new system that is a 'greyhound' versus one that is a 'hog.' Then they can apply pressure to get the system they want. The measurement problem will challenge product developers, though. With DB2, a small test database causes transactions to behave very differently than they will in a production environment."
 SPE HOT OFF THE PRESS

Ralph Terkowitz, vice president of data processing at the Washington Post, described that organization's approach to software development. "We rigidly adhere to SPE methods. Designers understand performance-design principles and apply them from the outset to create a workable system plan. Then we use lower-Case tools such as Cullinet's ADS/Online fourth-generation language [now owned by Computer Associates, Garden City, N.Y.] to produce software.

"With this combined approach, all our new systems meet their performance objectives." For SPE modeling, "we use simple, pencil and paper models to make design decisions for individual systems," he said. "We use Best/1 to evaluate the overall effect of all new systems to make sure that we do not 'run out of gas.'"

Dean Berry, consulting engineer at Covia, Rosemont, Ill., the United Airlines reservation system provider, described an early success with applying SPE to their North American Fare Quote System.

"The system finds the lowest available airfares between two cities, taking into account seat availability, alternative routing and so forth," Berry said. "Software developers collaborated with performance engineers throughout the system reengineering life cycle. Developers had been 'burned' in the past by performance and wanted to prevent problems.

"Together, we not only improved the quality of the fare quotes -- to 100% accuracy -- we also reduced DASD I/O by approximately 60% and CPU utilization by approximately 38%, while increasing the functionality to include every applicable fare."
 According to Berry, Covia had no Case tools for the first project, but plans to use Atlanta-based Knowledge Ware's IEW Workbench with the established SPE techniques for its new international fare quote system.

In all these cases, the practitioners "manually" combined the two technologies. As a result, they encountered communication problems, such as: gathering data from diverse groups, matching performance results to current software status, justifying changes and so forth. Many of these problems would not occur if the support tools were better integrated.

TOWARDS INTEGRATION

Bill Inmon, senior principal with American Management Systems, Denver, said, "Case has been focused on the business function to the exclusion of everything else. As Case matures, and its focus broadens, it is inevitable that the focus will turn to SPE."

Kenneth Kolence of Kolence Associates, Palo Alto, Calif., was the 1974 recipient of the Computer Measurement Group's (CMG) AA Michelson award for his pioneering work in performance measurement. He is a strong advocate of Case and SPE integration. "The motivation for my early measurement work was to provide the theoretical and practical foundation for the design plus performance combination," he said. "Most current SPE tools, though, are used after the design is complete; they use detailed design data, and report its performance characteristics. Not only does this require duplication of effort (because the tools are not integrated), it also addresses the wrong problem.
 "Tools need to support the earlier design synthesis process to enable designers to explore alternatives to the fundamental function and form of the system," Kolence explained. "My research focuses on higher-level methods that will enable this exploration."

Alan Howard, vice president of operations for Applied Computer Research in Phoenix, said, "Those who endeavor into the Case world should already understand that a basic premise behind Case is quality. SPE should be high on their list of quality concerns. In the SPE area there is no shortage of 'reactive' tools, but that is not where the economics are. The economics of SPE are early in the life cycle where tools are very sparse."

Robert Gray, manager of SPE at MCI, pointed out another serious problem caused by the lack of tool integration. "In our environment, we have very tight development schedules," he said. "Performance engineers must gather design data and construct independent performance models. If they find design problems, developers may not have time to rebuild the design description.

Even though the design is suboptimal, they must live with their first design pass. This often results in excessive hardware capacity costs, large ongoing operational costs, crisis tuning efforts and resulting deterioration of code quality. After delivery, performance becomes a tangible problem, and the second release of the software imposes a performance requirement.

"If the tools were integrated, designers could explore alternatives before committing to designs," Gray said. "Exploring performance alternatives would be equivalent to exploring design alternatives, thus the design revisions would be a byproduct of the exploration."

 According to Gray, there is also an economic barrier. "Who pays for development costs versus operational costs? If the developer is not accountable for ongoing operational costs, their priority will be on reduced development costs even though the net cost is much higher."

BGE's Cherbonnier added that "even though my own 'homegrown' spreadsheet models indicated performance problems, developers were skeptical of the results. A performance tool integrated with the Case tool would have had more credibility." This observation was seconded by most of the practitioners cited in this article.

Furthermore, models and tools alone are insufficient. Both Case and SPE need to be integrated into a formal software development process and formal analysis and design methods. Then developers will better understand the role of Case and SPE, what steps are needed and when they should be conducted. Thus, both technologies will be used more effectively.

To promote the method and process integration, some members of the CMG are forming a task force to build the SPE and capacity planning steps into IBM's proposed AD/Cycle and SAA process models. (Contact CMG Chicago headquarters at 312-938-1228 to participate.)

 Computer-aided engineering and computer-integrated manufacturing (CAE-CIM) have inspired both Case and SPE evolution. Are engineering tools better integrated than software engineering tools? Dr. David Loendorf, staff scientist responsible for CAE-CIM tools at Los Alamos National Laboratory in California, said. "Most current CAE-CIM tools integrate analysis and design. They evolved together because the same engineer was interested in both design and analysis [models], and created computer tools to provide an integrated system."

With so many problems to be solved by Case tools, will vendors have time to address SPE integration? There are already some good signs.

Cadre Technologies, Providence, R.I., offers Teamwork as its comprehensive Case tool. Joe SiSanto, simulation product manager, said, "We believe that dynamic modeling of both behavior and performance is critical to good software design. Accordingly, we provide worldwide marketing and support for the ADAS performance evaluation project, developed by the Research Triangle Institute, Research Triangle Park, N.C. ADAS provides graphical performance evaluation capabilities, including animated simulation and color feedback, to aid in interpreting its results.

"The ADAS developers provide us with continuing support and enhancements," DiSanto said. "We plan to smooth the transition between design and performance models. Our first step is to provide a bidirectional graphical translator between Teamwork and ADAS. A designer can create either a performance or design model and automatically translate it to the other. We also plan to enhance the simulation capabilities, and to fully integrate the two models.

"Teamwork is widely used for real-time embedded systems; it is also useful for MIS systems," DiSanto continued. "Some of our MIS users have found it valuable for configuring distributed processing systems with a mix of hardware systems and network interconnections. The models enabled them to evaluate capacity, fault tolerance, throughput and response time."
 A REFRESHING APPROACH

A product with a refreshing approach to integration--even though it applies in a specialized environment seldom found in MIS applications--is CardTools from Ready Systems.

Likewise, the Microelectronics and Computer Technology Corporation (MCC) in Austin, Tex., a joint research consortium of U.S. computer manufacturers, has incorporated performance modeling capabilities into Verdi, their visual tool for designing distributed systems. Among other functions, it depicts a color animation of system behavior and reports times (mean, variance, distribution) between start and end markets. Verdi is currently only available to MCC's member companies.

Some vendors of SPE tools are not waiting for Case vendors. The first generation of performance modeling tools were oriented to capacity planners who were familiar with performance models and used them to study hardware trade-offs. However, the new generation of SPE tools is built for developers with some performance modeling skills who wish to use tools for software trade-off studies as well as hardware sizing.

PerSpective from Advanced System Technology in Denver combines elements of both Case and performance modeling tools. Traditional data and control-flow descriptions of software are integrated into stimulus-control-flows that are more amenable to performance evaluation. Specification of hardware architecture completes the information required to produce standard measures of performance using both analytic and discrete event simulation models.

 Robert Goettge, president, said, "A prototype version of PerSpective was used to model the U.S. En-route Air Traffic Control software. The software specification consisted of more than 50 stimulus-control flows containing over 2000 modes. Performance predictions were sufficiently accurate that inconsistencies in physical data placement among centers were detected."

Apriori Systems, Inc., Chicago, has announced a tool that will allow software developers to model software performance. The analyst will describe business factors, database and process design and evaluate resulting performance metrics. Apriori plans to offer the product later this year.

Metron, based in the U.K., offers a performance engineering system consisting of an interactive, expert system, Perseus, combined with the firm's Athene systems modeling tool.

Meanwhile, the original vendors of software performance modeling tools, BGS Systems, Inc. (Crystal) and Performance Systems, Inc., Rockville, Md. (Scert II), offer enhancements to their products as users identify desirable new functions. Crystal, for example, provides a flat-file interface that, together with a user-defined translation utility, allows information from Case tools to be imported.

Another new tool bridges the gap between Case and SPE. Design Enhancer, marketed by Applied Computer Research, Phoenix, addresses the logical database design phase. It asks designers questions about the application, the execution environment and the database. It then produces reports suggesting additional data need, critical areas to watch and so forth.
 Richard E. Fiori, a senior analyst in data services at Arizona Public Service, has used Applied Computer Research's Design Enhancer to evaluate new subsystems for the Customer Information System. He said, "It asks questions about our logical data model [and other system characteristics], and suggests alternatives when it detects problems. Once it suggested that I should separate operational data from decision support data. It was right. I was 'pounding' on one set. Now I keep that principle in mind. It serves as a good communication and documentation aid. It does not require you to make changes, but you have a checklist ad can document your actions and reasons."

SPE provides a means for developing software that meets its performance requirements without sacrificing other desirable qualities. While there are tools that support software analysis and design and tools that support SPE, they are currently poorly integrated, if at all.

Despite the difficulties involved in manually combining Case and SPE tools, several users have been successful in making SPE an integral part of their development process. More effective use of these techniques will require better automated support in the form of tightly integrated Case and SPE tools. Recent trends in tool development are encouraging, but there is still room for improvement.

 Smith, Ph.D., a consultant based in Santa Fe, N.M., specializes in SPE seminars and consulting services. She received CMG's AA michelson award for her work in SPE. She is author of Performance Engineering of Software Systems, recently published by Addison-Wesley.

Williams, Ph.D., is a Boulder, Colo.-based consultant who specializes in pre-implementation support for software systems development. His work emphasizes methods, tools and support environments for the description and anlysis of software designs. He is the author of over 50 technical articles.



http://findarticles.com/p/articles/mi_m0SMG/is_n9_v10/ai_8714994/pg_8