Structured Estimating Improves Quality of Sensing/Mapping Software Cost Estimates
By Gary Constantine

Moving to a structured estimating process has helped the Imagery & Geospatial Systems Division of Raytheon Company (Raytheon IGS) substantially improve the quality of its sensing/mapping software cost estimates. The process of cost estimating is crucial to major sensing/mapping systems providers such as Raytheon IGS because it has a major impact on the profitability of large programs. In the past, these processes were less structured and delivered variable results. But in recent years, companies have implemented a more formal process in which standardized metric elements are used to provide a quantitative basis for generating program estimates. The accuracy of estimates has increased and, in addition, the structured approach helps build increased confidence in the accuracy of estimating processes that simplify downstream negotiations.

Screen shots of the SEER-SEM cost-estimating tool.

     Raytheon IGS is a worldwide developer and integrator of high-technology electronic systems. As a market leader, Raytheon IGS develops and produces scalable remote sensing ground systems that receive and process raw image data from remote sensing satellites and aerial photography sources. Related systems support environmental and resource management, law enforcement, surveillance and reconnaissance, and hydrographic mapping. IGS offers extensive and comprehensive information technology services that include consulting, systems engineering, management and integration, operations and maintenance, and training for high-performance computing environments.

Complex Estimating Challenge
Large sensing/mapping software projects such as those developed by Raytheon IGS present a very complex estimating challenge. The methodologies, languages and standards used in the software development process are constantly changing. The fact that nearly all the value added to the end-product results from human ingenuity increases the difficulty of the measurement process. Traditional bottoms-up measurement methods vary widely in accuracy because they depend, to a great extent, on the skills and experience of the estimator. Another problem with the unstructured approach is that the lack of documentation created during the estimating process makes it difficult to quickly and reliably evaluate the inevitable changes and concerns that occur during the negotiation process.
      In addition, the spreadsheets that are historically used to document bottoms-up estimates have built-in limitations. One problem with a spreadsheet model is that it is vulnerable to ever-increasing errors as the model becomes more complex. Something as simple as entering the wrong formula in a single cell could change the final estimate by millions of dollars. This type of error is difficult to catch because of the difficulties in auditing a one-of-a-kind spreadsheet model. Another problem with spreadsheet models is that they provide a single-point answer that does not normally take into account range estimating.

Standardized Methodology
To overcome these challenges, suppliers have long worked with a variety of structured estimating tools. The basic advantage to the structured approach is that it provides a standardized methodology to use historical data for refining the estimating process. Accuracy increases over time as the metrics used for generating initial estimates are calibrated to actual project results. As a byproduct of the estimating process, this structured approach also generates the phase distribution of effort and scheduling information that can dramatically improve the project oversight function. The importance of this effort increases as the size and complexity of systems continue to grow, and the pressure increases to minimize overall project costs.
      One of the tools to have enjoyed widespread success in this field is SEER-SEMª from Galorath Inc. (El Segundo, Calif.). SEER-SEM accepts a wide range of input parameters that include size, personnel, complexity, environmental constraints, platforms and applications, and development and acquisition standards. This tool also allows the creation of customized parameters. SEER-SEM's multiple knowledge bases address client-server, embedded, distributed, and stand-alone systems, varied MIS applications, graphics, and signal processing environments.

Generating Estimates
The estimating process begins with SEER-SEM when members of the proposal team develop a solution set for meeting the requirements of the RFP (request for proposal). At this point, the members of the estimating team begin building a model by entering parameters that organize the costs in an understandable manner. An example of a typical parameter is one that measures the level of formality of the program. A manned space flight, for example, requires a very formal program run to military standards so that high-level designs, detailed designs and test plans are offered at stand-up presentations that require a considerable amount of preparation. The majority of military programs, on the other hand, now use high-grade commercial standards that simply involve technical interchange meetings that demand much less preparation.
      The program then generates an estimate for each phase of the project, organizing the costs in any manner that is desired by the project team. The transparency of the estimate makes it possible for the project team to easily consider its reasonableness and, if necessary, make adjustments to the program parameters. An important advantage of SEER-SEM is that it allows the user to enter probability levels for each parameter, thus making it possible to determine the impact of changes in various program parameters. This results in the estimate including a "most-likely" value as well as a range of probable values based upon the uncertainty factors that were entered by the estimators.

High Levels of Accuracy
Raytheon IGS estimators use the SEER-SEM model for rough-order-of-magnitude estimates, firm proposal estimates, and management evaluation of software development activities. The key to achieving high levels of accuracy is through the division's comprehensive calibration efforts. The model is calibrated by evaluating completed programs in which the costs are a known factor. The person performing the calibration interviews the engineering lead on the program in an effort to understand such variables as the skill level of the developers involved, and the characteristics of the project. A model is created to estimate the program after the fact. This is an iterative convergence process consisting of comparisons between the model's parameter selections and resultant cost estimate, and the real world's program characterization and cost.
      After calibrating a considerable number of programs, general trends begin to emerge. One begins to understand the capabilities of an organization and the requirements of their customers as they fit into the continuum that is measured by the model. It is possible to compare the skill level of different groups within an organization and identify trends such as an increase in performance of a particular group as its members gain more experience. This process is critical to a successfully structured estimating program. By devoting attention to it, Raytheon IGS has been able to dramatically improve both the accuracy and consistency of its estimates over time.

Aids the Negotiation Process
Much of the advantage to using structured estimating tools comes well after the estimating process has been concluded. One characteristic of structured estimating is that it delivers not only a number, but also a methodology that makes it possible to trace exactly how that number was generated. The fact that estimators can easily provide backup information to explain the estimating process helps increase customer confidence. In many cases, even after the bid is accepted, the negotiating process may continue. The model is extremely useful during this process to provide an audit trail and make it possible to evaluate the impact of changing program assumptions.
      The model can also provide a useful framework for evaluating the performance of the overall project. Comparing actual results to detailed model estimates can provide an early awareness of problems and help determine their impact on the overall cost and schedule of the program. In certain cases, early results from a program may cause estimators to question the parameters that were used in creating the model. As a result, the model could be quickly updated to factor in the effect of changing the earlier assumptions and to determine the impact on the overall program's budget and schedule. In addition, the model can greatly assist management's decision-making by allowing them to compare the expected results of different program management alternatives.
      While Raytheon IGS Division users have access to a wide range of estimating tools, over the years most users have shown a preference for SEER-SEM. The division has been very pleased with SEER because of the high-quality estimates it produces and the high level of acceptance by its customers.

About the Author:
Gary A Constantine is the manager of Parametric Estimating for the Imagery & Geospatial Systems Division of Raytheon's C3I Systems. In 1985 he joined the former E-Systems Corporation, a legacy Raytheon company, as a member of its first full-time parametric estimating group. Over the past sixteen years, Gary has been responsible for managing and coordinating the development, implementation, and government review of several parametric-based estimating systems. Gary is an active member of the International Society of Parametric Analysts (ISPA) where he is a three-term member of the ISPA Board of Directors, currently serving as chairman of the board of directors. Gary is also an active member of several other professional societies and is a plank owner member of the joint DoD/Industry Parametric Cost Estimating Initiative. In 1994, ISPA recognized his contribution to advancing the use of parametrics by awarding him its prestigious "Parametrican Of The Year" award.

Back