UNIT 62 - SYSTEM EVALUATION

UNIT 62 - SYSTEM EVALUATION

Compiled with assistance from Warren Ferguson, Ferguson Cartotech, San Antonio, TX
• A. INTRODUCTION

• B. STRATEGIC PLAN

• C. REQUEST FOR PROPOSALS (RFP)
  • Contents of the RFP
  • Distribution of the RFP
  • Vendor proposals

• D. HARDWARE AND SOFTWARE ISSUES
  • Software
  • Hardware

• E. SYSTEM CHOICE
  • Evaluation factors
  • Two stages of evaluation
  • The winning proposal
  • Risk factors

• REFERENCES

• EXAM AND DISCUSSION QUESTIONS

• NOTES

UNIT 62 - SYSTEM EVALUATION

Compiled with assistance from Warren Ferguson, Ferguson Cartotech, San Antonio, TX

• once the functional requirements study is complete and management gives the "go-ahead", the next step is to develop the document which will solicit proposals from interested GIS vendors
  • this document is the Request for Proposals (RFP)

• results from the RFP will produce a number of different GIS options for the organization, each of which will have strong points and weaknesses
  • at this point, difficult decisions will need to be made in an attempt to match needs with products available in the current marketplace
  • management will need assurance that the system chosen is the best option available
  • responses to the RFP will indicate the feasibility of achieving the project's goals

• an open attitude to the relationship with suppliers and the conduct of tests is essential
  • evaluations must be open to outside scrutiny
  • decisions may be (and frequently are) challenged by vendors and must stand up in court

• this unit examines these aspects of the system evaluation process:
  • the strategic plan
  • the RFP
  • hardware and software issues
  • system choice and reducing the risks

• a strategic plan is essential in defining the limits of the project
  • is important in providing guidance for many later decisions
  • provides a level of planning above that of the FRS, less specific to the system

• decisions are made regarding the scale of the desired project
  • will it be a small departmental activity or will it be integrated into operations of the whole organization?
  • will it be centralized or distributed?

  • how many people will be using the system at full implementation?

• need to address which activities need to be automated and which, if any, should remain manual

• how fast should acquisition of the system proceed?

• what are the priorities of data input, software development and output?

• should the project development be directed by a consultant or by in-house committees?

• how will the project be funded?

• the functional requirements study along with decisions made for the strategic plan form the basis for the request for proposals
  • success of the RFP is directly proportional to the quality of the analysis on which it is based

handout - Extracts from an RFP (2 pages)
• the RFP describes in detail the following aspects:
  • nature of the proposed database
  • sources of database contents
  • required functions to create and manipulate the database
  • specification of required products, including frequency

• specifies the functional requirements for the project, not the specific technical processes underlying the functions
  • must allow vendor to adapt capabilities of system to the organization's specific requirements
    • e.g. must not specify raster vs. vector, or other data structure alternatives, but allow vendor to choose most appropriate
  • the RFP must allow the vendor to determine the best configuration to satisfy the user's requirements
    • what size of CPU
    • how many input devices - digitizers, scanners etc.
    • how many output devices
    • what software options and enhancements
  • an RFP which is too rigid may exclude potential suppliers

• the details of the required proposal are made very clear:
  • defines all the requirements
  • outlines the form of response expected and format requirements
  • sets deadlines

• the RFP starts the formal relationship between organization and suppliers

• the RFP is sent to all interested suppliers
  • potential suppliers can be identified by polling, or by inviting response to an RFI (request for information) or RFQ (request for qualifications)

• potential suppliers might be invited to a preliminary meeting to ask questions, reach agreement that it is worth proceeding further
  • cost to vendor in responding to an RFP can be high, need to make sure it is worthwhile

• conventional approach is to distribute RFP, make first cut of vendors based on proposals received in response, then proceed with more detailed evaluations of the selected systems
  • in the early days of GIS (pre-1984) it was common to receive very few (two or three) responses to an RFP, particularly if the RFP was detailed, because of the poor level of software development in the industry
  • GIS industry has now advanced to the point where six to ten responses might be expected to an RFP for a large (multi-million dollar) project

• respond in detail to the customer's requirements

• include details of proposed system configuration
  • software
  • hardware
  • network and communications
  • workstations and digitizers
  • maintenance and training
  • costs

• vendors may have relatively poor data on rates of throughput for specific configurations
  • possible that the proposal is either under- configured (cannot meet the required workload) or over-configured (excess capacity)
  • further tests, such are benchmarks (see Unit 63) are often required to reduce these uncertainties as much as possible

Software

• the proliferation of GIS software available makes the choice of a single system difficult (see Unit 24)
  • however, there is no single best software for any particular application or organization
  • mandates, decision-making processes and data and product requirements make each installation unique

• software choices that will need to be considered by an organization include:
  • sophisticated applications-specific modifications of standard packages
  • systems with built-in customization options
  • immature systems with great potential for innovation

• different capabilities with regard to data model, functionality, output, database management system, etc.

  will each affect the overall operation of the GIS significantly and will need to be individually evaluated and compared

• decisions made with respect to hardware issues determine:
  • number of people that can work at one time
  • size of projects that can be handled
  • cost of purchasing and maintaining the equipment
  • need for a computer systems manager
  • start-up effort
  • update potential
  • vendor support and stability

• many of these issues will be addressed by the technical requirements laid out in the FRS and RFP

• however, there will be several trade-offs required in the final decision

• evaluation requires balancing many factors

• costs of hardware and software - will vary despite identical functionality

• speed and capacity of hardware

• quality and costs of support

• supplier's background
  • in addition to system capabilities, it is also necessary to evaluate suppliers on:
    • financial stability
    • position in the marketplace
    • reports from other users about quality of support
  • references are a useful way of obtaining this information
    • appropriate customer references should be supplied by each vendor

• does the vendor's proposal live up to the vendor's own claims?

• how does the vendor's proposal rate against other proposals?

• must be good enough to get the project funded

• winning vendor and customer may need to work together in making final presentations to management

• justifying selection of supplier is only one part of winning project approval
  • however a well-managed selection process is more likely to lead to a successful project

• each vendor's system has certain risks associated with its implementation
  • the vendor's product may not live up to expectations
    • e.g. the hardware configuration may be insufficient for the planned workload
    • e.g. the software may not carry out the functions as claimed

• many risks are associated with the project and become part of the final decision-making
  • several of these risks and uncertainties regarding hardware and software issues have already been pointed out

• other risks are much more subtle
  • e.g. since many vendors are US-based, foreign organizations must consider the stability of the value of the local currency against the US dollar

• the typical planning horizon for a GIS project is 5 years
  • most factors are very difficult or impossible to forecast this far ahead

• however good the planning, there is a risk that the system will not satisfy the end-users
  • in fact the winning vendor's system may fall short of requirements in several key areas
  • it may be necessary to modify the system definition because of limited vendor capabilities - some products may have to be dropped
  • in other cases, the final contract should require the vendor to develop software to deal with these problems

• when additional software development is required, the contract must include deadlines and penalties because success is heavily dependent on the additional software being supplied on time and fully debugged
  • this situation is still common because of the immature state of the GIS industry

• in view of these risks, an investment of 5% or even 10% of project costs in planning and system evaluation is more than justified

• organizations wishing to reduce these risks further may conduct one or more additional sophisticated, though costly, procedures before making the final commitment
  • these include:
    • benchmark tests
    • pilot studies
    • cost benefit analyses

Forrest, E., G.E. Montgomery and G.M. Juhl, 1990. Intelligent Infrastructure Workbook: A Management-Level Primer on GIS, A-E-C Automation Newsletter, P.O. Box 18418, Fountain Hills, AZ 85269-8418.

Guptill, S., 1988. "A process for evaluating GIS," USGS Open File Report 88-105. The report of the Federal Interagency Coordinating Committee on Digital Cartography (FICCDC) on GIS evaluation.

Smith, D.R., 1982. "Selecting a turn-key geographic information system using decision analysis," Computers, Environment and Urban Systems 7:335-45.

EXAM AND DISCUSSION QUESTIONS

1. Review the approach to system selection documented in Smith (1982). What are the arguments for and against the rigorous decision-theoretic approach used in this paper?

2. Discuss the steps in planning and choosing a GIS system. What are the risks associated with a project, and how are these reduced in the project lifecycle approach?

3. "The best-laid plans of mice and men...". Despite the use of a well-defined framework, mistakes inevitably happen in the best-designed projects. Discuss the weaknesses in the approach described in these units.

Back to Geography 470 Home Page

Back to GIS & Cartography Course Information Home Page