The Solution is Evolution: Solving the Problem of Mega-data Delivery
By Angela Daniels, Andrew Shepard and Sridhar Katragadda

It is often said that nothing in the universe is constant except for change. Within the realm of digital databases and electronic resources, this is the only paradigm. At Intermap Technologies Inc., this paradigm has been exemplified by the transition in the GLOBAL Terrain business unit from paper-map catalogs to discrete digital-map catalogs, to catalogs stored and queried in a spatially integrated relational database management system.

The GT Coverage maps used in the initial configuration of the GLOBAL Terrain web site (Figure 1)

In the Beginning
On a worldwide basis, Intermap Technologies has created over 700,000 square kilometers of digital elevation and orthorectified radar imagery map products using its STAR-3i interferometric radar mapping technology. This collection is held in its GLOBAL Terrain database and is packaged as 7.5-minute tiles (similar to the USGS DEM products). There are more than 4500 individual datasets covering more than 60 sites that include the digital surface elevation model (DSM), a digital terrain model of the bare earth (DTM), an ortho-rectified radar image (ORRI), and associated metadata and thumbnail image files. Each of these datasets occupies approximately 80MB of disk space, and the entire dataset occupies 287GB. In the period from 2000-2001, the number of collection sites grew from 42 to more than 60 and the number of tiles from 2963 to nearly 4500. These numbers increase daily as the STAR-3i system continues to collect data.
      In a plan for Intermap to collect data for the entire continental United States, the database will expand to 54,000 tiles, which would require nearly six terabytes of storage space. If Alaska were added, this database would grow to more than seven terabytes. Adding in a sizable portion of Europe and Asia could extend the database to well past 15TB.
      The management of this database requires knowledge not only of the physical location of the data, but also of the spatial reference and ancillary information for each dataset. As the database grows, so does the demand for information that is contained within, and for online access to it. Since the GLOBAL Terrain database is expected to increase more rapidly in the near-term, Intermap began evaluating its organization and dissemination methodologies for these products.
      When GLOBAL Terrain was launched in 1997, the mechanism for communicating the coverage available was through a series of static graphic maps. These maps were available as JPEG and PDF files, captured on a series of web pages and broken down by project area (Figure 1). These graphics were created by mission planning software and generally reflected a planned collection area. In some cases, these maps may not have encompassed actual data holdings. Each graphic was posted to the GLOBAL Terrain web site, along with information about that dataset. At the time, and even though the company created its own metadata for every file, those metadata were not compliant with National Spatial Data Infrastructure (NSDI) formats.

Evolution
In order to have a spatially accurate index of available data, a process was created that used final-product extents to generate a GIS layer to show all available tiles. This GIS database was used internally by both customer service and the company's sales force to determine exact locations of existing GLOBAL Terrain data (Figure 2). During this time, an effort continued to initiate the definition and creation of NSDI-compliant metadata for each file in the database, and to incorporate into a "folder" configuration all the associated files related to any tile.
      These changes, coordinated company-wide but actually accomplished by separate departments, eventually resulted in the awarding of a contract to PixSell Inc. for developing a database application called "GT-MIS." This web-based resource-powered by TerraSoar software from Core Software Technologies-is geographically enabled and can query multiple, distributed databases with a customizable map-based visual search. Initially, the system was designed and implemented on the LINUX operating system, and used MySQL as the database engine. Over time it has been migrated to a Windows NT environment, and the plan is to migrate the database to a Sequel Server or Oracle database system. This MIS was developed primarily for internal use and continues to be used solely in that manner.
      The creation of this database spurred an interest in having a similar resource available to the general public via the World Wide Web, one that could provide a more interactive interface to the GLOBAL Terrain database with the GT-MIS database as its foundation. Thus, the GT ArcIMS Server was born.

The New Paradigm
During the past year, Intermap and PixSell have jointly developed a more integrated spatial database and recently launched a web-enabled GIS using ArcIMS¨ software from ESRI. This software offers a more dynamic, robust means of organizing, utilizing and displaying all the data that is currently offered by Intermap. The web-enabled GIS platform provides a means of continuously updating the catalog and adding to the functionality of the system as the database grows. Having the ArcIMS interface directly with the Internet database also improves customer and data distributor service by enabling efficient access to all the information necessary to evaluate the spatial distribution, pricing, and quality of these datasets. The development of this system also allows Intermap to participate in ESRI's Geography Network, where ESRI users can access certain datasets online by means of an e-commerce-based subscription service.
      ArcIMS software is a powerful Internet mapping solution that provides a framework for organizing and deploying centralized geographic information system (GIS) services and data to a broad audience. Intermap users and customers can gain access to data remotely, integrating them with data from many other Internet and local sources. ArcIMS also offers various methods for enabling those remote users to analyze and interpret the data one provides. As Intermap is a part of the Geography Network, customers can use the map service to access GT data and analyze it locally, while maintaining the storage remotely. ArcIMS is also scalable, enabling the server to handle demands of multiple map requests and multiple datasets. Because this software uses image rendering to deliver snapshots of the data in the form of compressed image files, it can serve up raster and vector data sets to the end-users. Users of the GT ArcIMS system can also perform predetermined or dynamically created queries to derive specific information about that data. Since the client processes can be in either HTML or JAVA, there is no need for the end-user to buy specific software to achieve database access.
      This paradigm is a perfect fit for the way that GLOBAL Terrain needed to handle its growing database. The interactive ArcIMS interface can be used to query the database and, unlike previous representations of the GLOBAL Terrain coverage maps, it is actually possible to display thumbnail images of the data as well as metadata information about individual datasets.
      Using functionality built directly into the software, both internal and external users of GLOBAL Terrain data and ESRI software can access the data index information stored in the database, and the actual data itself using Intermap's intranet system and the Internet-based Geography Network. ArcCatalog and ArcMap (applications that are a part of ArcGIS) are able to work with the remotely stored data through ArcIMS in the same way that they work with locally stored data. This is a powerful concept that takes advantage of the ability of the Internet to be a storage device for geospatial data.

In the Future
As the GLOBAL Terrain database continues to grow, challenges will remain with data delivery issues. Currently, the data is delivered to the customer as individual 7.5 x 7.5-minute tiles. When orders are for large numbers of tiles (i.e., larger areas), delivery and ingestion can be problematic. The next logical step is to move from individual tile-based delivery to database delivery - giving the customer a database file that can be easily integrated with existing datasets.
      Considering the size and growth potential of the GLOBAL Terrain database, distributed storage makes perfect sense. And in today's digital economy, distributed access and e-commerce are also natural phases in the evolution of the GLOBAL Terrain business model. Intermap's first step into this arena is greatly facilitated by using ArcIMS and the Geography Network as offered by ESRI. The merging and advancement of these technologies serves as a reminder that nothing is constant except for change.

About the Authors:
Angela Daniels and Andrew Shepherd are both employees of Intermap Technologies Inc., Denver, Colo. Sridhar Katragadda is an employee of PixSell Inc., Bay St. Louis, Miss.

Back