Managing and Serving Large
Datasets:
TNRIS Serves Texas and Beyond
Chris Williams
About TNRIS
The Texas Natural Resource Information System (TNRIS),
a division of the Texas Water Development Board (TWDB), is
the clearinghouse and archive of natural resource data in
Texas. TNRIS acts as a central access point for
distribution of GIS data, as well as the overseers of
development and maintenance of the state’s official base
layers.
The TNRIS approach is built on three pillars: data,
a user community, and the technology of distribution. If
any one part is not available or deficient, the system
fails in its mission: the development of clean, automated
methods for GIS data distribution that meet the needs of
the entire user community. This is a challenge that TNRIS
takes seriously.
The Data Landscape
When discussing data distribution, it’s important
to understand the history of the data landscape in Texas.
It underscores the amount and nature of the data
distributed.
In 1996, the Texas Legislature passed Senate Bill 1
enacting the Texas Strategic Mapping Program, known as
StratMap, which is responsible for the creation and
maintenance of the state’s seven official GIS base
layers (Table 1).
StratMap relied on partnerships with other state
and federal agencies, counties, Councils of Governments (COGs),
academic institutions, universities, private companies and
consultants, and most importantly, the general public, for
the initial production of StratMap data. Those
partnerships continue to provide avenues for data
enhancement and updates to the StratMap layers.
TNRIS provides a framework of consistent base
layers, with standardized data schemas, which create the
quality and consistency between data products that users
demand. For example, a county can download part of the
StratMap transportation layer and then share it with the
surrounding counties without having to worry about
re-classifying or manipulating data. The standard schema
approach alleviates data translation issues.
In 2004, StratMap will begin working with the Farm
Service Agency (FSA–part of the USDA) to provide updated
DOQQs (Figure 1) for the entire state at least every two
years. The program will begin working with the new vendor
for updates to the transportation and boundary layers, and
will provide users of hydrography data the long-awaited
version of NHD available in geodatabase format at the
1:24k scale.
StratMap is only one of the data sources at TNRIS.
TNRIS also hosts the Texas Digital Aerial Photo Archive (TxDAPA)
and Texas Mapping Project (TxMAP). These programs deal
with scanning and registering of hardcopy datasets, like
historical topos and photos, to reduce the paper
products’ time in public circulation. TNRIS also
provides resources for designation as an Earth Science
Information Center (ESIC), houses the state’s
Borderlands Information Center (BIC), the Geographic Names
Coordinator (GNIS), and hosts a pilot project that’s
part of The National Map (Figure 2). TNRIS has scads and
scads of data, most of which supports the data
requirements of a program area, or two, somewhere in the
country, and perhaps the world.
The Users
The vast amounts and types of data that TNRIS holds
are eclipsed only by the variety of uses in the community.
The user community includes everyone from GIS
professionals to elementary school teachers, from retired
geophysicists to university students looking for data for
term papers, from farmers and ranchers doing land and
resource management, to government agencies doing water
and wastewater studies, from private consulting companies
that access the data onsite to an anonymous Web surfer who
finds the data illustrating the location of his house in
relation to an aquifer recharge zone. Roughly 200 people
per month walk through the door at TNRIS with a custom
data request. Unique Web users number between 40,000 and
50,000 per month and download between 600 and 800
gigabytes of data via FTP per month. ArcIMS requests are
also off the chart (Table 2).
Use numbers grow every month, but the one number
that doesn’t, is the number of TNRIS employees,
currently 12 full-time staff with four unfilled positions.
The Information Services team—the team charged with
developing the data distribution systems at TNRIS—therefore
leverages a few core technologies to develop systems that
are automated and can withstand the demands of an entire
state’s user base.
Using Technology
TNRIS uses technology to build creative solutions
that meet data needs and eliminate duplication of effort
sometimes involved in data updates. How does TNRIS manage
distribution for of all this data for all these users in a
flexible way? The system leverages assorted Web and
database technologies to catalog and provide multiple
access points to data sets. Currently, the access points
available include HTML-based, FTP-based, and ArcIMS-based
options.
TNRIS uses a custom form of the Geography Network,
known as the Texas Geography Network, to catalog GIS data
sets in Texas. TNRIS also uses two data delivery
mechanisms, HTTP and FTP-based access file-based data
(compressed into ZIP files) for download (Figure 3). For
users who are just interested in visualizing data, TNRIS
provides Web access to ArcIMS Web services that are built
on StratMap and other data sources. Typically, the map
services are pre-packaged Internet-based maps that support
basic functions.
The latest data delivery endeavor, introduced just
last month, will provide virtually unlimited access to
TNRIS’ ArcSDE Database through a Clip-Zip-Ship custom
application. The application is Web-based and is built on
Safe Software’s Feature Manipulation Engine (FME), a
format translation tool (among other things). The solution
supports the extraction of data from a data source by a
user-defined “box” into a variety of GIS formats.
Shape files, DWG, and TAB files are the most popular. The
application can then re-project the resultant dataset into
almost any coordinate system. Currently, the projection
systems listed are Texas-centric, but can be expanded. The
result of this wizard-like process is a ZIP compressed
file that is immediately available for download. The file
contains data only for the area of interest, in the custom
data format for the user’s software, and is in the
custom projection for easy vertical integration with the
rest of the user's data (Figure 4).
Conclusion
Utilizing standard data frameworks that were
developed through partnerships, TNRIS makes an effort to
meet the needs of a large user base. StratMap data
provides the central component for the development and
maintenance of base layer data that is distributed through
multiple access methods: walk-in, FTP, and data
extraction. The TNRIS data distribution credo remains:
data, user community, and technology.
About the Author
Chris Williams, the TNRIS database administrator,
has been working with ArcSDE and Microsoft SQL Server to
manage StratMap and other TNRIS data sets for the last
four years. He has worked with GIS in Texas for 10 years.
Chris holds a Bachelor of Science degree in Geology from
the University of Texas at Austin.
A special thanks to Brent Porter,
TNRIS Webmaster, for creating the graphics used in this
article.
Back
