| GPS Consumer Series: Batch Processing of GPS Data, Part 1 of 2
By Chuck Gilbert Introduction
Work fascinates me - I could sit and watch it for hours. Please don't misunderstand me, even those of us who love our work, and who work hard, can still appreciate any tool that reduces the amount of time we spend on repetitive or tedious tasks. This month we will explore automation in GPS software; features that speed and simplify the processing of GPS data. Pre-defined configurations and intelligent defaults
Automation of a GPS processing task can take many forms. At the most simple level, a GPS program that allows the user to pre-set all configurable items could be described as having some degree of automation. (I realize that this is a stretch, however, I'm sure that this is probably the case in some GPS advertisements.)
A little more sophisticated than a simple, pre-configuration would be a program with the ability to anticipate what data set(s) you will require next and what type of process will be desired. Such a program could provide the user with automatic file selection and intelligent defaults that change based upon the context of what the user is actually doing at that moment. (See Figure 1.) The problem with either of the two ideas described above is that the user is still doing the work. Even if the program makes things faster and easier, the user is still required to sit in front of the computer and push buttons to make things go. Batch processing
A giant step beyond intelligent defaults is the concept of batch processing. The a batch processor is one program that can start and run all of the required processes with a single command. For all users of GPS for GIS data capture, the collection, processing, and export of GPS data involves several processes that are quite independent of one another. A typical work flow is outlined below:
¥ Data Transfer - First, regardless of GPS or GIS manufacturer, all GPS-based data collection systems require data transfer to transfer the collected data from the field device to a PC or other computer.
¥ Differential Processing - Second, if the user did not use real-time differential correction, their data will almost certainly require post-processing. Differential post-processing can improve the spatial accuracy from about 100m to better than 1 meter.
¥ Data examination - Third, almost all users will wish to view their data before they export it to their GIS, CAD or database.
¥ Data Editing - Fourth, if the quality of the data is not satisfactory, the user may wish to perform some simple edits prior to export.
¥ Export - Fifth, the final stage is typically the export of the data from its native GPS format to an appropriate GIS compatible interchange format.
Users will typically need to take some or all of the steps outlined above. A batch processor could allow the user to return to the office after data collection and simply press one start button to accomplish all of the desired tasks.
The best possible scenario is a program that will do all of the above simultaneously; that is, it will support pre-set configurations, will supply intelligent defaults, and will operate in batch mode. There are GPS processing programs on the market today that offer all of the above. Such programs truly make the task of GPS processing as painless as possible. Another batch of benefits
The benefits of a good batch processor go beyond merely relieving the boredom of repetitive tasks. Perhaps the biggest benefit of a batch processor is that users can be certain that the same processes, options, and settings have been applied to their data every time. In many companies, new data is collected, processed, and exported every day. There is a constant risk that the data processed today may get accidentally processed with a different configuration than the data of previous days or weeks. When some configurations are changed, such as QA constraints, the resulting data may be of poor quality. If configurations such as the datum, projection, or coordinate system have been inadvertently changed, the resulting data could have spatial shifts of up to hundreds of meters.
Typically, differential correction programs and data export programs will have a myriad of options. In the interest of data integrity and consistency, a batch processor can help to ensure first, the same set of programs are run every time, and second, that all these programs are configured in the appropriate manner.
Another benefit of a batch processor is that an untrained user could very easily initiate a long series of complex processes. This reduces the training requirement for an organization and also helps to leverage the time of the in-house GPS expert. How to tell a good batch from a bad batch
Let's examine a few of the features that users might want to look for in a batch processor. At the risk of being obvious, the first thing to look for in a batch processor is what processes it is able to run in batch mode. At a minimum a batch processor should be able to perform the following tasks:
a) data transfer from the field device to the computer,
b) differential processing of all rover files against multiple base files if necessary,
c) export of the GPS data from it's native GPS manufacturers format to a GIS, CAD or database format of the user's choice.
Another vital consideration regarding a batch processor is the log file created whenever the processor is run. A few questions to consider are:
¥ Is a log file automatically created? This is crucial. When a batch process is performed correctly it can save many hours in front of the computer. However, this can be a double edged sword. When used incorrectly, one mouse click could possibly lay waste to dozens of files, representing hundreds of hours of field data collection. A good log file will go a long way toward helping users recover from an erroneous batch run.
¥ If a log file is created, does it contain all the processing details or just a processing summary? My preference is an expandable log file. Such a file may initially show only a summary of the processes, however with one click of the menu, the file can be expanded to display the full details of every process that occurred. Another very nice twist is that the batch process can also spawn other unrelated log files. For example, imagine a batch processor that was instructed to first do a differential correction, followed by an export of the data. When a large number of data files are being processed it can be very convenient to have separate log files; such as a batch log that reports whether the batch process was a success, an independent export log file that includes all of the details related to the export process, and finally a log file that is specific to the details of the differential correction.
¥ Are the records in the log file time-tagged? It can be very useful if the data in a batch log are all time tagged. The time tags can help a user identify the relative requirement for various parts of the processing sequence. Knowing which processes and files process quickly and which process slowly can help a user to streamline the processing. The user may elect to perform certain processes by hand and others by batch.
¥ Is the log file overwritten by subsequent batch runs? When a process fails, the most natural thing in the world is to do it again. Unfortunately, if the batch log file(s) are overwritten in subsequent runs, valuable trouble shooting information could be irretrievably lost. It is best if the log files are named automatically and will not be immediately overwritten.
¥ Can the information in the log file be copied, pasted, or otherwise easily saved? Depending on the contents of the various log files, some users may wish to save or manipulate the log file details in a database of a spreadsheet. It is best if the log files details can be conveniently saved and/or moved into other applications. A secondary consideration is whether the batch processor will display data to the user for visual inspection. This is not a common batch function because such interactive processes (such as the visual inspection of GPS data for obvious errors or flaws) do not lend themselves well to automation. Some users choose to do visual inspection in the GPS software before export and others prefer to perform visual examination of the data after it has been transferred to the local GIS or database. Where the visual examination is performed usually just a matter of the GIS coordinator's personal preference.
Another lower priority in the evaluation of a batch processor is the editing of data. Again, editing is generally a more interactive process and not well suited to automation. Typically, the editing functions in a batch processor will be limited to the filtering out of data that do not meet certain predefined quality criteria, such as "remove data collected with too few satellites," or "remove data that has not been differentially corrected successfully," or "remove data that did not meet my minimum accuracy requirement." Often the editing of data is done in the GIS environment instead of in the GPS software. I rate this as a lower priority for a batch processor because the cleaning and editing tools available in a GIS or CAD system are usually superior to those in the GPS software. Summary
Consistent and accurate GPS processing is so important that I would not even consider purchasing a GPS program that cannot be run almost entirely in batch mode with intelligent defaults. A well designed batch processor will make nearly all of the required GPS processing fast, easy, and extremely reliable. This one tool can make the difference between a GPS program that is complex, temperamental, and only accessible to a highly trained GPS expert; or a simple yet powerful tool that can be run in local offices by field crews and can reliably handle large quantities of data on a daily basis. About the Author:
Chuck Gilbert has over a decade of experience as a GPS user. He has been employed as an applications engineer for Trimble Navigation since 1989. Back |
