Cross Sensor Fusion
The sky's (no longer) the limit
By Gene Davis

The human body is an excellent example of cross sensor fusion. Sight, sound, and smell are blended by the body to render more complete analyses to the human brain. The brain then directs bodily operations, making adjustments based on continuous inputs from our human sensors - our eyes, ears, and nose. In performing flight operations, additional sensors (equilibrium - inner ears - and tactile - seat of the pants "feel") are brought into the equation to allow pilots to cope with three-dimensional maneuvering of aircraft.
    Just as science is augmenting the body's natural sensors to allow greater accomplishments in the realm of flight and space, science is creating an array of sensors that contribute to many endeavors, from exploration for oil to improved crop yields through "scientific farming." "Cross sensor fusion, the blending of inputs from a variety of sensors, promises significant advances in analytical capabilities across a broad spectrum of operational fields," according to Mark Lucas, chief technical officer for ImageLinks.
    From the first U.S. satellite, Explorer I launched 40 years ago, sensor information began to support operations. Sampling of the radiation belts around the Earth indicated that it would be safe for men to operate beyond the immediate atmosphere of the Earth. Our earliest satellites were listeners and photographic sensors. Out of the photographic satellites came weather tracking and forecasting. Mapping and geological analyses followed. The use of multiple images enhanced the range of information available and fostered development of overlay processes that produce composite maps. An outgrowth of military intelligence needs, this process reveals geological formations that predicted oil deposits.
    The sophistication has grown until the satellite sensor systems now produce panchromatic (black and white), multi-spectral (color), and radar imagery of very high quality. When merged (called cross sensor fusing) by companies such as Imagelinks of Melbourne, Florida, the imagery becomes a key component for advanced analytical tools and operational support systems in a variety of fields.
    Optical multi-spectral systems that include Landsat TM and SPOT are referred to as passive systems. They rely on sunlight reflected from the Earth to image the surface. RADARSAT, by comparison, uses Synthetic Aperture Radar (SAR) microwave signals to pulse the Earth. These signals, reflected back from the Earth, create radar imagery like that used on aircraft and ships since World War II. Radar satellites typically employ a wavelength suited for atmospheric penetration. It overcomes conditions such as cloud cover, fog, dust, hail, smoke, and darkness inherent in passive systems. "This ability provides the user with significant advantages when it comes to viewing under conditions that preclude observations made by aircraft or optical satellites," according to the RADARSAT Geology Handbook.
    The images from any sensor are corrected for satellite motion (roll, pitch, and yaw). "The ImageLinks processing then adjusts the imagery for spatial position. Imagery is registered using the best elevation model available. At that time, the imagery is then suitable for cross sensor fusing with other sensor data," says Lucas. "Combining information from sensor platforms creates new solutions for old and new industries."

Limitations and Solutions
Limitations on use of satellite sensor data generally fall into four categories: temporal, spatial, spectral, and cost.
    Temporal, or time related limitations, may have to do with the conditions to be sensed (rapidly changing conditions such as forest fires or volcanic eruptions) or the sensor equipment (number, frequency, and direction of sensing platforms). With an ever-increasing number and variety of satellite sensors systems available, the user has an expanding array of imagery options. "The ability to obtain effective cross sensor fusions provides the user with the ability to use diverse sensors in a substitute mode as well as in combination," according to Lucas. "In a darkened (night time or smoke covered) sensor survey area, for instance, where a forest fire or molten lava is being tracked, improved coverage, techniques, and integrating processes may allow thermal or radar data sensors to substitute for visual sensor data information."
    Spatial (size and location) obstacles are being overcome by several developments. Access to all locations on the Earth grows as the number of orbiting satellites increases. Sizing of objects on the Earth can be problematic. For instance, the ability to vary the size of a radar beam and the angle of the beam allows the capture of significant terrain features. However, "the phenomenon of foreshortening and layover causes distortions in both size and distance relationships of objects. Using sophisticated computer programs, highly experienced personnel, and cross sensor fusion, companies are able to compensate for distortions and provide accurate representations of the areas surveyed," according to industry experts.
    Spectral problems have included lack of access to key frequencies which relate to specific data requirements. For instance, it was mentioned that elements such as smoke, fog, clouds, or darkness rendered optical frequency sensors essentially inoperative. The development of equipment using thermal and radar frequencies has improved the ability to collect data under any atmospheric conditions for a wide spectrum of commercial activities. The integration of data from a variety of sensors, using different frequencies for interrogation, yields better solutions. For instance, in scientific farming, adding radar data to Landsat thermal data could provide information about soil, sediment, and bedrock conditions. This information could lead to better understanding and solutions than through analyses of surface conditions (as revealed by Landsat) alone.
    Cost is an important factor in any endeavor, but especially in a commercial one. Costs, to be effectively analyzed, should be measured by considering total costs to obtain the required information in a format immediately usable. New methods of obtaining information should be explored. Can satellite and/or aerial information, properly fused, reduce exploration teams? Does simulation using fused imagery and fly through software techniques permit airlines to train their crews without operating costly flights? We know it provides the pilot familiarization with new landing sites. Less flying time yields a reduction in risk. Reductions in risk can bring savings in insurance and other risk associated factors. The proliferation of satellite sensors, in terms of numbers and nations boosting satellites, should increase worldwide competition to provide sensor services and therefore result in reduced cost. Still, the most effective means of reducing costs in the short term is the innovative application of technologies that are being developed.

Applications
The number of applications where cross sensor fusion can contribute significantly is limited only by the imagination. Simulation applications in aviation are visible, practical and cost effective. The key is accurate geo-positioning of the imagery which can best be obtained by using a feature correlation algorithm rather than polynomial warping.
    When the C-5 military aircraft came out in the early 1970s, the aircraft had inertial navigation systems on board and self-analyzing diagnostics for the mechanical systems on the aircraft. The initial and advanced simulator trainer for the aircraft, however, used small-scale relief maps, anchored vertically on three story high walls, over flown by a miniature television camera to provide pilots with realistic terrain simulation. "Cross sensor fusing of panchromatic 5 meter data and multi-spectral 30 meter data from satellites provide the approach base map. Aerial imagery is co-registered and inserted to provide the airport /landing site detail. A pilot can "fly" over any terrain in the world. The pilot will experience the same visual sensations as he would flying over the actual terrain," says Lucas. His visual perspective will change with the altitude of the aircraft. With motion included, the realism is sufficient to "route" qualify pilots for areas they have not flown.
    This training is invaluable for military flying, which is not routine and could occur in any location in the world. Visual flying is often the required mode of operation. Safety and mission effectiveness are dramatically improved with such realistic simulation. This simulator application is equally important for commercial flights, even when instrument flying is the norm. Takeoffs and landings in marginal weather are safer when a pilot is thoroughly familiar with sight clues in the airport area. With cross sensor fused data hosted in a robust simulation system, all types of flying conditions are possible.
    There are additional applications for companies that fly in remote regions, such as Alaska or parts of Latin America. The U.S. State Department could employ this level of simulation for its anti-narcotics operations, where crews operate in difficult terrain and hazardous conditions. Illegal crop identification training would be greatly improved through simulations employing cross sensor fusion imaging.
    The petroleum industry is one of the greatest geological explorers. Its geologists and geophysicists search for outcroppings of sedimentary beds that indicate a potential source of crude oil. While it still requires a physical presence to actually detect and extract the oil, the technology discussed in this article has vastly expanded the ability to search for new fields. Radar satellite images, as an example, "provide geologists with a unique and complementary data source with respect to contact, structure, lineaments and land forms," according to their market literature. Extensive research has been done to identify the particular visual bandwidth associated with differing mineral and geologic content. Using radar for the intensity source by controlling the angle and direction of collection and fusing the information with the visual bandwidth will yield a reliable indication of mineral or geologic content.
    The versatility of today's satellites permit them to approach targeted areas in a manner that provides varied and viable sensor inputs. The ability of radar to penetrate vegetation provides imagery that complements optical and other thematic collections. Accurately registering the data and fusing it with visual bands will provide the exploration company with a first look for significant materials at a specific geographic position, potentially saving the company costs for a man in the field.

Summary
The exploding technology of sensor platforms and the continuing incredible developments in computer equipment and software applications hold the promise of increasingly effective tools for users of sensor data. Some industries have already embraced these technologies. There are others that have not yet grasped the value of the tools available. Temporal, spatial, spectral, and cost questions are being addressed daily. Solutions are forthcoming - especially when the questions have been identified.
    Simulations requiring "virtual reality" visions of operating areas - no matter how remote - will continue to improve as the state of the art in cross sensor fusing advances. Geographic explorers will undoubtedly make greater use of cross sensor fusing products in the future. Archeologists, environmentalists, logging companies, and even urban planners need to follow developments in this field. Technology solutions for sensor requirements employing cross sensor fusing await only for the challenges to be posed.

About the Author
Gene Davis is a free-lance writer who specializes in applications of Space Age innovations to real word situations. His works include technical dissertations on NASA by-products as well as informational pieces. He currently consults on the technical aspects of Federal Procurement Regulations for the Space Industries as well as doing free-lance writing.

About the technical contact:
Mark Lucas is the chief technical officer for ImageLinks. A retired Air Force Officer with extensive experience in remote sensing satellite programs and implementation, Mr. Lucas provided the detailed technical information regarding the unique registration process used by the ImageLinks software.

Back