Software Radar Architecture
Frank D. Lind (1), Tom Grydeland (2), Philip J. Erickson (1), William Rideout (1), and John M. Holt (1)
(1)MIT Haystack Observatory, Route 40, Westford, MA 01886 USA (2)Dept. of Physics, University of Tromsø, Tromsø, Norway
Abstract
We discuss the structure of software radar systems and the architectural elements necessary for applying these systems to ionospheric radar and radio science applications. We describe the architectural requirements which result from the needs of a radio science system including requirements of performance, reliability, coherence, and traceability. We show that these requirements can be met using a Software Radar Architecture combined with particular design choices. We argue that the described Software Radar Architecture enables the implementation of radio science data systems which are significantly more capable than systems using traditional approaches. In particular a single architecture can exploit high performance computing capacity to implement multiple radio science instruments simultaneously. We focus here on the principles involved in the implementation of the software radar architecture in a manner which supports the operation of multiple instruments simultaneously. This discussion applies to a general class of radio science instrumentation including active and passive radar, beacon based techniques, and general spectrum monitoring. We frame these discussions in the context of practical examples from systems that we have implemented. Introduction
Experimental ionospheric radio science involves the acquisition of information about the near space environment using a variety of active and passive techniques that rely on the propagation or scattering of radio waves by the ionospheric medium. These experimental systems are constructed of components which couple an appropriate sensor into the electromagenetic environment allowing for the transmission or reception of signals on selected frequencies and bandwidths of the spectrum.
The data systems used with scientific radar systems such as incoherent scatter radars have been rapidly evolving rapidly in recent years. This evolution is largely driven by developments in electronics and computing technology which have allowed significant improvements in the capabilities of these systems. This rapid development has created a significant technological challenge in that an architecture and system which is optimal at one point in time will rapidly become obsolete. This obsolescence can manifest itself both in terms of system performance as well as in terms of the long term maintainability of the scientific instrumentation. It can become a significant challenge to keep systems operational without implementing full replacement data and control systems on a regular basis. Reimplementation is a very costly process due to the high complexity of modern radio science systems and the necessity of validating the performance of a new design in numerous ways. Such validation is not always done but it is vital if the data produced by radio science instrumentation is to have lasting value.
Software Radar is an architectural approach which we have previously discussed [Holt et al 2000; Grydeland et al 2005] and which has been demonstrated to be applicable to the data acquisition needs of multiple incoherent and coherent scatter radar systems. The Software Radar architecture is unique in its ability to take advantage of technological change to improve the capabilities and performance of a radar system. This ability is not accidental and is primarily the result implementing in software the largest portion of the instrument as is possible. Maximizing the use of software to implement the functionality of a radio science system minimizes the number of custom hardware components in a system. It also provides the greatest benefit from increases in computational and network capacity. By using general purpose software for these implementations the Software Radar approach also helps to ensure the long term viability and functionality of the underlying "instrument implementations" both for the signal processing and analysis portions of the system. This architectural approach also enables the development of instrumentation systems and networks which share computational and network resources to enable capabilities which would otherwise be impossible. The software centric approach also has important implications for the validation of radio science instrumentation systems and the scientific data which they produce.
The Architecture of Radio Science Experimental Systems
[FIGURE : 4 tier Information Structure Diagram ; scientific objectives, operational requirements, end user requirements ; experimental system (hardware, software, network, computing); implicit information structures (signal chains, data flow, control flow, coherence); experimental record (command, status (low volume), data (high volume), profiling, debugging)]
key ideas :
the experimental system is a result of the larger scale objectives and requirements an important requirement for an experimental system is the ability to understand the instrument function and how it modifies the geophysical observable that is being measured
In any experimental systems the instrument function can be a dynamic function of time. Modern systems can often control the behavior and evolution of the instrument function. This can be due to the configuration of the hardware, software, and the implicit methods by which the information flows in the instrument. Such behaviors can become so complex as to be essentially non deterministic when combined with a changing operating environment. Multiple instrument functions can also effectively exist for the same information stream from a given sensor.
The experimental system contains the traditional elements : hardware, software, network, and computing. These are the obvious elements with the "hardware" being the most traditional and modern information systems taking the place of pure "human" intelligence with software, networking, and computing systems.
The experimental system also contains non-obvious elements : signal chains, data flow, control flow, and coherence. These are the implicit information structures of the system. In most modern system the implicit information structures can be as or more important than the hardware in determining the response of the instrumentation and the meaning of the resulting experimental data. Historically it was only necessary to document and understand the behavior of system hardware. In any modern system the implicit information structures must be strongly revision controlled as they can change dynamically and greatly impact the quality of the experimental data. This need for careful revision control of implicit structures is particularlly true of software defined instrumentation such as a Software Radar system. The experimental system ultimately produces an experimental record. It is insufficient to simply have the data by itself. One must have both command, status (low bandwidth), data (high bandwidth), profiling, and debugging information. You must know both what you commanded the system to do as well as what actually happened. The correspondence between the commanded and actual state of a system is a function of both the system design (synchronous versus asynchronous) as well as the coherence of the underlying elements which make up the instrument. Information on the state of the system is necessary for the reconstruction of the instrument function as well as for the validation that a particular observed effect is in fact geophysical in nature and not an artifact of the system.
Coherent Software Radar Systems
[FIGURE : Block diagram of a coherent software radio system. This is a revised and more detailed version of the one in many of my talks. Also included will be timing digitization which isn't in prior versions of the figure. The ability to acquire selected 'state' variable information from other instrumentation is important to the generality of applying our architecture to a wide variety of systems.]
key ideas :
Radar and instrumentation implemented primarily in software Derivative of software radio in communications but not entirely the same idea Bandwidths of Voltage Level Data Sampled Coherently is the Common Factor High speed, low jitter, high dynamic range A/D converters Timing Digitization Coherent GPS locked oscillators and PPS driven UTC alignment Minimal analog interface components
System information flow is asynchronous with coherence maintained with information alignment at the A/D converters. This coherence can be local or global depending on the time alignment used (i.e. GPS PPS or a Radar IPP). This implies a later synchronization pattern for software elements which require a synchronous alignment of inputs. The latencies of the information transport network are variable largely due to our choice of transport. It probably is possible to do everything in hard real time. There are examples of CORBA stuff which works this way.
Contrast with a hardware based implementation (e.g. RADAC like system but don't mention RADAC explicitly) and describe and outline the differences in these types of architectures as well as where some of the same ideas are implicitly implemented by hardware or firmware (i.e. weaving pattern in the firmware). Clarifly the benefits and limitations of each approach. We should elaborate here why our solution is the more flexible, interesting, and hopefully productive approach. This is a good place to start on answering 'why not just put it all in a single FPGA?'.
[FIGURE : Overview diagram of the receiver designs : baseband IQ, non-zero IF, direct RF, direct RF to network and a photo of actual hardware?]
Discuss the use of different analog down conversion architectures and point out that for performance reasons most systems have converged on non-zero IF or direct RF acquisition. Describe how high performance analog conversion, combined with high speed ADCs, leads to software radio systems which can be truly multirole. This isn't different than in traditional 'software radio' approaches. Discuss the traditional 'software radio' approach and how it differs from our 'software radar architecture'. This will focus on comparing and contrasting the requirements for adaptability, latency, and performance. A key idea here is that for radio science systems the underlying performance qualities and metrics are significantly different than for software radio systems used in other contexts. This implies that a different architecture will be most optimal for the needs of our application. Software Radar is that architecture and the fundamental property is that :
RF Bandwidth <=> Network Bandwidth
This is possible because radio science latency requirements are far more forgiving than in traditional Software Radio systems. A great deal of latency in time (minutes, hours, years, decades, centuries?) between the acquisition of the data and its use can occur without the intent of the system being significantly compromised. Different types of applications can also share different data streams and computing resources in a manner which is proportional to their operational requirements. Thus a scientific or operational result which is of high priority can have computational resources allocated preferentially to its production. Results which might be of great importance but less required immediacy can be produced with greater latency and a lower allocation of computational assets (but often requiring greater storage assets). This is essentially making the traditional time space trade off found in most computing systems an variable in the use of resources within a Software Radar system.
In producing such systems it is NOT the particular digital receiver which is important. It is how the receiver is used (i.e. coherently) and all the surrounding infrastructure (i.e. software radar) which is important. Hardware comes and goes, well written software lasts a long time (may examples more than a decade), the architectural approaches and patterns are the universals which underly these systems. Where performance is required it may be necessary or beneficial to implement elements of the system in hardware (e.g. digital downconverters) however the general idea is that it is preferable to use a pure software implementation unless there is a sufficient benefit at a low enough cost. This may be in the difficulty of implementation (cost driven), the need for an otherwise impossible performance level (requirement driven), or the cost effectiveness of later upkeep (maintainability driven).
Computational evolution will gradually blur the distinction between the hardware, firmware, and the software. Eventually the software may be implicitly used to reconfigure the hardware to maximize performance. An example of this the implementation of Software Radio in FPGAs where the firmware is the "soft" component of the design and the FPGA provides performance. This firmware level will grow over time to look far more like general purpose software than the other way around. Thus it is far more likely that a general purpose software implementation can exploit future 'firmware capabilities' than the other way around. There are advantages for radio science to this approach due to the nature of signal processing being focused around multiply accumulate and fft like operations in many cases which are currently easy to map into FPGAs. However the mapping of more general operations into FPGAs will eventually result in FPGA integrated supercomputers (e.g. SGI FPGA boards) which dynamically reconfigure for maximal parallelism. At the instrument level the computer will be eliminated in favor of firmware processor implementations on FPGAs that are tightly integrated into the instrument hardware (e.g. microtronix firefly).
The merger of the software into the firmware and instrument hardware should be exploited and prepared for in modern instrument designs. The architectural ideas of a coherence layer which interfaces with the sensor elements will remain the key interface between the analog world and that of the software radar implementation.
Structure of a Software Radar System
[FIGURE : Software Radar Architecture Figure. This is an expanded and more detailed version of the one in the signal processing paper. Incorporate details of the software patterns involved in each element as well as the major category (below) into which each part of the system falls. Use color]. key ideas :
Coherent Interface Layer . Here we need to discuss the requirements placed on a coherent interface layer by the needs of typical radio science data systems. This includes alignment in time, stability of frequency, time transport, and global coherence, and the problems which can be caused by by variability in these quantities. We can provide an overview of the typical alignment and stability requirements for monostatic ionospheric radar, multistatic ionospheric radar, beacon observations, spectrum monitoring, and channel characterization. Examples of problems in such coherence can be given of the impact of oscillator drift, UTC alignment desynchronization, and glitching in the information stream (i.e. data loss). Mitigation can be discussed here with the focus on the capabilities which a software radar approach makes possible that would otherwise be impossible. For example clutter locked frequency adaptation for oscillator drift. We can also discuss the need in later stages to tolerate data stream glitches in a robust way. The principle is to assume they will happen but have elements to detect them and remove their impact on the signal processing and analysis chains.
Distribution Boundary. We need to discuss the potential different properties of the boundary between the coherent interface layer and the information transport. The important thing here is not to go into the distributor pattern but to discuss the potential properties and behaviors of this boundary. An example is a system which emits small packets of data versus one that organizes full IPPs and the trade offs in performance, flexibility, and tolerance to data loss in such a system. This is a good place to explore some of our discussions and ideas that occurred in the MIDAS-W format split that resulted in MIDAS-W CW format. A lot that discussion was pretty good and can be added here in a more general context.
Information Transport. The primary idea here is the idea of the network as the inherent transport for the voltage level representation of the RF signals, digitized timing, and system status information. A strong implication of this is that a single network can support many instruments when the bandwidth of the network significantly exceeds the bandwidth requirements of a single instrument. These instruments can be 'real hardware' (i.e. extra A/D converters) or they can be 'meta instrumentation' which results from the synthesis and recombination of information already captured either from a single sensor or via multi-sensor fusion. We need to discuss different potential approaches here including writing to disk and subsequent access, unreliable multicast, reliable multicast, replicated unicast, and hardware bus based approaches (e.g. raceway between FPGAs). We can also briefly discuss URIs here and the use of them to identify the location of information within the system 'namespace'. Thus we will introduce namespaces here briefly but defer their description in detail until the follow on data management paper. People will also probably want some discussion of performance scaling across current technologies but we should keep this type of thing very general as we don't want the paper to be strongly dated a decade or ten from now. Information Latency Systems. These are data recorders, software delay elements, and means of producing time space tradeoffs. We can discuss Terastores here a small bit just to give an idea of how we do data recording. An important concept here however is the use of delay elements to allow the deferal of synchronization decisions in the system. This is essentially trading latency/space for certainty of information and state. It is for example the difference between our batch data processing system and our real time processing system. This trade off also enables different types of scientific analysis. For example JMH's Wavelet analysis of entire scans or experiments produces a science yield that would otherwise be impossible without this explicit information latency and buffering.
Signal Processing and Analysis Systems. The main point here is that these can be any kind of computing system but highly parallel clusters of general purpose computers are very useful. Such parallel clusters play directly into the sharing of computing resources between different instruments. This enables large amounts of computation power to be allocated for specific tasks of realtime or high latency processing. Such reconfiguration can be done dynamically and can exploit Grid computing solutions that have been developed as tools for other problem domains. We can discussion the overall organization of these computing elements into particular signal and analysis chains. The challenges of automating such allocations can be discussed but we will need to leave examples of such systems to later (after we actually implement them... ;).
Visualization, Monitoring, and Introspection Systems. This is the application of computing and software elements to the display and monitoring of what is being done by the instrument and the Software Radar Architecture. We can discuss both the tools we have developed which simulate traditional instrumentation (signal displays, A-scopes), tools for debugging the system, and applications of statistical analysis to large volumes of information from the system to characterize the performance. One thing to highlight here is the 'regeneration' of traditional instrument behaviors and flexibility through the 'virtual' implementations. Another thing to highlight is the difficulty in producing truly general solutions to such visualization elements. We should think about and discuss the interface boundaries associated with these types of tools and try to see if there is a more general approach.
Control Systems. We can discuss briefly several possible control architectures here and perhaps describe in detail a multicast based event architecture which parallels the design and implementation of the voltage transport architecture. It is important to discuss the differences in requirements for control information where often time alignment or latency requirements may be quite important. We can certainly discuss 'adhoc' or 'user controlled' architectures and their benefits and difficutlies (i.e. start it all at the command line!). One important control system to describe is that of the 'expectation' pattern which is used for monitoring the ongoing realtime behavior of the architecture. A very important point is that keeping track of the behavior and state of all elements of a Software Radar system is a very dynamic and difficult challenge. Human intervention and monitoring is the primary approach we have been using but 'agent' based monitoring, notification, and correction will be required to scale up the complexity of these systems. We may be able to tolerate latency but both 'incorrectness' and 'non-completion' are properties we cannot tolerate well in our architecture. It is the function of the control system to address these needs. At some level we also should probably defer a detailed analysis of this area until later. We have a lot more work to do on control systems and the ones we are using are certainly suboptimal in their reliability. They also tend to be quite brittle. We probably need a 'Software Radar Control Systems' paper as well as additional investigations in this area.
Databases and Search Engines. Databases (e.g. Madrigal) and search engines (e.g. voltage level search ; Madrigal global search) are a fundamental part of this architecture. They allow for data mining for both scientific and operational purposes. In the Software Radar architecture they can operate all the way down to the voltage level. We haven't gotten to that with Madrigal (which deals with experimental records) but such search capabilities are possible. We can use examples from the test Search Engine which was built for MIDAS-W data as well as insights from Madrigal here. One of the things we should think about and discuss is the relationship between the "experimental record" and the database. This is fundamental to the design of Madrigal and the relationship between the software radar system, the analysis chains, and the experimental record is really fundamental. We can again mention namespaces here and the organization of the information space into which the software radar commands, status, and data flow. One of the difficulties we can highlight here is the need to "understand" what you are looking for in the data prior to searching. In a human sense this is "having seen enough data" while in a statistical sense this is a statistical characterization of the data along with a principal component type of analysis. We can also discuss correlation based approaches a little bit for finding 'similar' behaviors in the information from the system by using user identified 'subpatterns'. This is again an area we may want to defer to future papers and work to a great extent but some discussion is warranted. We should especially highlight how the Software Radar Architecture differs from traditional approaches which also generate experimental records and place them in a database. One possible example of this is database query driven reanalysis of existing data. More of the detail on this probably belongs in the Software Radar Data Systems paper.
Simulation and Debugging Systems. We can discuss the potential for synthesizing signals at either the voltage or higher levels in the sofware radar architecture for the purpose of validating and debugging system problems. One example of this is the signal injection program I recently put together and I'm sure that Phil has some others from the debugging of the analysis system. An important point here is that the Software Radar architecture allows an essentially infinite possible number of simulation and debugging systems. Discussing the tendancy of the Software Radar Architecture to discover 'problems' in systems could go here. There is also the temptation and ability to produce 'software fixes' and corrections to underlying system problems. This is both a benefit and also a great potential time sink as there is often a reasonable desire to 'fix' all the problems found. This can lead to significant development projects and the guidance here should be to approach such efforts with restraint and a strong evaluation of the cost/benefits involved.
Voltage Level Radio Science Data
[FIGURE : Examples of Voltage level data of multiple signal types. ISR, FM passive radar, AM radio, satellite beacons] key ideas :
Coherent samples of bandwidth limited voltages of the RF spectrum, combined with knowledge of the analog transfer function before the sampler, and translated to the network are the fundamental level of information in a Software Radar System. In most cases the correct representation of this voltage is as baseband IQ data. There are cases when keeping the real IF voltages is appropriate however this imposes a later signal processing stage of mixing and baseband conversion which for large bandwidths is often better done in hardware or firmware.
The data is distributed at this lowest level and with minimal processing to maximize the potential value which can be extracted from it. When possible the full retention of this voltage level data provides the best possibility of applying new techniques for analysis and the correction of problems which may be discovered in the instrumentation or subsequent signal and analysis chains. It is not always necessary to process all the voltage level data in real time for many applications. Some applications only require 'best effort' rates to be quite useful.
Voltage level representations can be organized in different ways. For example single uninterrupted blocks, framed for a radar IPP, or a packetized representation. We need to discuss the different trade offs of such implementations here and where they tend to be important in the signal processing and analysis chains. Producing different voltage level representations may sometimes be necessary for some applications.
Voltage level data formats and interface APIs are one place where effort and care should be taken to ensure long term accessibility and ease of use.
The voltages by themselves are not very meaningful. It is very necessary to have significant meta information about both the analog transfer function prior to the voltages as well as the particular performance characteristics of the A/D portion of the system. If digital down converters are used it is necessary both to understand the digital transfer function as well as the analog one. Thus the true difficulty of a fully general voltage level data format is that it must capture the full extent of the signal chain and experimental hardware prior to the samples being generated. The only fully general solution for this is to utilize revision control of the implicit information structures and then use hyperlinking to associate a particular voltage level stream with the surrounding meta information. There is a balance between what information goes “in with the voltages” and what information is indirectly available through hyperlinks. We can discuss the tradeoffs here but I think that the fundamental idea should be that the coherence quantities (time alignment, acquisition location) and the net RF channel characteristics (center frequency, sampling rate, sampled bandwidth, dynamic range of the representation) are quantities that are sufficiently valuable to transport along with the voltage level data. We actually put hyperlinks into the MIDAS-W formats but didn't really use them because the surounding version control infrastructure for the rest of the signal and analysis chains doesn't exist in a real form at this point.
Control and Data Management Patterns
[FIGURE : Diagram of the different software radar patterns and their categorization.]
key ideas :
Each pattern should be elaborated and described. The list of patterns includes : distributor, listener, recorder, replayer, filterer, delay, trigger, bridge, proxy, weaver, others? We should be stylistically compatible with our type of descriptions in the last paper. We already described generators, selectors, transforming elements, and consumers in the context of the signal chain pattern. We may want to say more about these patterns here.
Persistence Management
key ideas :
Persistence Management is about how a software radar system stores and organizes its information state as a funciton of time. It is about both the storage of the objects which make up the state of the system as well as the revision control of the implicit information structures associated with the system.
Good persistence management is one of the features which we would use to distinguish a fully “production ready” system from one which is not. The concept of a “production” radio science system is one which we may want to explore and elaborate in more detail in this paper.
It would be nice to have some good examples of where a persistence managed system can have a problem addressed or corrected. We have been using version control of the software at some level to provide persistence management of our implicit information structures. This is kind of the brute force approach.
We are going to detail our approach to Persistence Management in the Radio Science Data Systems paper. However the concept itself is part of the overall architecture regardless of its actual implementation.
It would probably be good to discuss the importance of a good I/O interface layer here which isolates the I/O calls in the software from the details of the transport mechanism.
The implicit information state of the the system should be separated as much as possible from the code and archived in a version control system.
Performance and Capacity Scaling
[TABLE : Comparison of Software Radar Implementation performance. Computing power, network bandwidth, memory, disk storage, RF bandwidth, number of independent RF channels, realtime processing capabilities, multi-instrument capabilities]
key ideas :
What level of performance and capability is needed to implement particular types of radio science systems using Software Radar techniques? What kind of network bandwidth is necessary and how much data can we practically record and manage? Point out that the internal switch bandwidth available in a network switch is vastly higher than the bandwidth of the individual channels. Using IGMP this allows one switch to support far more producer/consumer pairs than would otherwise be possible. Discuss the possibility of architectures where the firmware of the network switch is changed to perform computation on the data packets. This approach could be very useful for some types of high performance realtime applications (e.g. beamforming). How much computing power is really needed to manage the data, monitor, and control things? What kind of latency can be tolerated for different applications? A table showing the basic factors and performance levels for the systems we have constructed would be a good starting point (i.e. Prototype MIDAS-W, production MIDAS-W, EISCAT-Svalbard, MIDAS-M, EISCAT-Imager, etc.)
A discussion of applications other than ISR would be a good thing to have here. In particular I'm sure we can do any of the acquisition needed for passive radar, beacon tomography, scintillation, and spectral monitoring. Multirole instrumentation is being tried with MIDAS-M and this is one of the strengths of our approach. Discussing how much capacity is needed in a system for these different applications might be useful to those considering our techniques for their applications. One area where we may not be competitive is performance per watt. This is only going to be an important metric for distributed instruments in remote locations. It will improve with time but there are still areas where it is better to use a cheap DSP or FPGA.
Another important point to discuss here are trade offs in cost versus reliability. For our architecture it is easy to add redundant mechanisms to increase reliability. These do come at a cost and it should be discussed. For example it would be possible to build a system which multicast on two networks and where everything happened redundantly after the A/D converter. Interestingly the cost would be fairly linear for such a system.
What lessons do we have for software radar systems which are implemented with different interconnect systems? For example if we built such a system using multiprocessors attached to a shared serial bus we could use a fairly similar architecture. How about using USB 2.0 as an interconnect. Wireless mesh networks?
We should also think carefully in this section about what people might want to do with this architecture which we haven't already explored.
Supercomputing and Radio Science Instrumentation
[FIGURE : An illustration showing multiple Software Radar based instruments sharing computing infrastructure.]
key ideas :
The Software Radar architecture allows the deep integration of computing and networking into the instrumentation itself. By integrating general purpose software as far down into the instrument as possible we gain the maximum flexibility. The representation of the data and system state are at a relatively low level and this directly enables the use of the information for purposes beyond its original intent.
Additional capabilities can be added to a Software Radar system by adding additional software and computing elements. It is the software development time which becomes the strongest contraining factor. Even very large amounts of computing power can be used in some contexts. This opens instrumentation to the domain where Supercomputing can be easily applied for producing scientific benefit.
The Software Radar architecture becomes more capable with time and hardware driven changes are at the periphery of the system. Improvements in computing and networking technology have already resulted in substantial gains in capability from the early version of MIDAS-W. While this trend will eventually reach an end it will be at a point of substantially greater capability than is currently possible.
The sharing of computing resources between multiple instruments is enabled. Instruments can use resources collectively and in a manner which optimizes different types of goals (i.e. realtime resolution, user feedback, number of independent channels of observation, etc.). Meta instrumentation can also be synthesized by the combination of instrument information streams and via sensor fusion of these streams. This can potentially be done in