Software Radar Data Systems
Philip J. Erickson (1), Frank D. Lind (1), Tom Grydeland (2), William Rideout (1), and John M. Holt (1)
(1)MIT Haystack Observatory, Route 40, Westford, MA 01886 USA (2)Dept. of Physics, University of Tromsø, Tromsø, Norway
Abstract
We discuss the implementation of Software Radar data systems with examples from the implementation for the signal processing and analysis of Incoherent Scatter radar data. The Software Radar framework enables great flexibility in the processing of radio science data including the application of numerically intensive forms of analysis, reconfiguration of the instrumentation dynamically for optimal science yield, and delayed reanalysis of data in ways which were unavailable or impossible at the time the data was acquired. We also describe the general framework by which such data systems are organized to manage large volumes of data in a production environment. These more general structures include the Radio Science Object Web, the use of Namespaces for persistence management, the choice of representations for experimental records, and the application of Grid computing.
Introduction
The Radio Science Object Web
key ideas :
We have mentioned persistence management of the implicit information structures in a radio science system previously. The Radio Science Object Web is the abstract representation of these structures as a distributed collection of hyperlinked objects that will evolve in time. Here we are starting with the information content and relationships in the radio science instrument with an eye towards building systems where this representation is explicit rather than implicit. Understanding the techniques involved in how to represent, manipulate, persist, and search the Object Web is necessary for the construction of robust and highly flexible Software Radar Data Systems.
The Radio Science Object Web is at any instant in time the distributed collection of hyperlinked information associated with an experimental system. This web will change dynamically as information is created, transformed, moved, replicated, and destroyed. The objects in this web will be composed primarily of XML and binary information. These objects will represent more than the data produced by an experimental system. The objects will contain all the meta information associated with the architecture of the system, the organization of hardware and software elements, debugging information, command and status messages, and profiling information on the performance and throughput of the system. Explicitly capturing this meta information as well as the data provides an expanded basis for understanding the experimental process and for controlling complex instrumentation. It is also the foundation for visualizing and automating an experimental system using Software Radar techniques. This is in contrast to systems where many of the relationships between the data, software, and computing architecture are implicit. Systems where the Object Web has an implicit representation do not fully record their activity and the relationships between different elements of the web are usually known at best by understanding the detailed implementation (hardware and source code) and at worst only by the people involved in the construction and operation of the system (job security ;).
A large part of realtime processing, control, and data analysis in a scientific radar system is actually centered on the recording and timely retreival of information. Real time data streams can be processed, transformed, and displayed but without data recording at some level the system accomplishes very little. The level at which information is recorded in an experimental instrument is a strong function of the requirements and capabilities of the instrument (as well as of the budget used to create it). With the bandwidth and information density limits of current technologies it is always possible to construct a system in which so much information is generated that recording it for any significant length of time is impossible. Such a system may still be able to process the information flow in real-time and reduce it in such a way that it can be useful. Many radio science systems, however, do not have such high performance requirements when compared to existing technologies. For these instruments it is highly advantageous to record "everything" that occurs in the system. So, for example, a radar system will generate the control signals it sends, the responses of its hardware components, and the resulting samples from its receivers. In some modes the instrumentation may also generate a large quantity of debugging information. This information can be multicast to a network (thus making it widely available for real time processing and visualization), but it can also be recorded and made available for later analysis, high latency processing, and inspection. Real time processing can be very useful for providing immediate results and feedback to system operators. However it is often useful for the recorded output of the information to be used in constructing the experimental record. This stems largely from the ability to use the complete knowledge available after an experiment is complete to optimize the analysis and processing in ways that are not possible in realtime. Thus it is often the recorded state of the system that forms the basis for generating scientific results as well as for understanding the detailed operation of a given radio science instrument retrospectively. In radio science systems the generation of the experimental record is often accompanied by a large reduction in the overall information content. This is because the bandwidth and information density of the people who use radio science instrumentation is not really changing significantly. Retaining the entirety of the underlying information is useful when it is necessary to expand on or investigate a particular state or set of characteristics observed in the output of the instrument.
In general the recording of large quantities of data will occur in the filesystems of computers that provide hosting for elements of the Object Web. It is possible for performance reasons to construct systems where memory is used for this representation and there are significant performance advantages to this for applications that might need to numerically analyze all the data as a complete set. The computers implementing the recording will be connnected to the network but may be physically distributed in space and have different connectivity qualities (bandwidth, reliability, and latency). In many cases it may be advantageous to record information locally. This might be for I/O performance reasons, to preserve bandwidth, or simply to logically partition an experimental system. Other times it may be better to multicast the data and have a software process that records the information remotely (e.g. to a large redundant data store). In either case a standard means of naming and organizing data in the Object Web allows for the creation of automated tools that are capable of manipulating data objects in a generic manner.
Radio Science Namespaces and Persistence Management
key ideas :
The Radio Science Object Namespace is the information space associated with elements of radio science systems. Members of the namespace are used to construct paths that refer to elements in the Object Web. In general an individual instrument will define its own Namespace but there may be occassions where several instruments agree to maintain a common Namespace definition that allows them to share control, data, or processing resources.
The purpose of the generic Namespace structure is to impose a sufficient degree of uniformity and explicit definition on a Radio Science system to allow its state and data products to be manipulated using automatic tools. In particular it allows the use of version control systems to track changes in the organization of the information associated with an experimental radio science system. It also enables the construction of search engines which can use the Namespace structure to limit the required search space. This can greatly speed such searches for systems which have real time performance or responsiveness requirements. The physical representation of the structure is also easily manipulated by humans directly and this allows for debugging and straightforward fault correction using scripts or manual methods. The web structure defined by the Namespace can also be easily extended as necessary and Objects can even be migrated between Namespaces using an appropriate mapping template.
The Namespace is organized using a a hierarchial structure consisting of a set of directories followed by optional date and object fields. When a Namespace element is given without an fully resolved object it is assumed that the portion of the namespace refered to includes all objects below that point in the Namespace. This is not normally used in the definition of the namespace but can be quite useful for manipulating existing Object Webs (e.g. processing all data on a particular date). Note that the Namespace is defined statically so that even though elements of the namespace may be specified using an arbitrary path structure the depth and organization of this structure is fixed in the definition file. This limits the scope of the Namespace and prevents arbitrary dynamic reorganization of a system which can be hard to synchronize and track. Each computational element of an Object Web will keep an index file in its root path that defines the structure of all valid namespace entries to which it will respond. This definition is made using XML elements that reflect the allowed objects which can be present in the Namespace. This file can be obtained simply by accessing the HTTP server for the computational element and using the default path and to retreive the index. This Namespace configuration information allows a software entity to "discover" the structure of the system simply by being told the network address range of the system's computational elements (i.e. web crawling).
Namespaces are very general and this makes them hard to design. This is a real problem which deserves discussion. It mainfests itself when you sit down to actually build a namespace and find that there is really a pretty large range of stuff that needs to be specified for any kind of real radar system. You next wonder how to organize things and are faced with a bunch of choices where it is not always clear what the benefits and problems are going to be with a particular structure. This is largely an issue that arises from producing a representation framework for something very general and complicated (i.e. the organization of information in a radio science system). I'm hoping that this is really a problem more akin to the statement "making a complicated set of web pages is really hard" but once you build the right tools and see a lot of examples it gets easier. I'm also wondering if there is a higher design principle lurking here to make this process easier.
Another more subtle problem comes when building a Namespace for a real system where it is necessary to widely split the information that a particular object in the web might be producing. For example when representing some antenna positioning software in the program you might end up with something like :
system/hardware/antenna/pointing/command/ system/hardware/antenna/pointing/status/
where the above portions of the object web hold information on the state of a radar antenna. Associated with generating this state might be a software program that might need to have settings, status information (i.e. I'm reading the antenna pointing at 10 Hz), and debugging information. This might end up in the namespace thus :
system/software/antenna_position_reader/configuration system/software/antenna_position_reader/status system/software/antenna_position_reader/debug
Now the problem arises in that the objects under the system/software/antenna_position_reader have an implicit relationship to the objects in the system/hardware/antenna/pointing portion of the object web but this relationship is not explicitly contained in the namespace itself. The lack of this kind of relationship is at least partially related to the fact that we are using a tree structure instead of a fully hyperlinked structure at the Namespace level. The relationships could very well be specified in other ways. For example attributes of the namespace could be used to provide secondary hyperlinking. More likely is that the relationship is going to remain implicit and ultimately this means it gets coded into the software that interacts with the namespace (i.e. in the above example the source code of the antenna_position_reader program knows where to put things).
This isn't a terrible problem but it was bugging me because when you sit down and write a namespace you find yourself making these related structures but not having a good means of representing the relationships. I'm not sure there is a particularly clean solution to this problem that doesn't break some of the namespace advantages or greatly increase the complexity of the tools which are needed to mainpulate the namespace. If anybody has a good idea on solving this I'd like to hear it...
A Namespace will evolve in four primary ways : growth, pruning, refactoring, remapping.
How a namespace is going to evolve with time and use. How this growth and change impacts the software which uses the namespace is one question that arises from this issue. Another question regards the kind of tools that are required to help this kind of change occur easily and constructively. Note that I'm refering to changes in the Namespace itself here and not changes in the Object Web itself. I've identified four major ways a namespace is likely to change : growth, pruning, refactoring, and remapping. If I've missed one let me know.
Growth is where you add new elements to the namespace. This is likely to be the most common form of change for the namespace especially since we all know how scientist like to tinker with their experimental systems and software developers are even worse in this regard (just imagine what scientists who develop software are like ;). This kind of change is usually going to have a fairly low impact on software which has been previously written. New elements of the Namespace will just be ignored by programs which don't know about them. If enough elements are added to a system it might have some impact on the real time performance of activities like searching a namespace but this is a system level design decision that should be considered (i.e. if and how to add new elements).
Pruning is where you remove old elements from the namespace. This activity has a lot more impact on the namespace and the programs which manipulate it. In particular some software may depend on the old namespace elements and unless this software is also modified it may fail. In a large software system this may not be trivial and it could even be hard to find all the possible locations which need to be changed. This also creates a situation where information recorded using the old namespace cannot be manipulated with the new one. This requires that old object webs be processed using the old namespace or that a migration of the data to the new namespace be done. Depending on the nature of the changes this type of migration has a good chance of being impossible or at least unreasonably complex. One way to reduce the impact of pruning on a namespace is through the use of revision control. If the namespace is maintained in a revision control system (e.g. subversion ; we need directory manipulation) then software dependencies can be resolved by incorporating the namespace revision into the information retrieval. This can be costly in terms of data storage capacity and so true pruning where data is eliminated will be necessary. Another approach to help mitigate pruning changes is to use a search engine in software which needs particular classes of objects but where potential namespace brittleness might be a factor.
Refactoring of a namespace is where a subportion of the space is moved to another position in the tree. It is likely that this type of reorganization is going to be fairly common. Software which depends on the absolute position of objects in the object web will need to be modified to support this type of change. Software which obtains object URIs via a search engine like system will not be critically impacted so long as the search criteria are not changed by the refactoring. For this type of change data in a prior namespace structure will not be processable using the new namespace. However for refactoring it is fairly easy to support migration of data from an old namespace format to the new one. This could even be done using an automated mapping tool that used a definition of the mapping between the spaces to automate the migration of information from the old object web structure into the new in a one to one mapping.
Remapping of a namespace is where a subportion of the space is moved to another portion the tree in combination with growth, pruning, and possibly with a reformatting of the underlying data into new object types. This is the most complex type of modification to the contents of a namespace and will almost always be a special case. It will most likely produce a significant impact on software systems which supported the previous namespace and will probably necessitate rewriting their code in a major way. It is possible to create some tools to support or at least aid the automation of this type of namespace change. In particular an automated mapping tool might allow a many to one mapping with an associated reformatting tool (kinda like a mime type for information translation) to unify the underlying information from the old object web prior to writing it to the new one.
When a Namespace exists across more than one authority a search engine is required for a clean design. The idea that limiting the Namespace to a tree structure to ensure fast searches is something I've talked about repeatedly. What has started to become clear to me is that a major way of interacting with an Object Web and a Namespace is through a search engine and that this is really a requirement for a clean design in a distributed system. In particular while many programs are going to know exactly where in a namespace they are going to wish to create and read objects there is a great advantage to obtaining the needed URIs through search criteria. In particular the judicious use of search criteria can result in programs that are relatively insensitive to the actual structure of the namespace. For example a program could query a search engine based on a namespace attribute for data with a particular attribute (say 'IQVoltageData') and in a particular time range (say January 2003). The search would return a list of URIs which satisfied the criteria and where the search engine would have been instructed in its configuration which authorities (i.e. host machines) to search for the data. If a system had several distributed receivers each generating data in a different part of the Namespace this query would return the fully resolved URIs for this information distributed among all the hosts. A control program might decide what information to move based on such a query (e.g. move data to a centralized host or initiate multicast playback) while a processing program might decide how to configure its processing based on the locality of the URI (e.g. if all needed data is local the program uses a file: transport while if data is remote it uses uses http: for processing).
Thus to write really general programs using Namespaces that are tolerant of namespace changes and able to work cleanly with distributed systems it is necessary to use a search engine and an associated web crawler. It may be possible to adapt off the shelf tools (search engines and crawlers) to this type of application but for searches involving Namespace attributes we are probably going to have to build at least a custom query interpreter to pre evaluate some portions of the search and the limit the query that goes into the search engine itself. It will also probably be useful to have a web crawler that can use Namespace attributes to know what parts of an Object Web are likely to change rapidly. This could help the crawler focus its update activity on the Namespace subtrees (and potentially authorities through outside information; i.e. a list of where portions of a given Object Web are hosted) that have the fastest rate of change. Another aspect of this is that for some searches the returned results will not match the expectations of the client (e.g. A program expects new data in the Object Web but it isn't returned in the search). In these cases it will probably be necessary to allow for the client program to force the search engine to re-crawl the appropriate portion of the object web to confirm that the needed information does not exist. This would kind of be like not finding something you wanted with google but then being able to go activate its underlying web crawler and target it in the likely location in order to get you the links you wanted.
A particular object in a Namespace can't be thought of independently of the path through the tree to that object. This is a point of confusion that came up a couple of times. In discussions of how namespaces change and objects migrate between different Namespaces it is never right to think of a particular object independently of the path in the namespace at which it resides. Thus you don't move just the leaf node between two object webs, you move or transform the entire path from the root of the web to and including the leaf itself. Namespace migration tools are probably necessary to aid in managing the exchange and migration of Object Webs between Namespaces. Examples of this for a single Namespace system are discussed above and this is an area where a lot of interesting investigation could be done. In particular the data remapping operations from one namespace to another could be used to extract information from one instrument (e.g. an ISR) and format it for another system (e.g. a real time ionospheric model). This is more sophisticated than simple namespace version control and of course can get arbitrarily complex. Another possible example is the unification of data from a large number of separate instruments (e.g. distributed receivers) into a larger meta instrument namespace (e.g. coherent imaging on a continental scale). I have a feeling that this could be a very powerful capability and reason for using namespaces. It will also take a lot of work to develop it in a generally useful manner.
Object Web Transports
A general discussion of the pros and cons of ways to move data about in the Software Radar is the idea here. We have primarily been using unreliable multicast for the raw voltage level data but then applying other approaches (http, file, and xmlrpc) for later processing stages that need greater reliability. Discussion of how the transports impact our ability to make the processing robust and reliable would also be a good idea.
The communication of information from one computational system to another relies on the use of one or more underlying transport protocols. The transport protocols provide well defined functionality and a simplified software interface for communicating with different levels of flexibility, relibability, and security. The RSDP uses three primary transport protocols for its operations : The Unreliable Multicast Transport Protocol (UMTP), the Hypertext Transport Protocol (HTTP), and direct File I/O (FILE). A fourth protocol, the Reliable Multicast Transport Protocol (RMTP), is envisioned but will be implemented and defined at a later date. UMTP is defined separately (see the UMTP Definition Document) and provides efficient, stream oriented, one to many communications protocol that is useful for distributing data and control information. HTTP is defined by the World Wide Web Consortium and is the basis for Web based information communications. Use of HTTP by the RSDP allows for easy integration of radio science systems into the Internet infrastructure, an easy way for resolving hyperlinks, and a means of reliable information transfer where real-time performance is not the most critical factor. FILE I/O is of course at the basis of data storage in most modern operating systems and this transport method has the advantage of being associated with persistent storage devices.
Object Web URI Generation
Uniform Resource Indicators are the means by which we identify elements within the object web. The URI is constrained by the Namespace so that only URIs which conform to the Namespace for a particular instrument are meaningful to that instrument.
A Uniform Resource Indicator (URI), as defined by the W3C (RFC2396), is a "compact string of characters for identifying an abstract or physical resource." The identification of information in the RSDP will be done using URI conventions, refer to RFC2396 for a detailed specification. From RFC2396 the generic structure of a URI is defined as :
<scheme>://<authority>/<path>?<query>
In the RSDP three basic schemes will be used for URI naming of Object Web elements. These three forms correspond to the two transport protocols that will be used. Thus URIs of the form :
umtp://<authority>/<path>?<query>
will utilize the unreliable multicast transport protocol for resolution of the URI, while URIs of the form :
http://<authority>/<path>?<query>
will follow the standard HTTP URI definitions and URIs of the form :
file://<authority>/<path>?<query>
will refer to object web elements hosted in a local file system.
In all cases the <authority> will be a computational host in the Radio Science Object Web which is running an appropriate server and the <path> will resolve to the path of the element of the Object Web in the file system on the given computational host. The <path> definition is independent of the scheme used to refer to and transport them and is defined by the Radio Science Object Namespace. The particulars of a <query> are defined by the individual transport protocols. In the case of HTTP this is highly dependent on what services are hosted by the web server. In most systems it will be necessary to have a Naming service which is used to map elements of the Object Web to particular servers and which also provides the ability to deterimine the <authority> associated with a subpartition of the Namespace. Alternatively the namespace itself may contain the information needed to perform the mapping of subpartitions to hosts so long as the structure is relatively static (least dynamic) or a list of hosts can be searched for the needed portions of the Object Web (most dyanmic).
Object Web Representation
By using standard filesystems to organize as well as store the elements of the Object Web we can exploit the tools built to manipulate these systems (QUESTION : Would we be better off using a version control system instead of a file system to manage the storage? i.e. objects in the web would be created and then checked in under subversion control). Such a structure is also easy to inspect and manipulate if it becomes necessary. Taking this approach to the organization of information in the Object Web comes down to conventions for structuring of individual filesystems in the context of the larger distributed system and for refering to elements of the filesystem in a well defined and predictable manner. It is also necessary to address issues related to the migration of data in the Object Web from host to host and the merging of related information webs.
A form of EBNF is used to define the Namespace used for Object Web Representation. This grammar uses elements which are separated by commas. The use of hard brackets such as [<element>] indicates the element occurs zero or one time. The use of soft brackets such as {<element>} indicates the element occurs zero or more times. Individual characters that are necessary are enclosed in single quotation ' ' marks. A vertical bar '|' indicates an exclusive-or choice in the grammar. Mandatory repetitions are indicated using a number followed by an asterisk sign (e.g. 2*<element> requires an element to be repeated two times). The EBNF for an element of the Radio Science Data Protocol Namespace is as follows :
<rsdp_namespace_element> := <object_path>,[<object_element>] <object_path> := <path_element>,{<path_element>} <path_element> := <string>,'/' <object_element> := <time_dependent_element> | <object_name> <time_dependent_element> := <iso_date>,'/',[<time_dependent_object_name>] <time_dependent_object_name> := <prefix>,'@',<longsecond>,['.',<suffix>] <object_name> := <prefix>,['.',<suffix>] <prefix> := <string> <suffix> := <string> <iso_date> := 4 * <digit> , '-' , 2 * <digit> , '-' , 2 * <digit> <longsecond> := ['-'],<digit>,{<digit>} <string> := <alpha> , {<alphanum>} <alphanum> := <alpha> | <digit> | <mark> <alpha> := <lowalpha> | <upalpha> <lowalpha> := 'a' | 'b' | 'c' | 'd' | 'e' | 'f' | 'g' | 'h' | 'i' | 'j' | 'k' | 'l' | 'm' | 'n' | 'o' | 'p' | 'q' | 'r' | 's' | 't' | 'u' | 'v' | 'w' | 'x' | 'y' | 'z' <upalpha> := 'A' | 'B' | 'C' | 'D' | 'E' | 'F' | 'G' | 'H' | 'I' | 'J' | 'K' | 'L' | 'M' | 'N' | 'O' | 'P' | 'Q' | 'R' | 'S' | 'T' | 'U' | 'V' | 'W' | 'X' | 'Y' | 'Z' <digit> := '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' <mark> := '-' | '_' | '!' | '*' | '(' | ')'
Currently it is implied that the <longsecond> is referenced to the common Unix epoch (Jan 1, 1970) and is a 64 bit representation. The <iso_date> representation should allow for more than four digits in the year field but anyone using this protocol in a year requiring more than four digits should feel free to extend the definition in the obvious way.
The above BNF stuff may not belong in the paper but understanding how things are structured in an accurate way will be needed if we want to include detailed examples such as the Very Simple Radar (VSR) below.
Extracting and Archiving Elements of the Object Web
After an experiment is completed (but possibly during the operation of another one) it is useful to replicate the Object Web associated with the experiment from data distributed throughout the instrument and stored in the file system representation. The goal of this archival process is to organize the information produced by an experiment into a form suitable for backup to long term storage media, distribution to end-users, and manipulation by a database. This form is the Archival Web which is an Object Web organized using the normal file system representation but which has a single root path and is no longer distributed across the network. The extraction and archival process may be selective or complete but it involves three basic steps. The first is the retrieval of objects from the different storage locations in the system and their organization into a common file system. This retreival involves not just copying objects but also resolving their internal links and the retreival of the referred information. This process is similar to database building for an Internet search engine and can be implemented using HTTP as the transport for the most part. The second step is the relinking of the information that has been retreived. This involves a modification of the links contained within the data to reflect the new position of objects in the Archival Web and the substitution of appropriate links where the information could not be retreived. Finally a unified index file must be produced and placed in the root of the Archival Web directory structure. This index file provides the complete Namespace elaboration for the Archival Web. The result of the first two steps is a complete Archival Web for an experiment which is strictly self referential. The resulting Archival Web is then self contained and ready for use in other contexts. he third and final step involves the expiration of information within the distributed Object Web. This step deserves inclusion and additional comment because it is most likely not under the direct control of the extraction and archival process. Indeed, if an expiration schedule is used it may very well create a deadline for the completion of extraction and archival in some systems, while others may never need to reclaim data storage at all. Although nothing in the RSDP prevents operation of an instrument in a campaign or experiment driven mode the storage of information is organized structurally with the idea that many systems will be designed to operate using shared resources in an "always-on" manner. This mode of operation is well suited for providing input products into real-time models and data product generators but it also has the consequence that information in the Object Web from a given point in time may be part of many different "experiments". Each of these experiments might be extracted and archived at different times and the system could not expire the data until all of them were complete. Because of the strong dependence of this process on the schedule of the experimental system the expiration of information and its coordination with the archival and extraction process is the job of the instrument scheduling software (or person as the case may be).
Software Radar Namespace Example : The Very Simple Radar
[FIGURE : Diagram of the VSR from my URSI talk]
key ideas :
Here are some <namespace_element> examples to illustrate the organization of information in a very simple radar (the VSR). For simplicity the VSR is assumed to be a monostatic radar system that operates to a simple schedule and which is easily configurable. The VSR is assumed to have a fixed antenna and to produce data which is processed only to range time intensity plots that are plotted as graphics files (PNG). This radar system is assumed to exist on a single server (vsr.openradar.net) and the index file describing the namespace would be located at :
http://vsr.openradar.net/index.xml
for all the following examples the base path of the system is assumed to be vsr.openradar.net and the selected transport protocol is not particularly important. The VSR has configuration information which needs to be stored for the system. This configuration information defines a list of possible experiments and the state of the analog hardware for each experiment (three experiments total). The VSR represents this configuration information using XML data. The Namespace of the VSR represents the objects associated with the data as follows : vsr/configuration/experiments/experiment_list.xml vsr/configuration/experiments/setup/analog/experiment_001.xml vsr/configuration/experiments/setup/analog/experiment_002.xml vsr/configuration/experiments/setup/analog/experiment_003.xml
When the VSR is operating it can accept commands from its user. The only allowed command is to select a particular experiment from those defined in the experiment list object. When such a command is issued by the user the VSR records the information in a time dependent location of the VSR's Namespace. For example if the experiment has been changed several times at over the course of several days the following objects could exist:
vsr/commands/selected_experiment/2003-12-01/command@1070312475.xml vsr/commands/selected_experiment/2003-12-01/command@1070313700.xml vsr/commands/selected_experiment/2003-12-02/command@1070327598.xml
When the VSR finishes changing its experimental mode it records that the mode has been changed in a status object so assuming a 1 second latency for the previous commands the following status objects would be generated :
vsr/status/current_experiment/2003-12-01/status@1070312476.xml vsr/status/current_experiment/2003-12-01/status@1070313701.xml vsr/status/current_experiment/2003-12-02/status@1070327599.xml
In order to process data the VSR must configure its digital receiver. The VSR has a single receiver which operates on two channels simultaneously (channelA wideband, channelB narrowband). Each channel has a fixed configuration for all experiments :
vsr/configuration/digital_receiver/channelA/receiver_tuning.xml vsr/configuration/digital_receiver/channelB/receiver_tuning.xml
The digital receivers of the VSR generate data which is recorded to the local disk as voltage level samples in a binary format. The data is generated in a time dependent manner and is divided into blocks of one minute in size for channelA and one hour in size for channelB (the division is based on the implicit size of the files and is arbitrary). Some example files of voltage level data are :
vsr/data/channelA/2003-12-01/vsr_data@1070312476.raw vsr/data/channelA/2003-12-01/vsr_data@1070312536.raw vsr/data/channelA/2003-12-01/vsr_data@1070312596.raw ... vsr/data/channelA/2003-12-01/vsr_data@1070313701.raw ... vsr/data/channelA/2003-12-02/vsr_data@1070327599.raw vsr/data/channelA/2003-12-02/vsr_data@1070327659.raw
and so forth for data from receiver channelA while a subset of data from channelB might look like :
vsr/data/channelB/2003-12-01/vsr_data@1070312476.raw vsr/data/channelB/2003-12-01/vsr_data@1070316076.raw ... vsr/data/channelB/2003-12-01/vsr_data@1070313701.raw vsr/data/channelB/2003-12-01/vsr_data@1070317301.raw ... vsr/data/channelB/2003-12-02/vsr_data@1070327599.raw
Now the VSR requires a signal processing program to analyze its data and produce an RTI. The RTI can be produced from either the wideband data (channelA) or the narrowband data (channelB) at the selection of the user. The user commands are stored in :
vsr/commands/rti_control/2003-12-01/command@1070313701.xml vsr/commands/rti_control/2003-12-01/command@1070313900.xml vsr/commands/rti_control/2003-12-02/command@1070327599.xml vsr/commands/rti_control/2003-12-02/command@1070328000.xml
here the above commands are not recorded periodically but only when the user requires a mode change (four arbitrary times in the above example). In response to the commands the RTI changes mode and records this information in a status file with very low latency:
vsr/status/rti_state/2003-12-01/status@1070313701.xml vsr/status/rti_state/2003-12-01/status@1070313900.xml vsr/status/rti_state/2003-12-02/status@1070327599.xml vsr/status/rti_state/2003-12-02/status@1070328000.xml
The RTI processing program of the VSR generates one RTI ever hour by default. It will generate a new RTI file when an expeirment changes or the user commands a change from channelA to channelB or back. The most recent RTI is also stored in a special object so that it may be directly retreived by external users. These objects look like :
vsr/results/rti/latest/vsr_latest_rti.png vsr/results/rti/2003-12-01/rti@1070312476.png vsr/results/rti/2003-12-01/rti@1070313701.png vsr/results/rti/2003-12-01/rti@1070313700.png vsr/results/rti/2003-12-01/rti@1070313900.png vsr/results/rti/2003-12-01/rti@1070317500.png vsr/results/rti/2003-12-01/rti@1070321100.png ... and so forth.
Finally a VSR user might wish to use his graphical browser software to access all RTI and voltage level data from December 1, 2003. The VSR graphical browser will display RTI for a given experiment and then allow the user to click the RTI to show the underlying voltage level data. This requires coordinating the information stored in the VSR Object Web using the Namespace to access all the objects. The VSR graphical browser uses the experiment configuration to properly label its display by accessing :
vsr/configuration/experiments/experiment_list.xml vsr/configuration/experiments/setup/analog/
it takes status information from under to select the proper data for display from :
vsr/status/current_experiment/2003-12-01/ vsr/status/rti_state/2003-12-01/
and the experiment raw data files under :
vsr/data/channelA/2003-12-01/ vsr/data/channelB/2003-12-01/
along with the RTI images under :
vsr/results/rti/2003-12-01/
to generate and display the plots selected by the user from the correct RTI and voltage level data. In this case the VSR graphical browser only requires the user to input the date and it sorts through large numbers of status and data files using unresolved namespace paths. The browser software doesn't care about the VSR command information since it is not relevant to the display of its data. The VSR graphical browser does require a significant quantity of knowledge about the structure of the VSR namespace in order to successfully undertake its task. This knowledge could be hard coded, provided in a configuration file, derived from the index.xml file in some manner (possibly using additional attribute tags), or possibly deduce by searching through and analyzing the content of the namespace. Each of these approaches would have different trade offs in terms of complexity, performance, and generality.
The construction of Object Webs and Namespaces does not remove the necessity of determining how a program behaves in some manner. It simply provides a framework for locating information in an organized fashion. If the VSR were to change configuration (say by adding a steerable antenna) it would be necessary to modify the namespace to allow for the additional information this subsystem would require. This change could be version controlled so that the manner in which the underlying information structure was changed could be formally recorded. Additionally tools which did not care about the added antenna pointing information could ignore the new portion of the VSR Object Web so long as the poritions of the VSR Namespace they used were unchanged.
The above examples should give something of the feel for the use and generation of objects and paths using the Radio Science Object Namespace. The actually structuring of such Namespaces has a complexity which scales up rapidly with the requirements of a real radar system. In particular designing good (or even optimal) Namespaces for different kinds of radar systems can be quite a design challenge in its own right. This is particularly true for distributed radar systems or those for which the configuration of many software systems must be controlled and tracked (e.g. a full Software Radar Systems for an ISR).
An Overview of Incoherent Scatter Data Processing
[FIGURE : Overview figure of the stages involved in ISR processing]
key ideas :
This section should be a step by step overview of the steps involved in “production level” processing of incoherent scatter radar data. It should proceed from the voltage level IQ representation through to the final fitted CMST data at an appropriate level of detail. In particular we should provide an outline of the different software patterns which show up in our analysis chains. We need to provide an explanation of the difference between "Signal Chains" and "Analysis Chains" or perhaps we don't have any difference in which case we need to understand what is different between what was described in the "Signal Processing paper" and what we are doing to do production ISR analysis. I think a primary thing is that this processing is all about the "Weaving" step at some level. Where we take the asynchronous world and produce a synchronous representation. We have shown that is it possible to process data in earlier papers as well as some of the intermediate products. Putting down a detailed view of the whole process may look a little redundant unless we are careful how we structure the exposition.
Structure and Implementation of Analysis Chains
key ideas :
The idea of an analysis chain goes beyond just the processing of ISR data. We should discuss different examples of analysis chains and try to extract the general principles in organizing such chains here. What boundaries are important? How do synchronization concerns change the way we structure the chains? What kind of mechanisms do we use to identify and filter out bad data or information which would cause problems in later processing stages? How do we partition the analysis chains for performance? What are the principles by which we partition the chains into different stages? What kind of representations are used at the boundaries between stages (i.e. memory objects, files, etc). We need to identify the patterns which are used as well as those which showed up in the earlier paper.
Information Synchronization and Boundary Layer Protocols
key ideas :
The analysis and production of the scientific record is the process of producing a synchronized representation of the information that can be used by a later transformational element to produce useful output. This synchronization step may involve significant filtering in order to select the needed elements. It may also take multiple stages of such synchronization combined with the application of different signal chains and the combination with lower level information in many cases in order to arrive at the final output product. These Weaving steps form boundary layers in the analysis and processing and at these layers a variety of protocols may end up being used to communicate across the layers. It is very important to understand what needs to be preserved at these boundaries and what can be discarded. How these layers are structured also is important when establishing data representations. The data representations at these boundaries may be of use for other purposes than the subsequent processing (i.e. the monitoring of intermediate results, the productions of status displays, or the debugging and profiling of the system operation).
Weaving Pattern Implementation
key ideas :
The Weaving Pattern combines asynchronous elements present in the Object Web of the Software Radar system and presents a synchronous representation of them for use by other portions of the radar system. How do we implement such a pattern? What kind of problems arise in the synchronization? How can periodicities in the sampling lead to problems? What if the later stages expect a level of synchronization or performance which is not actually present?
Organization and Implementation of the Experimental Record
key ideas :
What goes into the experimental record?
How do we structure it?
How do we represent it for long term archival?
In some ways this is probably going to end up being a restatement of some of the ideas that went into the development of CMSTs as those are our current ISR experimental record. One of the weaknesses of the current implementation is that the analysis and signal chains to produce a "CMST" have gotten complex enough that the "command portion" of the CMST is not a sufficient representation of the intent of the experiment or the information structures necessary to produce it. We should think about how to deal with this as it will be an ongoing problem. At some level we need to have an abstract method of representing these systems which can be used to describe the implementation. This could then be version controlled and the experimental record could hyperlink to this information. Such an organization might be brittle in time because of hyperlink migration issues which also are a problem in normal web based systems. One more robust way to handle this problem is to simply take the approach that the only real representation that is of record is the Archival web containing all the information necessary to produce a particular "experimental record". Thus a structure such as a CMST becomes a transient representation of a particular experiment and analysis and while we may keep this representation for convenience it should not be considered as separate from the underlying information which represents it. This is the "storage is cheap" keep everything approach. The dynamic version of this is "database driven reanalysis from the voltage level.".
We should also discuss other possible experimental records for different radio science applications.
Grid Enabled Software Radar Data Processing
key ideas :
"Grid processing " is a current buzzword and we are actually using it primarily as a reasonably sophisticated batch processing system. As a batch system it is nice because the end user doesn't see the complexity of getting access to a lot of computing resources. The overall ideas behind a lot of the Grid techniques are more than just batch processing although production implementations don't really exist yet. The mapping of our signal processing and analysis onto Grid systems may result in our being able to exploit a lot of software infrastructure that is slowly getting developed. This is probably going to be necessary as we scale up Software Radar systems from architectures with less than a hundred processors to systems involving thousands of CPUs. Some of the Grid ideas such as 'grid filesystems' and 'replication services' would be quite useful for us if production implementations existed. So far they don't but we should explore the grid processing ideas in the context of how we might manage the analysis, processing, and search of vast amounts of radio science data in a Software Radar system.
Software Radar Enabled Data Processing Techniques
key ideas :
One of the really wonderful things about the Software Radar approach is the large number of new data processing techniques which can be applied. Some of these techniques cannot be done in realtime and require immense computing resources to accomplish. Others are refinements of existing approaches which are fairly easy to implement in a Software Radar system. We should dicuss and give examples (below) of a number of these techniques as they have been applied in the radar system. Another important aspect of this is that these techniques can be applied to Software Radar data even if they are developed long after the data is collected. By keeping the lowest level of information possible we retain the greatest degree of flexibility in the signal processing and analysis we apply to the data.
One area where the Software Radar is definitely producing "processing techniques" is in the correction of problems which are detected within the systems. This has happened both at EISCAT and at Millstone and is probably a general feature of having access to so much information about the behavior of the system. Some times these are problems with the data itself but other times it can be the detection and correction of problems with other aspects of a radar system (e.g. control system problems such as non-deterministic behavior).
What general classes of such techniques do we find that we tend to create and apply?
One important aspect of these approaches is the need to be careful about which ones have been validated extensively and which ones have not. It is possible to produce two signal and analysis systems both of which have the same intent but which are not equivalent due to a difference in the amount of testing. For example a correlator that has been extensively validated might be considered a "production" correlator while one which has not been tested too much is "experimental" in nature. The structures and architecture we have discussed for Software Radar does not provide a means of differentiating between these two classes at this point. We would of course like everything to be "production" quality but in practice we know that will never be the case. This deserves more discussion here and we can at least note how we go about making the transition in a software radar to a "production level" of implementation and what we do to preserve that against later system changes.
Dynamic Clutter Subtraction
key ideas :
We have implemented phase adaptive clutter subtraction in the current processing system. It would be good to discuss this implementation here and give examples. We probably want to make sure to do some testing with the echotek cards to determine the performance difference with a coherent A/D converter.
Multi-pass Adaptive Reanalysis
key ideas :
Phil did a little study a while back about using an initial analysis pass to estimate scale heights in the ionosphere and then reanalyzing the data in a manner which is optimal given that knowledge. The particular example showed how to improve topside recovery. This is a good place to document this technique as well as to highlight the general potential of the Software Radar for this type of approach.
Short Integration L1-norm Filtering
key ideas :
One of the really wonderful things that has been implemented in the current ISR analysis chain is the use of L1-norm filtering on short integrations to form longer integrations. This has been very successful at getting rid of all manner of outliers in our data and has been a great way for us to eliminate satellite and interferer contamination of our records. We should document this approach and show examples of analysis with and without the technique.
Customizing Data Products using Multiple Parallel Analysis Chains
key ideas :
We currently run multiple ISR analysis chains in parallel to produce single pulse, alternating code, and alternating code with clutter subtraction outputs. This technique of using parallel computing to implement different types of analysis is one of the outgrowths of our architecture and multicasting technique. We should discuss how we implement and manage these parallel chains of analysis. The management of these chains is key since the startup, shutdown, and correct operation of each must be assured. We also may want to include performance metric information to help with understanding how many of these chains it is possible to run in parallel on our current platforms.
Databases for Voltage Level and Experiment Record Level Search
key ideas :
As part of a high school student project we implemented a voltage level search engine which would traverse MIDAS-W CW data and apply several different types of mini analysis chains to make measurements and return plotted results based on user supplied search criteria. This search engine works and is quite nice if you know exactly what you are looking for but is a brute force implementation (i.e. no web crawler building a reduced index of the information for use in the search). This is an important lesson from this effort. We found that a great deal of knowledge about the nature of the voltage level data was necessary to get useful search results. The searching was also very I/O intensive and this was really a killer for its real use. It is probably possible to do better than this effort. In particular using an user "identified feature" along with a correlation approach might be quite useful for finding some things in the data. Techniques such as principle component analysis are also probably going to be necessary to automatically provide the needed statistical characterization of classes that are present in the system. We should document and describe this effort as an example of the possibilities enabled by the Software Radar approach.
Search Driven Data Analysis
key ideas :
We may want to discuss the idea of using the higher level database to drive searches which require experimental data to be analyzed from the voltage level in order to produce the desired result. The most obvious such application is for a user to request shorter integration periods for an experimental record which is currently stored in the database at a fixed resolution. This is probably a good example and it is something which people will easily understand. (i.e. the scientist looks at the data -> wants a different resolution -> queries the database -> database initiates reanalysis from voltage level -> new experimental records are generated -> database delivers experimental records to scientist probably with bigger error bars). Enabling this type of technique is something which the Software Radar approach allows and this is another idea which can highlight the flexibility of our system.