A Product is a group of files or content concerning seismic events. Examples of product types include ShakeMap, Moment Tensor and Origin. A product usually consists of a directory of files, but may also include a stream of bytes, links to internet resources, and key/value pairs. Products provide a standard method for bundling textual and multimedia data, where each product is uniquely identified by source, type, code, and update time. These key attributes are described as follows:
source and type
combination.source, type, and code,
but different update times are different versions of the
same product.While some products can be associated with a single seismic event, others are not. The harmonic tremors detected by a particular helicorder, for example, could easily have originated from several seismic events.
Products are created by USGS and non-USGS clients, distributed using PDL, and utilized by various USGS and non-USGS automated systems such as Real-time Earthquake Maps, the Comprehensive Catalog of seismic events (Comcat), ShakeMaps, the PAGER earthquake alert system and the Earthquake Notification Service. PDL is designed to support both existing and future needs for seismic information.
PDL focuses on distribution and triggering, and integrates with other systems for processing. After a product is created it is sent to a distribution hub. The distribution hub creates a notification that includes a URL where the full product can be downloaded. Clients connect to the distribution hub and receive notifications about available products. Based on configuration, a client may choose to download and process a product.
How received products are utilized is very system dependent. PDL provides two primary alternatives: a java API for direct integration, and a command line api for applications to be executed when products are received.
Products with the same source, type, and code, but different update times are considered different versions of the same product. Later updates to the same product replace previous versions, where older versions are retained according to PDL's configurable archive policies.
A brief status code. Typically UPDATE or DELETE.
Any value that is not DELETE is considered an update. A product
status may have different meanings depending on product type. "DELETE" always
has the same meaning: that a product has been cancelled.
Properties are optional name/value pairs that describe the product. They are used to quickly identify characteristics of a product without needing to parse its content. Examples include the "alertlevel" property of the PAGER product, and the "magnitude-type" property of the origin product.
Name/value pair definitions of product properties are generated when a client system sends a product to a hub. Properties can originate in two forms: as parameters in PDL Java command-line execution, or through a java process using the PDL API.
Links are optional and are a URI with a relation that describes how the resource is related to the product. Because the link is a URI, it can be either a URN or URL. One example would be including a "background" related link for a product. Another example would be identifying a related product that was used as an input.
A Notification is a small message with Product's source, type, code, update time, URL for download, and URL expiration date. Notifications are broadcast when a product becomes available for download. Typically client systems use notifications as a basis for making product download requests from hubs.
When a source deletes a product, the delete is technically equivalent to sending a product, except the status is now "DELETE" and contents of the sent product are optional. This deleted version of the product is distributed the same way as any other update.
Product Distribution provides a standard solution for data distribution.
Data producers and consumers are loosely coupled. Producers broadcast a product, via redundant hubs, and all clients are notified it is available. Only clients that are interested download the full content for processing. This simplifies both sending and receiving data because all producers and consumers only need to communicate with the hubs. Because producers and consumers initiate connections to the hubs, most users do not need to reconfigure firewalls.
Producers use a private key to sign products when they are sent. Hubs verify this signature using a public key before distributing products to ensure only authorized producers are able to send products. Clients may optionally verify these signatures as well.
Clients can be configured to trigger processing when new products arrive, either using command line arguments or a Java API, eliminating polling latency. In many cases this has reduced processing time from minutes to seconds.
The Product Distribution Layer (PDL) is an improvement over an earlier system called the Earthquake Information Distribution System (EIDS). EIDS distributes XML products, but does not support large files or binary content like images. While PDL uses EIDS as an option for distributing notifications, it improves on EIDS' XML mechanism by providing for any type of content, including images, large files and whole directory structures.
The following diagram illustrates the creation and distribution of a product:
The PDL Indexer associates related products into events. The indexer uses authoritative regions to choose preferred products when multiple sources contribute the same type of product for the same event. It replaces an earlier indexing system named QDM (Quake Data Merge), which processed fixed-length CUBE messages. The Indexer:
PDL's processing model provides several distinct advantages:
A NotificationReceiver receives Notifications for products and keeps a NotificationIndex of all received notifications. Whenever a notification is received, it is queued for processing by each NotificationListener. Each listener has a separate queue, and uses a separate thread to process queued notifications one at a time. This prevents a slow listener from blocking a fast listener.
When a listener processes a notification, it may request the product from the receiver. Any number of listeners may request the same product, but the receiver will only download once. Once the receiver has retrieved the product, the listener processes its contents.
Periodically, the receiver will remove expired notifications from its index, and downloaded products that are no longer needed. By default a receiver only keeps downloaded products for 15 minutes, just long enough for listeners to process, to minimize disk usage.
ExternalNotificationListeners process products by executing an external command line process for each received product. The ExternalNotificationListener does not associate related products together, for that functionality use the Indexer.
See Receiving Products for more information
Every time the Indexer changes the index, it notifies IndexerListeners about the change. IndexerListeners are similar to NotificationListeners but listen to Indexer changes (which are triggered by products) instead of listening to notifications for products. Because the indexer keeps track of previously processed products, duplicate and non-preferred information can be filtered. Additionally, updates that do not change an event's location or magnitude can also be filtered.
See Indexer for more information
A SocketProductSender sends a product directly to a ProductHub, which stores the product and sends a Notification to all connected NotificationReceivers. Product hubs register a public key for each sender, and verify product signatures to ensure only known senders may distribute products.
See Sending Products for more information
Custom Notification and Indexer Listeners can be integrated with PDL using the Java API. Other languages need to use the ExternalNotificationListener and ExternalIndexerListener.
NotificationListeners must implement the gov.usgs.earthquake.distribution.NotificationListener interface. Most users can extend the gov.usgs.earthquake.distribution.DefaultNotificationListener class.
IndexerListeners must implement the gov.usgs.earthquake.indexer.IndexerListener interface. Most users can extend the gov.usgs.earthquake.indexer.DefaultIndexerListener class.
See the Java API Documentation for more information.