Senders must use their private key to sign products. Receivers may use a sender's public key to verify signatures.
ssh-keygen -t dsa -b 1024 -m PEM -C 'Comment describing key' -f <keyfilename>
Do not assign a key passphrase. This creates two files: a private key named “<keyfilename>” and a public key named “<keyfilename>.pub”.
As with all public/private keypairs, NEVER share your PRIVATE key.
Give your PUBLIC key to everyone who needs to verify your product signatures.
Send your public key and information about your products to jmfee@usgs.gov. You will receive a source
and type that can be used to send products.
This will let receivers know what they can expect. View existing documentation for examples of other product documentation.
Send your products using the assigned source and type.
A product is a specific version of any content produced in response to an earthquake, information about an earthquake, or another product.
A product usually consists of a directory of files, but may also include a stream of bytes, links to internet resources, and key/value pairs.
The source, type, and code uniquely identify a product and its associated content. Update times indicate different versions of the same product, and products with more recent update times supersede products with less recent update times.
source and type
combination.source, type, and code,
but different update times are different versions of the
same product.A brief status code. Typically UPDATE or DELETE.
Any value that is not DELETE is considered an update.
When a product producer deletes a product they send an update
with a status DELETE, which is distributed like any other
product.
Contents are optional, but are usually present when the status is
not DELETE. Each content has a path, which corresponds to a
filename.
Each product may have one piece of content that is a stream of bytes. It has an empty path and isn't stored in a file. Typically this content is read from standard input, and also delivered to consumers via standard input.
A product may include links to other products and resources. Links have a relation, which speciÞes how the resource is related to the product. There may be multiple links for each type of relation.
PropertiesKey value pairs. Each property may only have one value. These can be used to determine whether further processing is required. "Standard" properties include:
All of these properties are optional, but a combination of either
eventsource and eventsourcecode OR eventtime,
latitude, and longitude are recommended for
event-related products for association purposes. The inclusion of any of the
time- or space-bounding properties will make products searchable using these extents.
Producers generate a signature which may be verified by both hubs and consumers. The signature consists of a hash of all product contents and attributes, which is encrypted using a private key and verified using a public key.
Production hubs require a signature, and a keypair must be registered before products are sent.
The information contained in this section is a recommendation, not a requirement however recommendations can be seen as a best practice when generating new products.
Properties should be defined for important attributes of your product. They allow recipients to quickly determine whether they should process a product, without extensive parsing.
For example, eventsource and eventsourcecode
are used by the indexer to associate a product with an event. Other
examples include dyfi number of responses, losspager
alert level, and shakemap maximum MMI.
See the product documentation for other examples of product properties.
It is recommended you include with your product a “contents.xml” file. This file describes all of the contents you are sending with your product. It helps the web page to better display your product and ultimately determines what content users will be able to easily download from the web site.
The contents.xml file format is defined with a contents.xsd file. You must understand how to read XML schema (XSD) files in order to properly generate a contents.xml file.
When mapping products into PDL, it is important to consider how content is generated and updated when deciding what individual products should contain.
For event-related products, the group of content is everything related to that event.
For non-event-related products, the group depends more on how content is generated and updated. For example, station related content might be broken up by station. It's possible that this content would need to be further divided based on update intervals.
A product code must be unique forever, from the same
source and type. Product creators also reuse
the code to update or delete the product.
For event-related products, an eventid makes a good code because
it also must be unique forever. Examples include : us2011abcd
and nc12345678.
For non-event-related products another ID scheme is required. Timestamps are often useful in this situation when the products are related to a time window.
An internal database key can be used.
java -jar ProductClient.jar --send [send arguments]
The Command Line API is used to create new products, and by external listeners to receive products.
These arguments are mandatory.
us, nc, and ci.
shakemap, pager, and dyfi.
us2009abcd and nc12345678 --trackerURL=http://ehppdl1.cr.usgs.gov/tracker/ These arguments are optional.
2010-01-14T14:11:28.691-07:00.UPDATE. Product generators
may use any status without spaces. However, the status must be used
consistently for all products of that type. UPDATE, and DELETE.
--status=DELETE. This
updates the status to delete. Product generators may send content
along with a "deleted" product. 2010-01-14T14:11:28.691-07:00 and 2010-01-14T21:11:28.691Z.
--property-eventtime=<eventtime>.--property-latitude=<latitude>.--property-longitude=<longitude>.us, nc, and ci.
2008abcd and 12345678.--property-magnitude=<magnitude>.--property-depth=<depth>.--property-version=<version>.us2008abcd. --property-eventid=<eventid>.<uri>
must be a valid URI as defined by RFC2396 or RFC2732. --content. Content mime-type
for content read from standard input.<file> is a path to a file that
is product content. The file's name and modification date are
preserved. The mime type is inferred from the file extension. The file
is added at the root level of the product.<directory> is a path to a
directory that contains product contents. Any directory hierarchy
within <directory> is preserved.Sending a new product is the same as sending an update to a product.
Line continuation characters (\) added for readability, all arguments would appear on one line.
% java -jar ProductClient.jar --send \ --source=us \ --type=dyfi \ --code=EVENTID \ --directory=DYFIPRODUCTCONTENTS \ --privateKey=PRIVATEKEYFILE \ --eventsource=EVENTNETWORK \ --eventsourcecode=EVENTCODE \ --eventtime=2010-10-25T15:10:24Z \ --latitude=12.345 \ --longitude=23.456 \ --property-num-responses=17000
In the above example, a dyfi product is sent. Source, Type, and Code uniquely identify this product. Directory contains all the product files. The Private Key is used to sign the message. Event properties eventsource, eventsourcecode, eventtime, latitude, longitude are all included. In addition, a dyfi specific property "num-responses" is included that indicates how many people have filled out the dyfi questionaire.
To delete a product, reuse the same source, type, and code, and set the status to DELETE using the --delete argument.
Line continuation characters (\) added for readability, all arguments would appear on one line.
% java -jar ProductClient.jar --send \ --source=us \ --type=dyfi \ --code=EVENTCODE \ --eventid=EVENTCODE \ --delete \ --privateKey=PRIVATEKEYFILE
In the above example, a dyfi product is deleted. Source, Type and Code uniquely identify which product is being deleted. The --delete argument sets the status to DELETE. The Private Key is used to sign the message.
Product Senders use an OpenSSH DSA keypair used to sign and verify their products.
ssh-keygen -t dsa -b 1024 -C 'Comment describing key' -f <keyfilename>
Do not assign a key passphrase. This creates two files: a private key named "<keyfilename>" and a public key named "<keyfilename>.pub"
As with all public/private keypairs, NEVER share your PRIVATE key.
Give your PUBLIC key to everyone who needs to verify your product signatures.
The private key is used by the sender when sending products, and the public key is used by receivers when verifying products.
Use the "--privateKey=<keyfilename>" argument when --send ing products.
This will check signatures before processing a product.
Update the ProductStorage configuration:
[my_storage] ... verifySignatures = true keychain = ..., key_<keyfilename> [key_<keyfilename>] type = gov.usgs.earthquake.distribution.ProductKey types = <producttype> sources = <productsource> key = <public key contents>
Both the sender and receiver compute a SHA-1 digest of all product contents and metadata (except the signature).
A sender uses their private key to encrypt this digest, and sends the encrypted digest as the signature.
A receiver uses the senders public key to decrypt the signature (only the senders public key is able to decrypt the signature). The receiver then compares the computed digest to the sender's digest.
If the sender and receiver digests match, the signature is verified and the content is what was originally sent. If the sender and receiver digests DON'T match, the product is not what was originally sent, or was signed using a different key.
Depending on the type of content your product contains (text/binary/compressed binary), transfer compression can significantly decrease network usage. However, this is a trade off between available CPU and network bandwidth.
A general rule is text or uncompressed binary content should use compression; compressed binary content should not use additional compression. You can read how to configure your transfer on the ProductSender configuration.
You can use the DeflateComparison utility test content you expect to send:
java -cp ProductClient.jar gov.usgs.earthquake.distribution.DeflateComparison FILE [FILE ...]
etc/test_products/quakeml/pde20100314080803960_32.xml, length = 611853 bytes no compression 0.000s, 0.0% reduction (0 fewer bytes) default compression (-1) 0.497s, 94.3% reduction (576916 fewer bytes) best speed (1) 0.444s, 93.4% reduction (571463 fewer bytes) best compression (9) 0.432s, 94.6% reduction (578911 fewer bytes)
etc/test_products/usa00040xz/us_shakemap_usa00040xz_1287260900624.xml, length = 2405991 bytes no compression 0.001s, 0.0% reduction (0 fewer bytes) default compression (-1) 1.819s, 39.2% reduction (943012 fewer bytes) best speed (1) 1.747s, 36.7% reduction (883394 fewer bytes) best compression (9) 1.502s, 39.3% reduction (944685 fewer bytes)
etc/test_products/usa00040xz/us_losspager_usa00040xz_1287260989064.xml, length = 1519786 bytes no compression 0.000s, 0.0% reduction (0 fewer bytes) default compression (-1) 0.934s, 43.8% reduction (665810 fewer bytes) best speed (1) 0.902s, 41.2% reduction (625532 fewer bytes) best compression (9) 0.966s, 43.9% reduction (667636 fewer bytes)