Package gov.usgs.earthquake.aws

Class AwsBatchIndexer

java.lang.Object

gov.usgs.earthquake.aws.AwsBatchIndexer

All Implemented Interfaces:

Bootstrappable

public class AwsBatchIndexer
extends java.lang.Object
implements Bootstrappable

Class to index a batch of products that have already been sent to the AWS hub. Reads a list of products to be indexed. Reads indexer from configuration file. For each product, fetch product information from the get_product AWS endpoint and call indexer.onProduct.

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	DATABASE_DRIVER_ARGUMENT	Argument for database driver
static java.lang.String	DATABASE_URL_ARGUMENT	Argument for database URL
static java.lang.String	FORCE_REINDEX_ARGUMENT	Force reindex argument
static java.lang.String	GET_PRODUCT_URL_ARGUMENT	Get product URL argument
static java.lang.String	INDEXER_CONFIG_NAME_ARGUMENT	Argument for indexer configuration name
static java.lang.String	INDEXER_CONFIG_NAME_DEFAULT	Default indexer configuration name
static java.lang.String	INDEXER_DATABASE_ARGUMENT	Argument for indexer database
static java.lang.String	INDEXER_DATABASE_DEFAULT	Default database for indexer

Fields inherited from interface gov.usgs.earthquake.distribution.Bootstrappable

RUN_EXCEPTION_EXIT_CODE

Constructor Summary

Constructors

Constructor	Description
AwsBatchIndexer()

Method Summary

Modifier and Type	Method	Description
Product	getProduct(ProductId id)	Get Product from endpoint.
java.net.URL	getProductUrl(ProductId id)	Use getProductUrl template to generate URL.
void	processProductId(ProductId id)	Fetch and Index a product.
void	readProductIdsFromDatabase(java.lang.String driver, java.lang.String url)	Read product ids (as urns) from database and submit to executor for processing.
void	readProductIdsFromStdin()	Read product ids (as urns) from stdin and submit to executor for processing.
void	run(java.lang.String[] args)	Called by Bootstrap after processing the Configurable interface.
void	submitProductId(ProductId id)	Submit a product id to the executor service for processing.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FORCE_REINDEX_ARGUMENT

public static final java.lang.String FORCE_REINDEX_ARGUMENT

Force reindex argument

See Also:

Constant Field Values

GET_PRODUCT_URL_ARGUMENT

public static final java.lang.String GET_PRODUCT_URL_ARGUMENT

Get product URL argument

See Also:

Constant Field Values

INDEXER_CONFIG_NAME_ARGUMENT

public static final java.lang.String INDEXER_CONFIG_NAME_ARGUMENT

Argument for indexer configuration name

See Also:

Constant Field Values

INDEXER_CONFIG_NAME_DEFAULT

public static final java.lang.String INDEXER_CONFIG_NAME_DEFAULT

Default indexer configuration name

See Also:

Constant Field Values

DATABASE_DRIVER_ARGUMENT

public static final java.lang.String DATABASE_DRIVER_ARGUMENT

Argument for database driver

See Also:

Constant Field Values

DATABASE_URL_ARGUMENT

public static final java.lang.String DATABASE_URL_ARGUMENT

Argument for database URL

See Also:

Constant Field Values

INDEXER_DATABASE_ARGUMENT

public static final java.lang.String INDEXER_DATABASE_ARGUMENT

Argument for indexer database

See Also:

Constant Field Values

INDEXER_DATABASE_DEFAULT

public static final java.lang.String INDEXER_DATABASE_DEFAULT

Default database for indexer

See Also:

Constant Field Values

Constructor Detail

AwsBatchIndexer

public AwsBatchIndexer()

Method Detail

run

public void run(java.lang.String[] args)
         throws java.lang.Exception

Description copied from interface: Bootstrappable

Called by Bootstrap after processing the Configurable interface.

Specified by:

run in interface Bootstrappable

Parameters:

args - array of command line arguments.

Throws:

java.lang.Exception - Exception

getProductUrl

public java.net.URL getProductUrl(ProductId id)
                           throws java.lang.Exception

Use getProductUrl template to generate URL. Replace "{source}", "{type}", "{code}", and "{updateTime}" placeholders.

Parameters:

id - which product.

Returns:

URL with placeholders replaced.

Throws:

java.lang.Exception - Exception

getProduct

public Product getProduct(ProductId id)
                   throws java.lang.Exception

Get Product from endpoint.

Parameters:

id - which product.

Returns:

Product object.

Throws:

java.lang.Exception - Exception

processProductId

public void processProductId(ProductId id)

Fetch and Index a product. Called from executor service to process product ids.

Parameters:

id - which product

readProductIdsFromDatabase

public void readProductIdsFromDatabase(java.lang.String driver,
                                       java.lang.String url)
                                throws java.lang.Exception

Read product ids (as urns) from database and submit to executor for processing.

Parameters:

driver - database driver

url - database url

Throws:

java.lang.Exception - exception

readProductIdsFromStdin

public void readProductIdsFromStdin()
                             throws java.lang.Exception

Read product ids (as urns) from stdin and submit to executor for processing.

Throws:

java.lang.Exception - Exception

submitProductId

public void submitProductId(ProductId id)
                     throws java.lang.InterruptedException

Submit a product id to the executor service for processing. If queue is too large (500 ids), blocks until queue is smaller (100 ids).

Parameters:

id - which product

Throws:

java.lang.InterruptedException - InterruptedException