Package gov.usgs.earthquake.indexer

Class Indexer

  • All Implemented Interfaces:
    NotificationIndexCleanup.Listener, NotificationListener, Configurable
    public class Indexer
extends DefaultNotificationListener
    The indexer receives products from Distribution, and adds them to the EventIndex. This class provides the following configurable properties (in addition to those inherited from DefaultNotificationListener):
    associator
    An object that implements the Associator interface.
    storage
    An object that implements the ProductStorage interface.
    index
    An object that implements the ProductIndex interface.
    modules
    A comma delimited list of objects that implement the IndexerModule interface
    listeners
    A comma delimited list of objects that implement the IndexerListener interface

    • Field Detail

      • TRUMP_PREFERRED_WEIGHT

        public static final long TRUMP_PREFERRED_WEIGHT
        Preferred weight for persistent trump.
        See Also:
        Constant Field Values

      • TRUMP_PRODUCT_TYPE

        public static final java.lang.String TRUMP_PRODUCT_TYPE
        Type for persistent trimp
        See Also:
        Constant Field Values

      • PERSISTENT_TRUMP_PREFIX

        public static final java.lang.String PERSISTENT_TRUMP_PREFIX
        Prefix for persistent trump
        See Also:
        Constant Field Values

      • ASSOCIATOR_CONFIG_PROPERTY

        public static final java.lang.String ASSOCIATOR_CONFIG_PROPERTY
        Property name to configure a custom associator.
        See Also:
        Constant Field Values

      • ASSOCIATE_USING_CURRENT_PRODUCTS_PROPERTY

        public static final java.lang.String ASSOCIATE_USING_CURRENT_PRODUCTS_PROPERTY
        Property to associate using current products
        See Also:
        Constant Field Values

      • DEFAULT_ASSOCIATE_USING_CURRENT_PRODUCTS

        public static final java.lang.String DEFAULT_ASSOCIATE_USING_CURRENT_PRODUCTS
        Default state for associate using current products
        See Also:
        Constant Field Values

      • STORAGE_CONFIG_PROPERTY

        public static final java.lang.String STORAGE_CONFIG_PROPERTY
        Property name to configure a custom storage.
        See Also:
        Constant Field Values

      • STORAGE_DIRECTORY_CONFIG_PROPERTY

        public static final java.lang.String STORAGE_DIRECTORY_CONFIG_PROPERTY
        Shortcut name to configure a file product storage.
        See Also:
        Constant Field Values

      • INDEX_CONFIG_PROPERTY

        public static final java.lang.String INDEX_CONFIG_PROPERTY
        Property name to configure a custom index.
        See Also:
        Constant Field Values

      • INDEXFILE_CONFIG_PROPERTY

        public static final java.lang.String INDEXFILE_CONFIG_PROPERTY
        Shortcut name to configure a sqlite index.
        See Also:
        Constant Field Values

      • MODULES_CONFIG_PROPERTY

        public static final java.lang.String MODULES_CONFIG_PROPERTY
        Property name to configure modules.
        See Also:
        Constant Field Values

      • LISTENERS_CONFIG_PROPERTY

        public static final java.lang.String LISTENERS_CONFIG_PROPERTY
        Property name to configure listeners.
        See Also:
        Constant Field Values

      • LOCAL_REGIONS_PROPERTY

        public static final java.lang.String LOCAL_REGIONS_PROPERTY
        Property name to configure local regions file.
        See Also:
        Constant Field Values

      • DEFAULT_LOCAL_REGIONS

        public static final java.lang.String DEFAULT_LOCAL_REGIONS
        Path to local regions file.
        See Also:
        Constant Field Values

      • ENABLE_SEARCH_PROPERTY

        public static final java.lang.String ENABLE_SEARCH_PROPERTY
        Property name to enable search socket.
        See Also:
        Constant Field Values

      • SEARCH_PORT_PROPERTY

        public static final java.lang.String SEARCH_PORT_PROPERTY
        Property name for search socket port.
        See Also:
        Constant Field Values

      • SEARCH_THREADS_PROPERTY

        public static final java.lang.String SEARCH_THREADS_PROPERTY
        Property name for search socket thread pool size.
        See Also:
        Constant Field Values

      • DEFAULT_ENABLE_SEARCH

        public static final java.lang.String DEFAULT_ENABLE_SEARCH
        Default value whether to enable search socket.
        See Also:
        Constant Field Values

      • DEFAULT_SEARCH_PORT

        public static final java.lang.String DEFAULT_SEARCH_PORT
        Default port where search socket listens.
        See Also:
        Constant Field Values

      • DEFAULT_SEARCH_THREADS

        public static final java.lang.String DEFAULT_SEARCH_THREADS
        Number of threads (concurrent searches) allowed.
        See Also:
        Constant Field Values

      • INDEX_ARCHIVE_INTERVAL_PROPERTY

        public static final java.lang.String INDEX_ARCHIVE_INTERVAL_PROPERTY
        Configurable property for index archive internal
        See Also:
        Constant Field Values

      • INDEX_ARCHIVE_POLICY_PROPERTY

        public static final java.lang.String INDEX_ARCHIVE_POLICY_PROPERTY
        Configurable property for index archive policy
        See Also:
        Constant Field Values

    • Constructor Detail

      • Indexer

        public Indexer()
        throws java.lang.Exception
        Default no-arg constructor. This gets called from the Configurable API. All configuration parameters are set in the "configure" method.
        Throws:
        java.lang.Exception - If the JDBCProductIndex throws an exception.

    • Method Detail

      • getAssociator

        public Associator getAssociator()
        Returns the current associator used to associate products to one-another and products to events.
        Returns:
        The current Associator.

      • setAssociator

        public void setAssociator​(Associator associator)
        Sets the given associator as the current associator to associate products to one-another and products to events.
        Parameters:
        associator - The associator to use from this point forward.

      • getProductStorage

        public ProductStorage getProductStorage()
        Returns the product storage component that is used to store products as they are received.
        Returns:
        The current product storage component.

      • setProductStorage

        public void setProductStorage​(ProductStorage productStorage)
        Sets the current product storage component used to store products as they are received.
        Parameters:
        productStorage - The product storage component to use from this point forward.

      • getProductIndex

        public ProductIndex getProductIndex()
        Returns the product index component used to index product information as it is received.
        Returns:
        The current product index component.

      • setProductIndex

        public void setProductIndex​(ProductIndex productIndex)
        Sets the product index component used to index product information as it is received.
        Parameters:
        productIndex - The product index component to use from this point forward.

      • addModule

        public void addModule​(IndexerModule toAdd)
        Adds the give indexer module to the current list of modules used by the indexer to handle products.
        Parameters:
        toAdd - The IndexerModule to add to our list.

      • removeModule

        public void removeModule​(IndexerModule toRemove)
        Removes the first occurrence of the given indexer module from the current list of known modules.
        Parameters:
        toRemove - The module to remove.
        See Also:
        LinkedList.remove(Object)

      • getModule

        protected IndexerModule getModule​(Product product)
        This method checks each module's support level for the given product, returning the first module with the highest support level.
        Parameters:
        product - the product to summarize.
        Returns:
        module best suited to summarize product.

      • addListener

        public void addListener​(IndexerListener toAdd)
        Adds a listener to this indexer. Listeners are notified when an event is added, updated, or deleted, or when a new product arrives and is un-associated to an event.
        Parameters:
        toAdd - The IndexerListener to add

      • removeListener

        public void removeListener​(IndexerListener toRemove)
        Removes a listener from this indexer.Listeners are notified when an event is added, updated, or deleted, or when a new product arrives and is un-associated to an event.
        Parameters:
        toRemove - The IndexerListener to remove

      • notifyListeners

        protected void notifyListeners​(IndexerEvent event)
        Send an indexer event to all registered IndexerListeners. Creates a NotificationEvent, with a reference to this object and calls each notificationListeners onNotification method in separate threads. This method usually returns before registered NotificationListeners have completed processing a notification.
        Parameters:
        event - The event that occurred to trigger the notification. Note: An IndexerEvent has a specific "type" to clarify the type of event that occurred.

      • hasProductBeenIndexed

        protected boolean hasProductBeenIndexed​(ProductId id)
        Check whether this product is in the index. NOT synchronized to allow multiple threads to access. readProductIndex.hasProduct is synchronized.
        Parameters:
        id - ProductId to check
        Returns:
        true if product has already been indexed.

      • accept

        public boolean accept​(ProductId id)
        Override the DefaultNotificationListener accept method, to always process products that may affect event association.
        Overrides:
        accept in class AbstractListener
        Parameters:
        id - the product id to check.
        Returns:
        boolean whether the product should be indexed.

      • onBeforeProcessNotification

        protected boolean onBeforeProcessNotification​(Notification notification)
                                       throws java.lang.Exception
        Check whether to skip products that have already been indexed.
        Overrides:
        onBeforeProcessNotification in class DefaultNotificationListener
        Parameters:
        notification - notification about to be processed.
        Returns:
        true to process the notification, false to skip
        Throws:
        java.lang.Exception - if error occurs

      • onProduct

        public void onProduct​(Product product)
               throws java.lang.Exception
        This method receives a product from Product Distribution and adds it to the index. Implementation follows from Product Indexer Diagram (pg.10) of ProductIndexer.pdf document dated 09/09/2010. Calls onProduct(product, false), which will not reprocess already processed products.
        Overrides:
        onProduct in class DefaultNotificationListener
        Parameters:
        product - The product triggering the event.
        Throws:
        java.lang.Exception - if an exception occurs.

      • onProduct

        public void onProduct​(Product product,
                      boolean force)
               throws java.lang.Exception
        Receive a product and add it to the index. Optionally, reprocessing a product that has already been processed.
        Parameters:
        product - The product triggering the event.
        force - Whether to reprocess products that have already been processed (true), or skip (false).
        Throws:
        java.lang.Exception - if error occurs

      • storeProduct

        public Product storeProduct​(Product product,
                            boolean force)
                     throws java.lang.Exception
        Stores a product
        Parameters:
        product - Product to store
        force - if should skip already indexed check
        Returns:
        Product if stored, null if not
        Throws:
        java.lang.Exception - if error occurs

      • summarizeProduct

        public ProductSummary summarizeProduct​(Product product)
                                throws java.lang.Exception
        Use modules to summarize product.
        Parameters:
        product - To summarize
        Returns:
        A product summary
        Throws:
        java.lang.Exception - if error occurs

      • indexProduct

        protected ProductSummary indexProduct​(ProductSummary productSummary)
                               throws java.lang.Exception
        Add product summary to product index.
        Parameters:
        productSummary - to add
        Returns:
        Summary added to index
        Throws:
        java.lang.Exception - if error occurs

      • getTrumpedProductId

        protected ProductId getTrumpedProductId​(ProductSummary trumpSummary)
        Get the productId referred to by a trump product.
        Parameters:
        trumpSummary - trump product with reference to product id.
        Returns:
        product id, or null if unable to parse product id.

      • getProductSummaryById

        protected ProductSummary getProductSummaryById​(ProductId id)
                                        throws java.lang.Exception
        Get a product summary object using its product id.
        Parameters:
        id - id to find.
        Returns:
        matching product summary or null.
        Throws:
        java.lang.Exception - if error occurs

      • setSummaryWeight

        protected Event setSummaryWeight​(Event event,
                                 ProductSummary summary,
                                 java.lang.Long preferredWeight)
                          throws java.lang.Exception
        Update a product summary weight
        Parameters:
        event - the event.
        summary - the summary.
        preferredWeight - the weight to set.
        Returns:
        event with updated summary.
        Throws:
        java.lang.Exception - if error occurs

      • resummarizeProduct

        protected Event resummarizeProduct​(Event event,
                                   ProductSummary summary)
                            throws java.lang.Exception
        Resummarize a product within an event.
        Parameters:
        event - the event.
        summary - the summary.
        Returns:
        event with updated summary.
        Throws:
        java.lang.Exception - if error occurs

      • checkForEventSplits

        protected java.util.List<IndexerChange> checkForEventSplits​(ProductSummary summary,
                                                            Event originalEvent,
                                                            Event updatedEvent)
                                                     throws java.lang.Exception
        Check for event splits (and split them if needed).
        Parameters:
        summary - the summary the indexer is currently processing.
        originalEvent - the event before the indexer made any changes.
        updatedEvent - the event after the indexer made any changes.
        Returns:
        List of changes made during this method.
        Throws:
        java.lang.Exception - if error occurs

      • splitEvents

        protected Event splitEvents​(Event root,
                            Event leaf)
                     throws java.lang.Exception
        Removes the leaf event (and all its products) from the root event. This method modifies the runtime objects as well as updating the index DB.
        Parameters:
        root - The root event from which all leaf products will be removed
        leaf - The event (with products) that will be removed from the root
        Returns:
        copy of root without the products that have been removed. The indexId property of leaf is updated to its new value.
        Throws:
        java.lang.Exception - if error occurs

      • mergeEvents

        protected Event mergeEvents​(Event target,
                            Event child)
                     throws java.lang.Exception
        Merges the child event (and all its products) into the target event. If the child event attempts to merge in a product that is the same as one already associated to the target event, the child version of the product takes precedence. Note: This only applies when the target and child product have the same type, code, source, and update time; i.e. the products are duplicates. This method modifies the runtime objects as well as the index DB. The child event is then deleted.
        Parameters:
        target - The target event into which the child is merged.
        child - The child event to be merged into the target.
        Returns:
        the updated event
        Throws:
        java.lang.Exception - if error occurs

      • checkForEventMerges

        protected java.util.List<IndexerChange> checkForEventMerges​(ProductSummary summary,
                                                            Event originalEvent,
                                                            Event updatedEvent)
                                                     throws java.lang.Exception
        Check and merge any nearby events or previously unassociated products that now associate.
        Parameters:
        summary - the summary currently being processed by the indexer.
        originalEvent - the event before any changes.
        updatedEvent - the event after the summary was associated.
        Returns:
        list of any merge type changes.
        Throws:
        java.lang.Exception - if error occurs

      • getPrevProductVersion

        protected ProductSummary getPrevProductVersion​(ProductSummary summary)
                                        throws java.lang.Exception
        Takes a summary return the previous
        Parameters:
        summary - A product summary
        Returns:
        The previous summary
        Throws:
        java.lang.Exception - if error occurs

      • getPrevEvent

        protected Event getPrevEvent​(ProductSummary summary,
                             boolean associating)
                      throws java.lang.Exception
        Find an existing event that summary should associate with.
        Parameters:
        summary - the previous event.
        associating - whether associating (vs archiving).
        Returns:
        previous event, or null if none found.
        Throws:
        java.lang.Exception - if error occurs

      • configure

        public void configure​(Config config)
               throws java.lang.Exception
        Loads parent, specific, and dependent configurations; in that order.
        Specified by:
        configure in interface Configurable
        Overrides:
        configure in class DefaultNotificationListener
        Parameters:
        config - the Config object with settings.
        Throws:
        java.lang.Exception - if configuration exceptions occur.

      • shutdown

        public void shutdown()
              throws java.lang.Exception
        Shuts down the Indexer. The parent shutdown method is called and then all executor services (from listeners) are shutdown in sequence.
        Specified by:
        shutdown in interface Configurable
        Overrides:
        shutdown in class DefaultNotificationListener
        Throws:
        java.lang.Exception - if exceptions occur while starting.

      • startup

        public void startup()
             throws java.lang.Exception
        Starts up the necessary parent, specific, and dependent processes, in that order.
        Specified by:
        startup in interface Configurable
        Overrides:
        startup in class DefaultNotificationListener
        Throws:
        java.lang.Exception - if exceptions occur while starting.

      • purgeExpiredProducts

        public int[] purgeExpiredProducts()
                           throws java.lang.Exception
        Checks the index for content that match a configured archive policy. Events are checked first and matched events are removed along with all their products. Listeners are notified of each archived event with an EVENT_ARCHIVED type. Unassociated products are checked next, matched unassociated products are archived and listeners are notified with PRODUCT_ARCHIVE type. Note: Product "age" is determined by when the earthquake for that product occurred and does not reflect how long the product has actually been in the index.
        Returns:
        Int array of size 2
        Throws:
        java.lang.Exception - if error occurs
        See Also:
        archivePolicies

      • removeEvent

        protected void removeEvent​(Event event)
                    throws java.lang.Exception
        Removes the given event from the Indexer ProductIndex and ProductStorage.
        Parameters:
        event - event to remove
        Throws:
        java.lang.Exception - If errors occur while removing the event

      • removeSummary

        protected void removeSummary​(ProductSummary summary)
                      throws java.lang.Exception
        Removes the given summary from the Indexer ProductIndex and ProductStorage.
        Parameters:
        summary - to remove
        Throws:
        java.lang.Exception - If errors occur while removing the summary

      • search

        public SearchResponse search​(SearchRequest request)
                      throws java.lang.Exception
        Search for products in this index.
        Parameters:
        request - the search request.
        Returns:
        the search response.
        Throws:
        java.lang.Exception - if error occurs

      • isDisableArchive

        public boolean isDisableArchive()
        Returns:
        disableArchive

      • setDisableArchive

        public void setDisableArchive​(boolean disableArchive)
        Parameters:
        disableArchive - boolean to set

      • getArchiveInterval

        public long getArchiveInterval()
        Returns:
        the archiveInterval

      • setArchiveInterval

        public void setArchiveInterval​(long archiveInterval)
        Parameters:
        archiveInterval - the archiveInterval to set

      • getArchivePolicies

        public java.util.List<ArchivePolicy> getArchivePolicies()
        Returns:
        the archivePolicies