This document presents recommendations for the technology infrastructure and setup requirements for deploying the ShakeCast V3 system. The contents of this report are intended for use by information technology (IT) professionals to serve as a technical reference during the deployment process.
Edit me

System Overiew

The recommendations contained in this document draw from the extensive experience of scientists and IT professionals at the United States Geological Survey (USGS) National Earthquake Information Center (NEIC), having operated robust 24/7 earthquake notification and reporting systems for the past 40 years for worldwide earthquakes.

The ShakeCast V3 system was developed through a research project in the Division of Research, Innovation, and System Information (DRISI) and continues work that began under a partnership between Caltrans and the USGS to develop, deploy and support an enhanced pilot version of ShakeCast within Caltrans. This new version adds behind-the-scenes capabilities and flexibility to accommodate a broader range of facility types (e.g., bridges, buildings, roads) and user-group information that enable more effective dissemination of information, and more accurate and better tailored messages per requirements of user groups.

System Environment

The project initially called for implementation of the ShakeCast V3 system running under a Linux environment. The application had been successfully installed on Red Hat Enterprise Linux Server 6, CentOS Linux 6, and SUSE Linux Enterprise 12 for both 32 and 64-bit systems. Specifically, the system on AWS is a ShakeCast VM in CentOS Linux.

The ShakeCast application was later ported to the Microsoft Windows Server operating system (2008 or later) as a standard installation package. The Caltrans ShakeCast system is an example of Windows VM deployment.

The ShakeCast administrator can decide the method and location of the deployment of a standard ShakeCast VM and to modify the configurations regarding security and access controls to meet specific needs.

Assumptions and Constraints

Our design goal for the ShakeCast reference implementation is that a modestly knowledgeable personal computer user can install the entire system and begin receiving useful, reliable, authenticated, location-specific reports of earthquake shaking with about an hour of effort. A further goal is to continue enhancement and extension of ShakeCast so that ShakeMap information can be made more readily usable by their organizations. These goals will require significant ongoing efforts from both civil engineers and software developers to improve the quality of input inventory and to define and produce custom products and information that benefit the entire ShakeCast community.

Specifically, the list of items to consider prior to a ShakeCast system implementation include:

  • Schedule for at least 1-2 hours for ShakeCast implementation after creation of the reference operating system (VM).

  • Budget for personnel cost, license fee of the VM operating system, and host VM for system installation. The ShakeCast application is free to use and modify. Consulting resources may be needed to prepare engineered facility inventory.

  • Access to support for administration of the operating system (user account and application), firewall policy configuration (inbound and outbound traffic), and SMTP server configuration.

  • Knowledge of Apache web server configuration, MySQL database configuration, system deamon process, and system/shell/Perl script languages in order to customize the installed system.

System Organization

Figure 1 shows main components of a typical ShakeCast installation. These components are universal to all operating systems and vary depending on platform specific implementation requirements. Detailed specifications for both hardware and software are described in Section. Implementation Support. The reference implementation consists of several major components applicable to every operating system, including:

  • One database server with earthquake and ShakeMap data, user's inventory data, and results of shaking and vulnerability analysis for earthquake-impacted facilities

  • One web server with USGS earthquake products (for example, ShakeMap, ShakeCast, PAGER, etc.), user and administrative interface, and full RESTful API services

  • One directory server with a base ShakeCast system of executable scripts and utility programs, configurations, and product and notification templates

  • Between four and up to seven daemon services to automatically retrieve earthquake information and products from USGS, perform shaking and vulnerability analysis, and send out notifications to ShakeCast users.

This report assumes all ShakeCast components are installed on the same system. It is possible to separate the database server and the web server from the main ShakeCast system. The directory server and the daemon services must reside on the same system. Before separating database and web components from the main ShakeCast system, the system administrator needs to make sure proper policies are in place for accessing these servers over the network.

It is recommended to plan for a dedicated server for a ShakeCast installation. Utilization of system resources by the application varies greatly and depends on both the number of facilities being evaluated which is known ahead of time, but also on the number of significant events in any earthquake sequence, which is not predictable. When implemented as a standalone system, the administrator can determine the allowed number of generic workers to run simultaneously based on the available resources. When ShakeCast is installed in an operating system sharing resources with other applications, the administrator will need to prorate resources in advance by reducing the maximum number of generic workers.

Figure 1  ShakeCast system component chart describing main components of a ShakeCast system. The number of daemon workers is configurable and only a subset of both daemon and generic workers are shown.

Management Overview

Description of Implementation

The procedure described in this section applies to a standalone installation of the ShakeCast system. Unlike the ShakeCast V2 system, which carries a server-specific identity, the V3 system is designed to be generic internally. Thus, once a base system is created, such as virtual machine, it can be used to replicate additional systems as needed without repeating the implementation.

A typical ShakeCast system implementation takes place in several phases in sequence, including host operating system preparation, application installation, activation, application configuration, inventory loading, and testing. The administrator needs to validate each main component of the system shown in Figure 1 during the implementation and to involve persons with appropriate expertise. An end-to-end functional test can only be performed after all main components are functioning properly.

Points-of-Contact

Key Person Role
Project/Program Manager Oversee the ShakeCast project and coordinate activities regarding research, implementation, and operation of the ShakeCast system
Virtual Machine Administrator Administer the host server for ShakeCast VMs and configure system components for the virtual machine
Security/Firewall Policy Administrator Administer user account, account privileges, firewall settings, and network interconnectivity (USGS web server and SMTP server)
ShakeCast System Administrator Install ShakeCast software; administer ShakeCast database inventory on facilities, earthquakes, groups, and users; maintain the ShakeCast system
USGS ShakeCast Team Provide training and technical support on the ShakeCast system

Table 1 Points-of-Contact

Major Tasks

This section lists major tasks of a typical ShakeCast implementation. Activation of the system is marked when ShakeCast daemon services are successfully launched. Each major task should be conducted in incremental phases, including:

  • reparation of host operating system with specified hardware and software specifications

  • Configuration of firewall wall policies with permitted inbound traffic (TCP/UDP) for HTTP, HTTPS, SSH, and RDP protocols for authorized domains

  • Configuration of firewall wall policies with permitted outbound traffic for EMAIL server on SMTP protocol, optional SSL/TLS protocols, the USGS domain (usgs.gov) on HTTP protocol, and optional PDL (39977) protocol

  • Creation of user account with administrative privileges for the ShakeCast installation (MS-Windows)

  • Installation of database engine (MySQL) with only local access and updated root password

  • Installation of web server (Apache) with secure layer option (openssl)

  • Installation of Perl with required modules listed in Sec. 3.1.2

  • Installation of supporting utilities for gnuplot, wkhtmltopdf, and git

  • Installation of optional web-based database management support (PHP/phpMyAdmin)

  • Installation of ShakeCast base system (default folder "C:\ShakeCast" for Windows and "/usr/local/shakecast" for UNIX)

  • Initialization of ShakeCast database with database account, schema, and default data inventory

  • Initialization of ShakeCast web server domain with ShakeCast web in both HTTP and HTTPS protocols and access control

  • Loading of facility inventory

  • Loading of user group inventory with defined earthquake monitoring regions

  • Loading of user inventory with at least one more ShakeCast administrator account and updated credentials for the default administrator account "scadmin"

  • Activation of the ShakeCast application

  • Functional test of the system with ShakeCast heartbeat (notification) and ShakeMap scenario (damage assessment)

System Security

The default setup of a ShakeCast system allows access via both the command line using SSH and the web with HTTP or HTTPS. The ShakeCast web server is designed to serve earthquake information to users and to allow administrators to conduct general administration of the system (Lin et al., 2014). Command line access via SSH (Linux) should be limited to only administrators. ShakeCast tasks not covered by the web interface are considered as advanced topics for experienced ShakeCast administrators. In the most secured setup of ShakeCast, the administrator can choose to disable access from the web and only permits SSH access.

Firewall and system level security setup are platform specific issues not covered by this manual. Even though ShakeCast implements a basic authentication scheme, it is highly recommended to implement system-level firewall policies to limit exposure to the Internet. These rules will take precedence over the ShakeCast-defined user authentication scheme. For inbound traffic, firewall policies are effective methods to define domains where users can access the products and information of the ShakeCast server. For outbound traffic, firewall policies should permit the USGS Web server http://earthquake.usgs.gov, the source for all earthquake products processed by ShakeCast. For ShakeCast systems receiving earthquake products via the USGS Product Distribution Layer (PDL) client, the program uses port 39977 to connect to the upstream hub server.

Implementation Support

Hardware

Recommended minimum hardware specifications for the ShakeCast system includes:

  • Single Intel Xeon E5-2670 equivalent processor.

  • 1GB RAM.

  • 30GB hard drive storage.

  • Low performance Internet connection (<1MB/s).

The above hardware setup is roughly equivalent to the "micro" instance on the Amazon Elastic Compute Cloud (Amazon EC2) in which the performance was assessed.

Depending on the size of facility and user inventory and the earthquake monitoring areas, more hardware resources will be needed in order to deliver anticipated performance. USGS earthquake products (ShakeMap, ShakeCast, lossPAGER, DYFI?, and others) for each processed earthquake usually consume 30-50 MB of hard drive space. For ShakeCast systems designated for earthquake response purpose, we recommend to at least double the minimum recommended hardware specifications. During the testing phase, the Caltrans ShakeCast V3 system hosting over 26,000 facilities and approximately 400 users in several groups required a VM with the following hardware-equivalent specifications for the primary and backup servers:

  • Quad Intel Xeon X5670 2.9GHz processors.

  • 8GB RAM.

  • 100GB hard drive storage.

  • High performance Internet connection.

Software

The ShakeCast V3 system is distributed for both Linux and MS-Windows operating systems. The system is built on an open-source stack of supporting applications shared by all platforms,

  • Apache Web server 2.x.

  • MySQL database 5.x.

  • Perl 5.14+ scripting language.

  • Modules: DBI, DBD::mysql, Text::CSV_XS, Config::General, enum, XML::Parser, XML::LibXML, XML::Writer, XML::Twig, XML::Simple, Template-toolkit, PDF::API2, PDF::Table, MIME::Lite, GD, GD::Text, GD::Graph, GD::Graph3d, HTML::TableExtract, Net::SSLeay, Net::SMTP::SSL, Net::SMTP::TLS, Authen::SASL, Archive::Zip, JSON, JSON::XS, File::Path, Image::Size, Mojolicious.

  • wkhtmltoimage conversion tool.

  • gnuplot image tool.

  • HTML5/Google Maps/markerclusterer/jQuery/Bootstrap/dataTables Web tools.

  • Optional PHP/phpmyadmin scripting language.

  • Optional git version control tool.

Linux-specific implementations:

  • Xvfb X virtual framebuffer display server (required for 64-bit systems and optional for 32-bit systems).

  • mailx as default mail utility.

  • ShakeCast services as background daemon processes.

  • Database backup cron job.

Windows-specific implementations:

  • SMTP as default mail protocol (supports both SSL/TLS security layers).

  • ShakeCast services as Windows system processes.

Facilities

For purpose of post-earthquake response, the ShakeCast system host needs to be installed on a high availability server. Depending on the IT infrastructure of the user's organization, it is desirable to have features of a typical network operation center, including:

  • 24x7 production environment with service monitoring and technical support

  • High level of data security and operational reliability

  • Switchover between primary and secondary ShakeCast server

  • Rapid backup and recovery of system.

Materials

For a standalone installation, the ShakeCast application and its supporting open-stack programs are freely available on the Internet, include:

User-specific materials related to ShakeCast implementation is applicable to post-installation configuration and database population, include:

  • Administrator credentials for the operating system, database, and web server

  • SMTP server access information and optional proxy server access information

  • Facility (bridge/building/roadway) inventory for the ShakeCast database

  • Group/user inventory for the earthquake monitoring regions and notifications

  • ShakeCast notification and product templates.

Documentation

This document provides background information on the best practice for the ShakeCast implementation. Additional information regarding installation, presentation and training can be found on the ShakeCast wiki site at

https://my.usgs.gov/confluence/display/ShakeCast/Home

Specific documents related to ShakeCast implementation include:

  • ShakeCast User Guide

  • Install ShakeCast V3 on AWS

Additionally the USGS maintains a ShakeCast listsrv mailing list and email support shakecast-help@usgs.gov to complement the static documentation. Users are advised to consult with the documentation before requesting for technical support.

Implementation Impact

After a successful launch of the ShakeCast system, it will begin to receive and process real-time earthquake information from the USGS. The system will interact with both users and administrators of the ShakeCast system and to deliver notifications based on established criteria. As results the ShakeCast administrator needs to anticipate demand from system-related activities as part the implementation, include:

  • Storage growth rate of the system (~50MB per earthquake per version) for incoming and processed earthquake products, typically on the order of 3GB/month for California events greater than M4.0.

  • Performance requirements and availability of the system during active processing period

  • Security requirements of ShakeCast and host operating system with appropriate user accounts and settings

  • System backup of ShakeCast and host operating system for recovery after system disasters

  • Help desk support of the system for end-users on ShakeCast web interface, processed products, and notifications.

Performance Monitoring

As part of the high availability requirements, the ShakeCast system needs to be monitored at both the system and the application levels.

System level monitoring includes:

  • Internet connectivity and associated security settings

  • User accounts and access privileges

  • CPU utilizations and storage

Application level monitoring includes:

  • ShakeCast system services

  • MySQL database

  • Apache web server

Configuration Management Interface

The ShakeCast administrative web interface is not available in the early stages of the installation process. The administrator should expect to use OS-specific tools for installing individual applications and for configuring access control. After the ShakeCast system is installed, the administrator can choose among several options to manage the application, including:

  • ShakeCast Administration Web Interface for general configuration, inventory management, user accounts and access control

  • Command line console and/or Microsoft Remote Desktop Connection for full access to all ShakeCast functions, worker management, daemon services, etc.

  • Command line console and/or Microsoft Remote Desktop Connection for system level administrative tasks.

REFERENCES

Lin, K., D. J. Wald, and L. Turner (2014). ShakeCast User Guide, U.S. Geol. Survey Open File Rep. xxx, 154 pp.

APPENDIX KEY TERMS

ShakeCast Terms and Terminology

The terminology used in ShakeCast may differ slightly from that used in other information systems.

ShakeCast Server

A ShakeCast Server is a computer that is running the ShakeCast Server Software. The server may or may not be acting as a server in the traditional sense: sending data downstream to another ShakeCast Server. Instead, a server may be only receiving data from another ShakeCast Server and making that data available on the web.

Upstream and Downstream

ShakeCast machines are more easily defined in terms of being upstream or downstream. An upstream machine sends ShakeMap data to a downstream machine. The request to send the data may originate on either machine.

Event

An event is a seismic event – an earthquake (either real or simulated). All events have a globally unique and permanent identifying number, called an event ID. The event ID is assigned by the seismic monitoring systems, not by ShakeCast or ShakeMap. Once created, events may not be deleted, although they may be marked as "canceled" to indicate that an event message was anomalous and should no longer be considered.

ShakeMap and ShakeMaps

ShakeMap is a software system for computing maps of shaking intensity. It uses data from networks of seismometers and other sources to estimate shaking intensity as measured by a variety of physical or instrumental metrics such as peak acceleration, velocity, spectral response, instrumental intensity, and so on. There are very few ShakeMap systems, all of them operated by teams of seismologists and their professional support staffs.

ShakeMap is also the name for the maps produced by the ShakeMap system.

Product

A ShakeMap Product (or just Product) is a result of ShakeMap processing. When ShakeMap processes a seismic event, it produces files and maps for many different metrics (i.e., peak acceleration, velocity, etc.). Each of these maps may be produced in many different data formats (i.e., as a grid of scalar values, as a GIS shapefile, as an image in JPEG format, as an image in PostScript format, etc.).

Each combination of event, metric, and format is a different product.

Facility

A facility is a location that is to be monitored by ShakeCast. A facility is typically a building, bridge, highway, or similar man-made structure. The location of the facility must be known so that ShakeCast can attribute various levels of shaking at that location, and the facility may have associated fragility measures in one or more of the shaking metrics.

Fragility

Fragility is the measure of likely damage at a particular facility when a certain level of shaking is exceeded, as measured in a particular metric (e.g., "peak acceleration at the period of one second").

Damage (Alert) Level

There are classes of fragility associated with each facility in each metric. The damage level is the class or category of shaking intensity experienced at a particular location. Damage levels are typically assigned as "No damage expected", "Some damage expected", and "Damage likely", or "green", "yellow", and "red". Damage levels are locally defined on each ShakeCast Server, and different organizations may use different categories or a different number of categories.

Notification

Notification is the process of electronically notifying a ShakeCast end user that a particular damage level is estimated at a particular facility from a certain event. Notifications can be delivered in a variety of electronic forms, including as an email message or an electronic pager message. The following figure shows one possible notification format. The format and content of notification messages may be altered by the local ShakeCast system administration.

[]:

Tags: getting_started troubleshooting shakecast_v3