IBM OGSA-DAI Replication

A number of IBM products enable data replication, including, DB2 DataPropagator (for data replication among relational database); DataPropagator NonRelational (for data replication among non-relational database platforms.); DB2 DataJoiner (Multidatabase server solution for accessing heterogeneous data as if it were in a single data store); IMS DataPropagator (for replication between IMS db and DB2 db); SQL Replication (updates are staged in relational tables before replicating); Queue replication (updates are written in messages that are transported through WebSphere MQ message queues).

DataPropagator appears in WebSphere Information Integrator earlier versions; since WebSphere II V8, it has been replaced by SQL Replication and Queue replication.

1.1 DB2 DataPropagator

There are 3 main components in DataPropagator architecture: Administration Interfaces (are used to create control table to store replication criteria), Change-Capture Mechanisms (use database log or triggers to captures the changes and store them temporarily in tables) and the Apply Program (copies data target DB by scheduled applying transactions accumulated). These components communicate to each other by using control tables. DataPropagator supports after-image (copy data after updating)/before-image (copy data before updating, useful for auditing or rollback) replication; full-refresh (copy all data)/differential-refresh (copy only the changes) replication; multiply levels of conflict detection (standard conflict detection/ enhanced conflict detection/ Row-replica conflict); allowing user indicating a sub set, a simple view or complex joins and unions replication. DataPropagator performs asynchronous replication only, allowing user to apply replication by specifying time intervals, events, or on demand. [7]

WebSphere II later version uses IBM SQL-based replication approach that maximizes flexibility and efficiency in managing scheduling, transformation, and distribution topologies. SQL replication uses similar replication approach as DataPropagator, however, a new component -- Alert Monitor -- has been added to its architecture. This component allows user to check the status or replication processes, keeps the performance information, and sends notification to user when problem happens. As DataPropagator, SQL replication captures changes using either log-based or trigger-based mechanism and inserts them into a relational staging table. An apply process asynchronously handles the updates to the target systems. SQL replication is designed for warehouses or marts, maintaining data consistency between disparate applications, or efficiently managing distribution and consolidation scenarios among headquarter and branch or retail configurations. [8][9][10]

The new IBM queue-based replication architecture uses message queues to transport data between sources and targets. As show in Fig3.1.2, changes are captured from the log and placed on a queue. The apply process retrieves the changes from the queue and applies them -- in parallel -- to the target system. A Q Apply browser is used to examine and track dependencies among transactions in the queue, and, based on dependency analysis, apply process allows data to be applied in parallel in a way that preserves data integrity. Q Replication is designed to support business continuity, workload distribution, and application integration scenarios, promising to offer low latency and high throughput replication with managed conflict detection and resolution.

2. Sybase Replication Solutions

Sybase’s replication has been available since 1993, pioneering in the field, characterized by its “publish-and subscribe” approach. Microsoft’s database technology is derived from that of Sybase, making use of similar replication strategies. [11]

The main components in Sybase replication architecture include, Replication Agent (running inside of each data publisher (see ASE – Sybase’s Adaptive Server Enterprise -- in Fig 3.2.1) detects changes in source database by reading transaction log); Replication Server (directs data changes to their target databases using pre-configured intelligent routes); Replication Server Manager (GUI-based powerful managing and monitoring tool, running on a single desktop, can create replication environments, monitor and manage the replication flow); Stable Queues (are a safety mechanism, storing data generated by each replication process, ensuring no data is lost, providing a configurable level of component fault tolerance).

When replicating, Replication Agent detects data changes and transfers to Replication Server (if Replication Server is unavailable, data stays in the transaction log); Replication Server connects with the target database, or other Replication Servers, and delivers data through optimized routes; Stable Queue is used here by Replication Servers to store data, and data is removed from the queue only after it has been successfully forwarded. Compare to other similar relational database replication vendors, Sybase solutions give more attention on system performance in terms of availability, reliability, network load balance, and transferring efficiency, and also advanced in providing comprehensive solution for mobile computing.

3. Oracle Advanced Replication

Oracle Advanced Replication aims to support a variety of replication requirements. Its 3 types of replication environments, namely, Multimaster Replication, Materialized View Replication, and Multimaster and Materialized View Hybrid Replication, support all kinds of database objects replicating, such as tables (a partial of table), indexes, views, packages, procedures and functions, etc. [6]

Multimaster Replication, employing peer-to-peer structure, (shows in Fig 3.3.1), enables multiple sites acting as equal peers, to manage groups of replicated database objects; Update happens at any sites; Changes to the tables are pushed to other sites by either asynchronous or synchronous processing; In a synchronous replication, changes immediately apply to all sites in a single transaction, which rolls back if fails at any site, ensuring no data conflict; In a asynchronous replication, each site has individual scheduled intervals to propagate changes, cannot guarantee data consistency in real-time, data conflicts(Update conflict – more than one sites tried to update the same row /Uniqueness conflict – more than one sites inserted rows having the same primary key/ Delete conflict -- a site updated a row deleted by the other site) may happen, yet, requiring cheaper hardware and network resources; Multimaster Replication is designed for mission critical database, ensuring continuous availability, providing read load balancing and survivability protection.

In Materialized View Replication, (as show in Fig 3.4.2), typically one mater site manages several non-mater sites, which keeps a copy or a partial copy of database objects in the mater site. Multitier Materialized View Replication is also possible, in which lower tier sites replicate database objects of higher tier sites (in Fig 3.3.3). Designed for Mass Deployment development, Materialized Views enable local access and disconnected computing, which improve response times and availability, and reduce network load; it also enables subset replication, which increase data security (different view can be granted different privileges). Materialized Views are updatable; three refresh methods are provided to ensure replication consistency, namely fast refresh(only update the changed rows), complete refresh(updates the entire duplicated objects), and force refresh (performs a complete refresh only when a fast refresh is not possible)

Hybrid Replication (Fig 3.3.4) is the combination of Multimaster Replication and Materialized Views. Multimaster Replication enables real-time refreshment, while Materialized Views propagate changes from multiple transactions in a more efficient, batch-oriented operation, but at less frequent intervals.

4. MySQL Replication Solutions

MySQL provides basic data replication services by a light weight Master and Slave structure [14]. In single-master replication, the master server writes updates to its binary log files. The binary log files serve as a record of updates to be sent to any slave servers. When a slave connects to its master, it informs the master of the position up to which the slave read the logs at its last successful update. The slave receives any updates that have taken place since that time, and then blocks and waits for the master to notify it of new updates. A sanity check is performed to compare result on both sites; if the query failed on one site but succeeded on the other, replication stops.

This simple structure can be arbitrarily combined to build complex architectures. (See Fig 3.4.1). These are essentially the same as Oracle master/ materialize view structures.

Fig 3.4.1 MySQL replication architectures [14]

One problem of MySQL replication mechanism is that in a slow network it is difficult for a slave to catch up with its master. In MySQL 4.0 this problem has been fixed by adding the relay log in the slave: slave read the replication queries from master’s binary log and put into its local relay log, another thread responses to read the queries from relay log and execute them. Another problem of MySQL’s replication is that since binary log only record update information, when add a new slave to a master, the user has to restart the master to ensure the snapshot of the database loaded to the slave is exact the point when slave begins to read from binary log. Moreover it’s almost impossible to add a new slave after the master has been running for a long period of time. The solutions for these expect to be provided in MySQL’s future work.

5. Microsoft SQL Server Replication

Microsoft SQL Server has supported replication since version 6.0. in SQL Server replication, a replicaiton resource is called as a publisher, and the replication target is called as a subscriber. It provides 3 types of replication strategies: snapshot replication, transactional replication and merge replication

Snapshot replication simply takes a "snapshot" of the data on one server and moves that data to another server (or another database on the same server). After the initial synchronization snapshot, replication can refresh data in published tables periodically—based on the schedule you specify. Although snapshot replication is the easiest type to set up and maintain, it requires copying all data each time a table is refreshed. Between scheduled refreshes, data on the original server might be very different from the data on replicas.

Transactional replication involves copying data from the publisher to the subscriber(s) once and then delivering transactions to the subscriber(s) as they occur on the publisher. The initial copy of the data is transported by using the same mechanism as with snapshot replication: SQL Server takes a snapshot of data on the publisher and moves it to the subscriber(s). As database users insert, update, or delete records on the publisher, transactions are forwarded to the subscriber(s).

Merge replication combines data from multiple sources into a single central database. Much like transactional replication, merge replication uses initial synchronization by taking the snapshot of data on the publisher and moving it to subscribers. Unlike transactional replication, merge replication allows changes of the same data on publishers and subscribers, even when subscribers are not connected to the network. When subscribers connect to the network, replication will detect and combine changes from all subscribers and change data on the publisher accordingly. Merge replication is useful when you have a need to modify data on remote computers and when subscribers are not guaranteed to have a continuous connection to the network. [22]

6. Globus Data Replication Service

The Globus project’s Data Replication Service (DRS) (Ann Chervenak, et al.) [3] generalizes the publication functionality of LIGO to provide a higher-level data management services. Fig 3.5.1 describes the workflow of data replication process in DRS.

a. Clients begin by creating a request file (2) (containing replication request file names and destination locations) and sent to the DRS (3) to create the Replicator resource.

b. The Replicator accesses the user’s delegated credential (5) and querying (6) the Replica Location Index to find the Local Replica Catalogs that contain mappings for the requested files. And then queries (7) each of the remote Local Replica Catalogs to get the physical file names associated with the requested files. Thereafter, the Replicator uses the Reliable File Transfer (RFT) service to create a RFT resource (8) and start the transfer.

c. Reliable File Transfer service retrieves the delegated credential (9), and performs the data transfer between sites (11).

d. Newly created replicas are registered (13) to its Local Replica Catalog. And the Local Replica Catalog updates (14) the Replica Location Index at remote sites to make the new replicas visible throughout the Grid environment.

Fig 3.5.1 Architecture of Globus Data Replication Service [3]

7. EDG Replica Manager

The EDG Replica Manager (Peter Kunszt, et al. 2000) [4], shown in Fig 3.6.1, implements 7 modules in its general replica management framework.

a. The Core module implements the main functionality of replica management, (replica creation, deletion, and cataloguing) by interacting with third party services, including Transport services, Replica Location services, MetaData Catalog services and Processing services;

b. The Optimization component is used to minimize file access times by pointing access requests to appropriate replicas and pro-actively replicating frequently used files based on access statistics gathered.

c. The Consistency module guarantees the consistency of replicas of a file and the meta-information stored in various catalogs.

d. The Subscription module takes care of subscription-based replication where data appearing at a data source is automatically replicated to subscribed sites.

e. The Session Management component provides generic check-pointing, restart, and rollback mechanisms to enhance fault tolerance to the system.

f. Collections are defined as sets of logical filenames and other collections.

g. The Security module manages the authentication and authorization, checking allowance for a user to create, delete, read, and write a file.

Fig 3.6.1 Architecture of EDG Replica Manager [4]

8. EGEE Grid Replication Solutions

The Enabling Grids for E-sciencE (EGEE) project [20] (new versions of EDG) builds on top of LCG-2 middleware [21], providing 3 main grid services, job management, data management, and resource information retrieving. The EGEE Data Management Services provides functions of uploading, downloading, and replicating grid data, which characterised as large (> ~20MB), read-only (cannot be modified/deleted), heterogeneous (ascii, binary etc.), and “files” level (no data “structures”), across different Storage Elements. Its data replication system is designed for improving data access efficiency and resilient to failure.

The architecture of EGEE Replication Management System (RMS) and its interactions with other Grid elements is illustrated in Fig 3.7.1.

Fig 3.7.1 EGEE Data Management services [21]

a. The Replica Manager (RM) presents a single interface for the Replication Management System to the user, and interacts with the other EGEE components, e.g. Resource Broker, and Storage Element.

b. The files in the Grid are referred to by different names: Grid Unique Identifier (GUID), Logical File Name (LFN), Storage URL, and Transport URL. Two types of File Catelogs keep mappings between different names:

The Replica Location Service (RLS) maintains information about the physical location of the replicas (mapping with the GUIDs). It is composed of several Local Replica Catalogs (LRCs) which hold the information of replicas for a single VO; and Replica Metadata Catalog (RMC) which stores the mapping between GUIDs and the respective aliases (LFNs) associated with them, and also maintains other metadata information (sizes, dates, ownerships...)
the LCG File Catalog (LFC) is created to overcome performance and functionality issues observed in RLS

9. Storage Resource Broker Project

Storage Resource Broker (SRB) project, proposed by the San Diego Supercomputer Centre, is client-server middleware that provides a uniform interface to access heterogeneous distributed storage resources including, file systems, database systems and archival storage systems in a logical manner using abstraction mechanisms. It also provides a way to access files and computers based on their attributes rather than just names or physical locations. SRB utilizes MCAT, a database system that keeps track of namespaces and the mapping of data objects to storage resources within the federation. MCAT is used to determine the physical location of a given data object, file attributes, metadata, access control lists, storage resource information, and user data.

The SRB middleware consists of one or more SRB Master daemon processes, and SRB Agent processes associated with each Master. The details of one SRB agent is depicted in Fig 3.8.1

Fig 3.8.1 the SRB Agent Architecture

The Dispatcher monitors incoming client requests and delivers requests to the High Level Request Handler (which communicates with MCAT to retrieve metadata information), or the Low-level Request Handler (which communicates with particular storage resource to obtain data). In a federated SRB operation scenario, the High Level Request Handler can also communicate with another SRB server and passes the original open request to a remote MCAT. Returned file handle is sent back to application, and client application may access the data item in a remote server, using the returned file handle. [18]

Replications enabled in SRB include MCAT Replication, Data Replication, Synchronous Replication, Asynchronous Replication, and Out of Band Replication. MCAT Replication allows a series of transactions to be captured on the master MCAT database and replayed at a remote MCAT database. The remote database would be a separate SRB environment with its own MCAT. Vendor specific replication, such as Oracle Advanced Replication or Data Guard could be used to move data. Yet, transaction-based SRB replication isn’t available. Data Replication is used for moving copies of data closer to the user. SRB maintains a hierarchical global name space. Each replica is allocated a persistent SRB identifier. A replica can exist in any type of resource and need not be in the same format. Synchronous Replication is via the logical resource definition and is integrated into the SRB open/create and write functions. In this case, SRB provides consistency of replicas. User are allowed to associate replication with containers/collections as well as individual objects. Asynchronous replication (offline) is possible, which can be initiated by the user or controlling client. The system allows a choice at real time to decide which specific replica should be used (Out of Band Replication), e.g. to use any replica, a specific replica, the next Round-robin replica, the nearest replica, replica with appreciate resource characteristics, a replica with appreciate timestamp or other characteristics, a replica with specific Meta characteristics (user defined metadata and annotations).Once replicated files can be locked while changes are applied. Basic versioning for data within SRB is also available. [19]

10. Grid Data Distribution

The Grid Data Distribution (GDD) [5] project aims to provide functionality supporting Grid Data Access and Notification. It develops variety of operations, such as publication operations, subscription operations, propagation operations, which can be used for Data Replication, third-party data delivery, workflow, etc.

11. IBM Grid Wrapper for WebSphere Information Integrator

IBM WebSphere II is an information integration middleware, providing a range of integration technologies, such as enterprise search, data federation, data replication, data transformation and data event publishing. To interact with a data grid, a new component, Grid Wrapper, was developed to glue WebSphere II and OGSA-DAI, which enables dynamic data resource binding. [15][16]

The architecture of Grid Wrapper was depicted in Fig 3.10.1.

Fig 3.10.1 IBM Grid Wrapper Architecture [15]

a. DB2 is put in front of all relational data sources to unify the access language;

b. WebSphere II provides federation capabilities;

c. OGSA-DAI enables transparent data access to grid data sources;

d. The Registry keeps metadata (data location, access means, structure, QoS, etc.), used for information discovery of the grid on behalf of applications;

e. Access Patterns encapsulate the semantics of data sources and data access means;

f. Transforms transform the SQL of remote data source to a form being recognised by the application.

IBM Grid Wrapper enables data federating in the grid. It reuses existing technology to deal with relational space, combining OGSA-DAI technology to interact with grid data resources, employing a Registry to remove the tight coupling of WebSphere II to the data sources, which allows run-time resource binding, and also enhance system fault tolerance – when data resource has been removed/ failed, it easily switches data access to a replica.

References

[1] SIMDAT project: http://www.scai.fraunhofer.de/710.0.html
[2] LIGO: http://www.ligo.caltech.edu/
[3] Wide Area Data Replication for Scientific Collaborations, Ann Chervenak, Robert Schuler, Carl Kesselman, Scott Koranda, Brian Moe
[4] Data Grid- Replica Management with Reptor, Peter Kunszt, Erwin Laure, Heinz, Stockinger, Kurt Stockinger
[5] Data Distribution in the Grid Environment, Dieter Gawlick, Vitthal Gogate, et al.
[6] Oracle 9i Advanced Replication Release 2 (9.2), http://www.lc.leidenuniv.nl/awcourse/oracle/server.920/a96567/toc.htm
[7] ReplicationGuide and Reference, http://www.pdc.kth.se/doc/SP/manuals/db2-7.1/html/db2e0/frame3.htm#index
[8] The Next Generation: Q Replication, Anthony J. Ciccone and Beth Hamel, http://www.db2mag.com/story/showArticle.jhtml?articleID=23903558
[9] A Practical Guide toDB2 UDB Data Replication V8, http://www.redbooks.ibm.com/redbooks/pdfs/sg246828.pdf
[10] WebSphere Information Integration, http://www-306.ibm.com/software/data/integration/replication.html
[11] Database Replication, Charles Thompson, http://www.dbmsmag.com/9705d15.html
[12] Replication Strategies: Data Migration, Distribution and Synchronization, Sybase White Paper, http://www.sybase.com/content/1028711/6143_whitepaper_v2.pdf
[13] High Performance MySQL: Optimization, Backups, Replication & Load Balancing, Jeremy D.Zawodny, Derek J. Balling, O’REILLY,
http://www.oreilly.com/catalog/hpmysql/chapter/ch07.pdf
[14] MySQL 5.0 Reference Manual, http://dev.mysql.com/doc/refman/5.0/en/replication-intro.html
[15] Bridging the integration gap, Part 1: Federating grid data, Adrian Lee, James Magowan, Patrick Dantressangle, Fabian Bannwart, http://www-128.ibm.com/developerworks/grid/library/gr-feddata/
[16] Grid Wrapper for WebSphere Information Integrator http://www.alphaworks.ibm.com/tech/gridwrapper
[17] SRB Project website: http://www.sdsc.edu/srb/index.php/Main_Page
[18] The SDSC Storage Resource Broker, C. Baru, R.Moore, A. Rajasekar, M. Wan, Proc. CASCON'98 Conference , Nov.30-Dec.3, 1998, Toronto, Canada.
[19] Introduction of Storage Resource Broker: http://www.e-science.clrc.ac.uk/web/projects/storage_resource_broker
[20] EGEE Project http://public.eu-egee.org/
[21] LHC Computing Grid: LCG-2 User Guide Manuals Series https://edms.cern.ch/file/454439/2/LCG-2-UserGuide.pdf
[22]http://www.awprofessional.com/articles/article.asp?p=169612&seqNum=1&rl=1

1. IBM Replication Solutions