org.apache.hadoop.hbase.replication.regionserver

Class ReplicationSourceManager

java.lang.Object

org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager

public class ReplicationSourceManager
extends java.lang.Object

This class is responsible to manage all the replication sources. There are two classes of sources:

• Normal sources are persistent and one per peer cluster
• Old sources are recovered from a failed region server and our only goal is to finish replicating the WAL queue it had up in ZK

When a region server dies, this class uses a watcher to get notified and it tries to grab a lock in order to transfer all the queues in a local old source. This class implements the ReplicationListener interface so that it can track changes in replication state.

Constructor Summary

Constructors

Constructor and Description
ReplicationSourceManager(ReplicationQueues replicationQueues, ReplicationPeers replicationPeers, ReplicationTracker replicationTracker, Configuration conf, Server server, FileSystem fs, Path logDir, Path oldLogDir, java.util.UUID clusterId, WALFileLengthProvider walFileLengthProvider) Creates a replication manager and sets the watch on all the other registered region servers

Method Summary

Modifier and Type	Method and Description
void	addHFileRefs(TableName tableName, byte[] family, java.util.List<<any>> pairs)
void	cleanOldLogs(java.lang.String key, java.lang.String id, boolean queueRecovered) Cleans a log file and all older files from ZK.
void	cleanUpHFileRefs(java.lang.String peerId, java.util.List<java.lang.String> files)
void	closeQueue(ReplicationSourceInterface src) Clear the references to the specified old source
void	closeRecoveredQueue(ReplicationSourceInterface src) Clear the references to the specified old source
void	deleteSource(java.lang.String peerId, boolean closeConnection) Delete a complete queue of wals associated with a peer cluster
FileSystem	getFs() Get the handle on the local file system
Path	getLogDir() Get the directory where wals are stored by their RSs
Path	getOldLogDir() Get the directory where wals are archived
java.util.List<ReplicationSourceInterface>	getOldSources() Get a list of all the old sources of this rs
ReplicationPeers	getReplicationPeers() Get the ReplicationPeers used by this ReplicationSourceManager
ReplicationSourceInterface	getSource(java.lang.String peerId) Get the normal source for a given peer
java.util.List<ReplicationSourceInterface>	getSources() Get a list of all the normal sources of this rs
java.lang.String	getStats() Get a string representation of all the sources' metrics
java.util.concurrent.atomic.AtomicLong	getTotalBufferUsed()
void	join() Terminate the replication on this region server
void	logPositionAndCleanOldLogs(Path log, java.lang.String id, long position, boolean queueRecovered, boolean holdLogInZK) Provide the id of the peer and a log key and this method will figure which wal it belongs to and will log, for this region server, the current position.
void	peerListChanged(java.util.List<java.lang.String> peerIds)
void	peerRemoved(java.lang.String peerId)
void	postLogRoll(Path newLog)
void	preLogRoll(Path newLog)
void	regionServerRemoved(java.lang.String regionserver)
void	removePeer(java.lang.String id) Thie method first deletes all the recovered sources for the specified id, then deletes the normal source (deleting all related data in ZK).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ReplicationSourceManager

public ReplicationSourceManager(ReplicationQueues replicationQueues,
                                ReplicationPeers replicationPeers,
                                ReplicationTracker replicationTracker,
                                Configuration conf,
                                Server server,
                                FileSystem fs,
                                Path logDir,
                                Path oldLogDir,
                                java.util.UUID clusterId,
                                WALFileLengthProvider walFileLengthProvider)
                         throws java.io.IOException

Creates a replication manager and sets the watch on all the other registered region servers

Parameters:

replicationQueues - the interface for manipulating replication queues

replicationPeers -

replicationTracker -

conf - the configuration to use

server - the server for this region server

fs - the file system to use

logDir - the directory that contains all wal directories of live RSs

oldLogDir - the directory where old logs are archived

clusterId -

Throws:

java.io.IOException

Method Detail

logPositionAndCleanOldLogs

public void logPositionAndCleanOldLogs(Path log,
                                       java.lang.String id,
                                       long position,
                                       boolean queueRecovered,
                                       boolean holdLogInZK)

Provide the id of the peer and a log key and this method will figure which wal it belongs to and will log, for this region server, the current position. It will also clean old logs from the queue.

Parameters:

log - Path to the log currently being replicated from replication status in zookeeper. It will also delete older entries.

id - id of the peer cluster

position - current location in the log

queueRecovered - indicates if this queue comes from another region server

holdLogInZK - if true then the log is retained in ZK

cleanOldLogs

public void cleanOldLogs(java.lang.String key,
                         java.lang.String id,
                         boolean queueRecovered)

Cleans a log file and all older files from ZK. Called when we are sure that a log file is closed and has no more entries.

Parameters:

key - Path to the log

id - id of the peer cluster

queueRecovered - Whether this is a recovered queue

deleteSource

public void deleteSource(java.lang.String peerId,
                         boolean closeConnection)

Delete a complete queue of wals associated with a peer cluster

Parameters:

peerId - Id of the peer cluster queue of wals to delete

join

public void join()

Terminate the replication on this region server

getSources

public java.util.List<ReplicationSourceInterface> getSources()

Get a list of all the normal sources of this rs

Returns:

lis of all sources

getOldSources

public java.util.List<ReplicationSourceInterface> getOldSources()

Get a list of all the old sources of this rs

Returns:

list of all old sources

getSource

public ReplicationSourceInterface getSource(java.lang.String peerId)

Get the normal source for a given peer

Parameters:

peerId -

Returns:

the normal source for the give peer if it exists, otherwise null.

preLogRoll

public void preLogRoll(Path newLog)
                throws java.io.IOException

Throws:

java.io.IOException

postLogRoll

public void postLogRoll(Path newLog)
                 throws java.io.IOException

Throws:

java.io.IOException

getTotalBufferUsed

public java.util.concurrent.atomic.AtomicLong getTotalBufferUsed()

closeRecoveredQueue

public void closeRecoveredQueue(ReplicationSourceInterface src)

Clear the references to the specified old source

Parameters:

src - source to clear

closeQueue

public void closeQueue(ReplicationSourceInterface src)

Clear the references to the specified old source

Parameters:

src - source to clear

removePeer

public void removePeer(java.lang.String id)

Thie method first deletes all the recovered sources for the specified id, then deletes the normal source (deleting all related data in ZK).

Parameters:

id - The id of the peer cluster

regionServerRemoved

public void regionServerRemoved(java.lang.String regionserver)

peerRemoved

public void peerRemoved(java.lang.String peerId)

peerListChanged

public void peerListChanged(java.util.List<java.lang.String> peerIds)

getOldLogDir

public Path getOldLogDir()

Get the directory where wals are archived

Returns:

the directory where wals are archived

getLogDir

public Path getLogDir()

Get the directory where wals are stored by their RSs

Returns:

the directory where wals are stored by their RSs

getFs

public FileSystem getFs()

Get the handle on the local file system

Returns:

Handle on the local file system

getReplicationPeers

public ReplicationPeers getReplicationPeers()

Get the ReplicationPeers used by this ReplicationSourceManager

Returns:

the ReplicationPeers used by this ReplicationSourceManager

getStats

public java.lang.String getStats()

Get a string representation of all the sources' metrics

addHFileRefs

public void addHFileRefs(TableName tableName,
                         byte[] family,
                         java.util.List<<any>> pairs)
                  throws ReplicationException

Throws:

ReplicationException

cleanUpHFileRefs

public void cleanUpHFileRefs(java.lang.String peerId,
                             java.util.List<java.lang.String> files)