org.apache.hadoop.hbase.regionserver.wal

Class AsyncFSWAL

java.lang.Object

org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL<AsyncWriter>

org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL

public class AsyncFSWAL
extends AbstractFSWAL<AsyncWriter>

An asynchronous implementation of FSWAL.

Here 'waitingConsumePayloads' acts as the RingBuffer in FSHLog.

For append, we process it as follow:

1. In the caller thread(typically, in the rpc handler thread):
   1. Insert the entry into 'waitingConsumePayloads'. Use ringbuffer sequence as txid.
   2. Schedule the consumer task if needed. See shouldScheduleConsumer() for more details.
2. In the consumer task(executed in a single threaded thread pool)
   1. Poll the entry from waitingConsumePayloads and insert it into toWriteAppends
   2. Poll the entry from toWriteAppends, append it to the AsyncWriter, and insert it into unackedAppends
   3. If the buffered size reaches batchSize, or there is a sync request, then we call sync on the AsyncWriter.
   4. In the callback methods:
      • If succeeded, poll the entry from unackedAppends and drop it.
      • If failed, add all the entries in unackedAppends back to toWriteAppends and wait for writing them again.

For sync, the processing stages are almost same. And different from FSHLog, we will open a new writer and rewrite unacked entries to the new writer and sync again if we hit a sync error.

Here we only describe the logic of doReplaceWriter. The main logic of rollWriter is same with FSHLog.
For a normal roll request(for example, we have reached the log roll size):

1. In the log roller thread, we will set waitingRoll(int) to true and readyForRolling to false, and then wait on readyForRolling(see waitForSafePoint()).
2. In the consumer thread, we will stop polling entries from waitingConsumePayloads if waitingRoll(int) is true, and also stop writing the entries in toWriteAppends out.
3. If there are unflush data in the writer, sync them.
4. When all out-going sync request is finished, i.e, the unackedAppends is empty, signal the readyForRollingCond.
5. Back to the log roller thread, now we can confirm that there are no out-going entries, i.e., we reach a safe point. So it is safe to replace old writer with new writer now.
6. Set writerBroken(int) and waitingRoll(int) to false.
7. Schedule the consumer task.
8. Schedule a background task to close the old writer.

For a broken writer roll request, the only difference is that we can bypass the wait for safe point stage.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	ASYNC_WAL_USE_SHARED_EVENT_LOOP
static java.lang.String	ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS
static boolean	DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP
static int	DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS
static long	DEFAULT_WAL_BATCH_SIZE
static java.lang.String	WAL_BATCH_SIZE

Fields inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

blocksize, closed, conf, coprocessorHost, DEFAULT_SLOW_SYNC_TIME_MS, filenum, fs, highestSyncedTxid, highestUnsyncedTxid, implClassName, listeners, logrollsize, maxLogs, numEntries, ourFiles, prefixPathStr, rollWriterLock, sequenceIdAccounting, shutdown, slowSyncNs, totalLogSize, walArchiveDir, walDir, walFile2Props, walFilePrefix, walFileSuffix

Constructor Summary

Constructors

Constructor and Description
AsyncFSWAL(FileSystem fs, Path rootDir, java.lang.String logDir, java.lang.String archiveDir, Configuration conf, java.util.List<WALActionsListener> listeners, boolean failIfWALExists, java.lang.String prefix, java.lang.String suffix, EventLoopGroup eventLoopGroup, java.lang.Class<? extends Channel> channelClass)

Method Summary

Modifier and Type	Method and Description
long	append(RegionInfo hri, WALKeyImpl key, WALEdit edits, boolean inMemstore) NOTE: This append, at a time that is usually after this call returns, starts an mvcc transaction by calling 'begin' wherein which we assign this update a sequenceid.
protected AsyncWriter	createWriterInstance(Path path)
protected void	doAppend(AsyncWriter writer, org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry)
protected boolean	doCheckLogLowReplication()
protected void	doReplaceWriter(Path oldPath, Path newPath, AsyncWriter nextWriter)
protected void	doShutdown()
void	sync()

Methods inherited from class org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL

abortCacheFlush, append, blockOnSync, checkLogLowReplication, close, completeCacheFlush, computeFilename, getCoprocessorHost, getCurrentFileName, getEarliestMemStoreSeqNum, getEarliestMemStoreSeqNum, getFilenum, getFileNumFromFileName, getLogFileSize, getLogFileSizeIfBeingWritten, getNumLogFiles, getNumRolledLogFiles, getPreallocatedEventCount, getSyncFuture, getWALArchivePath, logRollAndSetupWalProps, main, postSync, registerWALActionsListener, requestLogRoll, requestLogRoll, rollWriter, rollWriter, shutdown, stampSequenceIdAndPublishToRingBuffer, startCacheFlush, startCacheFlush, toString, unregisterWALActionsListener, updateStore

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

WAL_BATCH_SIZE

public static final java.lang.String WAL_BATCH_SIZE

See Also:

Constant Field Values

DEFAULT_WAL_BATCH_SIZE

public static final long DEFAULT_WAL_BATCH_SIZE

See Also:

Constant Field Values

ASYNC_WAL_USE_SHARED_EVENT_LOOP

public static final java.lang.String ASYNC_WAL_USE_SHARED_EVENT_LOOP

See Also:

Constant Field Values

DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP

public static final boolean DEFAULT_ASYNC_WAL_USE_SHARED_EVENT_LOOP

See Also:

Constant Field Values

ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

public static final java.lang.String ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

Constant Field Values

DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

public static final int DEFAULT_ASYNC_WAL_WAIT_ON_SHUTDOWN_IN_SECONDS

See Also:

Constant Field Values

Constructor Detail

AsyncFSWAL

public AsyncFSWAL(FileSystem fs,
                  Path rootDir,
                  java.lang.String logDir,
                  java.lang.String archiveDir,
                  Configuration conf,
                  java.util.List<WALActionsListener> listeners,
                  boolean failIfWALExists,
                  java.lang.String prefix,
                  java.lang.String suffix,
                  EventLoopGroup eventLoopGroup,
                  java.lang.Class<? extends Channel> channelClass)
           throws FailedLogCloseException,
                  java.io.IOException

Throws:

FailedLogCloseException

java.io.IOException

Method Detail

append

public long append(RegionInfo hri,
                   WALKeyImpl key,
                   WALEdit edits,
                   boolean inMemstore)
            throws java.io.IOException

Description copied from class: AbstractFSWAL

NOTE: This append, at a time that is usually after this call returns, starts an mvcc transaction by calling 'begin' wherein which we assign this update a sequenceid. At assignment time, we stamp all the passed in Cells inside WALEdit with their sequenceId. You must 'complete' the transaction this mvcc transaction by calling MultiVersionConcurrencyControl#complete(...) or a variant otherwise mvcc will get stuck. Do it in the finally of a try/finally block within which this append lives and any subsequent operations like sync or update of memstore, etc. Get the WriteEntry to pass mvcc out of the passed in WALKey walKey parameter. Be warned that the WriteEntry is not immediately available on return from this method. It WILL be available subsequent to a sync of this append; otherwise, you will just have to wait on the WriteEntry to get filled in.

Specified by:

append in class AbstractFSWAL<AsyncWriter>

Throws:

java.io.IOException

sync

public void sync()
          throws java.io.IOException

Throws:

java.io.IOException

createWriterInstance

protected AsyncWriter createWriterInstance(Path path)
                                    throws java.io.IOException

Specified by:

createWriterInstance in class AbstractFSWAL<AsyncWriter>

Throws:

java.io.IOException

doReplaceWriter

protected void doReplaceWriter(Path oldPath,
                               Path newPath,
                               AsyncWriter nextWriter)
                        throws java.io.IOException

Specified by:

doReplaceWriter in class AbstractFSWAL<AsyncWriter>

Throws:

java.io.IOException

doShutdown

protected void doShutdown()
                   throws java.io.IOException

Specified by:

doShutdown in class AbstractFSWAL<AsyncWriter>

Throws:

java.io.IOException

doAppend

protected void doAppend(AsyncWriter writer,
                        org.apache.hadoop.hbase.regionserver.wal.FSWALEntry entry)

Specified by:

doAppend in class AbstractFSWAL<AsyncWriter>

doCheckLogLowReplication

protected boolean doCheckLogLowReplication()

Specified by:

doCheckLogLowReplication in class AbstractFSWAL<AsyncWriter>