org.apache.hadoop.hbase.regionserver

Class CompactingMemStore

java.lang.Object

org.apache.hadoop.hbase.regionserver.AbstractMemStore

org.apache.hadoop.hbase.regionserver.CompactingMemStore

All Implemented Interfaces:

MemStore

public class CompactingMemStore
extends AbstractMemStore

A memstore implementation which supports in-memory compaction. A compaction pipeline is added between the active set and the snapshot data structures; it consists of a list of segments that are subject to compaction. Like the snapshot, all pipeline segments are read-only; updates only affect the active set. To ensure this property we take advantage of the existing blocking mechanism -- the active set is pushed to the pipeline while holding the region's updatesLock in exclusive mode. Periodically, a compaction is applied in the background to all pipeline segments resulting in a single read-only component. The ``old'' segments are discarded when no scanner is reading them.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CompactingMemStore.IndexType Types of indexes (part of immutable segments) to be used after flattening, compaction, or merge are applied.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.util.concurrent.atomic.AtomicBoolean	allowCompaction
static java.lang.String	COMPACTING_MEMSTORE_TYPE_DEFAULT
static java.lang.String	COMPACTING_MEMSTORE_TYPE_KEY
protected MemStoreCompactor	compactor
static long	DEEP_OVERHEAD
static java.lang.String	IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

Fields inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

active, FIXED_OVERHEAD, snapshot, snapshotId

Constructor Summary

Constructors

Constructor and Description
CompactingMemStore(Configuration conf, CellComparator c, HStore store, RegionServicesForStores regionServices, MemoryCompactionPolicy compactionPolicy)

Method Summary

Modifier and Type	Method and Description
protected void	checkActiveSize() Check whether anything need to be done based on the current active set size.
protected java.util.List<KeyValueScanner>	createList(int capacity)
protected MemStoreCompactor	createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
void	debug()
void	flattenOneSegment(long requesterVersion, MemStoreCompactionStrategy.Action action)
java.lang.String	getFamilyName()
MemStoreSize	getFlushableSize() Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set).
VersionedSegmentsList	getImmutableSegments()
CompactingMemStore.IndexType	getIndexType()
java.util.List<KeyValueScanner>	getScanners(long readPt)
protected java.util.List<Segment>	getSegments()
long	getSmallestReadPoint()
HStore	getStore()
boolean	hasImmutableSegments()
protected long	heapSize()
boolean	isSloppy()
protected long	keySize()
long	preFlushSeqIDEstimation() This method is called before the flush is executed.
protected void	pushActiveToPipeline(MutableSegment active)
void	setCompositeSnapshot(boolean useCompositeSnapshot)
protected boolean	shouldFlushInMemory()
MemStoreSize	size()
MemStoreSnapshot	snapshot() Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long).
void	startReplayingFromWAL() This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL
void	stopReplayingFromWAL() This message intends to inform the MemStore that the replaying edits from WAL are done
boolean	swapCompactedSegments(VersionedSegmentsList versionedList, ImmutableSegment result, boolean merge)
void	updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater) Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Methods inherited from class org.apache.hadoop.hbase.regionserver.AbstractMemStore

add, add, addToScanners, addToScanners, clearSnapshot, dump, getComparator, getConfiguration, getLowest, getNextRow, getSnapshotSize, resetActive, timeOfOldestEdit, toString, upsert

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

COMPACTING_MEMSTORE_TYPE_KEY

public static final java.lang.String COMPACTING_MEMSTORE_TYPE_KEY

See Also:

Constant Field Values

COMPACTING_MEMSTORE_TYPE_DEFAULT

public static final java.lang.String COMPACTING_MEMSTORE_TYPE_DEFAULT

IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

public static final java.lang.String IN_MEMORY_FLUSH_THRESHOLD_FACTOR_KEY

See Also:

Constant Field Values

compactor

protected MemStoreCompactor compactor

allowCompaction

protected final java.util.concurrent.atomic.AtomicBoolean allowCompaction

DEEP_OVERHEAD

public static final long DEEP_OVERHEAD

Constructor Detail

CompactingMemStore

public CompactingMemStore(Configuration conf,
                          CellComparator c,
                          HStore store,
                          RegionServicesForStores regionServices,
                          MemoryCompactionPolicy compactionPolicy)
                   throws java.io.IOException

Throws:

java.io.IOException

Method Detail

createMemStoreCompactor

protected MemStoreCompactor createMemStoreCompactor(MemoryCompactionPolicy compactionPolicy)
                                             throws IllegalArgumentIOException

Throws:

IllegalArgumentIOException

size

public MemStoreSize size()

Returns:

Total memory occupied by this MemStore. This won't include any size occupied by the snapshot. We assume the snapshot will get cleared soon. This is not thread safe and the memstore may be changed while computing its size. It is the responsibility of the caller to make sure this doesn't happen.

preFlushSeqIDEstimation

public long preFlushSeqIDEstimation()

This method is called before the flush is executed.

Returns:

an estimation (lower bound) of the unflushed sequence id in memstore after the flush is executed. if memstore will be cleared returns HConstants.NO_SEQNUM.

isSloppy

public boolean isSloppy()

snapshot

public MemStoreSnapshot snapshot()

Push the current active memstore segment into the pipeline and create a snapshot of the tail of current compaction pipeline Snapshot must be cleared by call to AbstractMemStore.clearSnapshot(long). AbstractMemStore.clearSnapshot(long).

Returns:

MemStoreSnapshot

getFlushableSize

public MemStoreSize getFlushableSize()

Description copied from interface: MemStore

Flush will first clear out the data in snapshot if any (It will take a second flush invocation to clear the current Cell set). If snapshot is empty, current Cell set will be flushed.

Returns:

On flush, how much memory we will clear.

keySize

protected long keySize()

Specified by:

keySize in class AbstractMemStore

Returns:

The total size of cells in this memstore. We will not consider cells in the snapshot

heapSize

protected long heapSize()

Specified by:

heapSize in class AbstractMemStore

Returns:

The total heap size of cells in this memstore. We will not consider cells in the snapshot

updateLowestUnflushedSequenceIdInWAL

public void updateLowestUnflushedSequenceIdInWAL(boolean onlyIfGreater)

Description copied from class: AbstractMemStore

Updates the wal with the lowest sequence id (oldest entry) that is still in memory

Specified by:

updateLowestUnflushedSequenceIdInWAL in class AbstractMemStore

Parameters:

onlyIfGreater - a flag that marks whether to update the sequence id no matter what or only if it is greater than the previous sequence id

startReplayingFromWAL

public void startReplayingFromWAL()

This message intends to inform the MemStore that next coming updates are going to be part of the replaying edits from WAL

stopReplayingFromWAL

public void stopReplayingFromWAL()

This message intends to inform the MemStore that the replaying edits from WAL are done

getSegments

protected java.util.List<Segment> getSegments()

Specified by:

getSegments in class AbstractMemStore

Returns:

an ordered list of segments from most recent to oldest in memstore

setCompositeSnapshot

public void setCompositeSnapshot(boolean useCompositeSnapshot)

swapCompactedSegments

public boolean swapCompactedSegments(VersionedSegmentsList versionedList,
                                     ImmutableSegment result,
                                     boolean merge)

flattenOneSegment

public void flattenOneSegment(long requesterVersion,
                              MemStoreCompactionStrategy.Action action)

Parameters:

requesterVersion - The caller must hold the VersionedList of the pipeline with version taken earlier. This version must be passed as a parameter here. The flattening happens only if versions match.

getIndexType

public CompactingMemStore.IndexType getIndexType()

hasImmutableSegments

public boolean hasImmutableSegments()

getImmutableSegments

public VersionedSegmentsList getImmutableSegments()

getSmallestReadPoint

public long getSmallestReadPoint()

getStore

public HStore getStore()

getFamilyName

public java.lang.String getFamilyName()

getScanners

public java.util.List<KeyValueScanner> getScanners(long readPt)
                                            throws java.io.IOException

Returns:

scanner over the memstore. This might include scanner over the snapshot when one is present.

Throws:

java.io.IOException

createList

protected java.util.List<KeyValueScanner> createList(int capacity)

checkActiveSize

protected void checkActiveSize()

Check whether anything need to be done based on the current active set size. The method is invoked upon every addition to the active set. For CompactingMemStore, flush the active set to the read-only memory if it's size is above threshold

Specified by:

checkActiveSize in class AbstractMemStore

shouldFlushInMemory

protected boolean shouldFlushInMemory()

pushActiveToPipeline

protected void pushActiveToPipeline(MutableSegment active)

debug

public void debug()