org.apache.hadoop.hbase.test

Class IntegrationTestLoadAndVerify

java.lang.Object

AbstractHBaseTool

org.apache.hadoop.hbase.IntegrationTestBase

org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify

Direct Known Subclasses:

IntegrationTestWithCellVisibilityLoadAndVerify

public class IntegrationTestLoadAndVerify
extends IntegrationTestBase

A large test which loads a lot of data that has internal references, and verifies the data. In load step, 200 map tasks are launched, which in turn write loadmapper.num_to_write (default 100K) rows to an hbase table. Rows are written in blocks, for a total of 100 blocks. Each row in a block, contains loadmapper.backrefs (default 50) references to random rows in the prev block. Verify step is scans the table, and verifies that for every referenced row, the row is actually there (no data loss). Failed rows are output from reduce to be saved in the job output dir in hdfs and inspected later. This class can be run as a unit test, as an integration test, or from the command line Originally taken from Apache Bigtop.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IntegrationTestLoadAndVerify.LoadMapper
static class	IntegrationTestLoadAndVerify.VerifyMapper
static class	IntegrationTestLoadAndVerify.VerifyReducer
static class	IntegrationTestLoadAndVerify.WALSearcher Tool to search missing rows in WALs and hfiles.

Field Summary

Fields inherited from class org.apache.hadoop.hbase.IntegrationTestBase

CHAOS_MONKEY_PROPS, monkey, MONKEY_LONG_OPT, monkeyProps, monkeyToUse, NO_CLUSTER_CLEANUP_LONG_OPT, noClusterCleanUp, util

Constructor Summary

Constructors

Constructor and Description
IntegrationTestLoadAndVerify()

Method Summary

Modifier and Type	Method and Description
void	cleanUpCluster()
protected Job	doLoad(Configuration conf, HTableDescriptor htd)
protected void	doVerify(Configuration conf, HTableDescriptor htd)
protected java.util.Set<java.lang.String>	getColumnFamilies() Provides the name of the CFs that are protected from random Chaos monkey activity (alter)
TableName	getTablename() Provides the name of the table that is protected from random Chaos monkey activity
Path	getTestDir(java.lang.String testName, java.lang.String subdir)
static void	main(java.lang.String[] argv)
void	printUsage()
protected void	processOptions(CommandLine cmd)
int	runTestFromCommandLine()
protected void	setMapperClass(Job job)
void	setUpCluster()
static long	swapLong(long value) Converts a "long" value between endian systems.
void	testLoadAndVerify()

Methods inherited from class org.apache.hadoop.hbase.IntegrationTestBase

addOptions, cleanUp, cleanUpMonkey, cleanUpMonkey, doWork, getConf, getDefaultMonkeyFactory, getTestingUtil, processBaseOptions, setUp, setUpMonkey, startMonkey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IntegrationTestLoadAndVerify

public IntegrationTestLoadAndVerify()

Method Detail

setUpCluster

public void setUpCluster()
                  throws java.lang.Exception

Specified by:

setUpCluster in class IntegrationTestBase

Throws:

java.lang.Exception

cleanUpCluster

public void cleanUpCluster()
                    throws java.lang.Exception

Overrides:

cleanUpCluster in class IntegrationTestBase

Throws:

java.lang.Exception

swapLong

public static long swapLong(long value)

Converts a "long" value between endian systems. Borrowed from Apache Commons IO

Parameters:

value - value to convert

Returns:

the converted value

doLoad

protected Job doLoad(Configuration conf,
                     HTableDescriptor htd)
              throws java.lang.Exception

Throws:

java.lang.Exception

setMapperClass

protected void setMapperClass(Job job)

doVerify

protected void doVerify(Configuration conf,
                        HTableDescriptor htd)
                 throws java.lang.Exception

Throws:

java.lang.Exception

getTestDir

public Path getTestDir(java.lang.String testName,
                       java.lang.String subdir)
                throws java.io.IOException

Throws:

java.io.IOException

testLoadAndVerify

public void testLoadAndVerify()
                       throws java.lang.Exception

Throws:

java.lang.Exception

printUsage

public void printUsage()

processOptions

protected void processOptions(CommandLine cmd)

Overrides:

processOptions in class IntegrationTestBase

runTestFromCommandLine

public int runTestFromCommandLine()
                           throws java.lang.Exception

Specified by:

runTestFromCommandLine in class IntegrationTestBase

Throws:

java.lang.Exception

getTablename

public TableName getTablename()

Description copied from class: IntegrationTestBase

Provides the name of the table that is protected from random Chaos monkey activity

Specified by:

getTablename in class IntegrationTestBase

Returns:

table to not delete.

getColumnFamilies

protected java.util.Set<java.lang.String> getColumnFamilies()

Description copied from class: IntegrationTestBase

Provides the name of the CFs that are protected from random Chaos monkey activity (alter)

Specified by:

getColumnFamilies in class IntegrationTestBase

Returns:

set of cf names to protect.

main

public static void main(java.lang.String[] argv)
                 throws java.lang.Exception

Throws:

java.lang.Exception