90 likes | 314 Views
HBase Dev Meetup. 09/11/2012. Enis Soztutar e nis [@] apache [dot] org # enissoz. Integration Testing. Hbase Integration Tests plan HBASE -6201 Create hbase -it module Ability to run integration tests on a given cluster m vn verify
E N D
HBaseDevMeetup 09/11/2012 Enis Soztutar enis [@] apache [dot] org #enissoz
Integration Testing • Hbase Integration Tests plan • HBASE-6201 • Create hbase-it module • Ability to run integration tests on a given cluster mvn verify bin/hbaseorg.apache.hadoop.hbase.IntegrationTestsDriver • Port candidate tests • Add more long running and fault injection tests • (Most of) Integration tests should be able to run over a mini cluster or a 100+ node cluster. • Integration tests should run in mini mode as nightlies (apache jenkins) • Our plan is to run the whole suite of Integration tests nightly/weekly on medium-sized clusters. Every org. should be able to do this easily. Architecting the Future of Big Data
Current Status • hbase-it module committed, HBASE-6203 • HBaseCluster interface for interacting with the cluster from system tests (HBASE-6241) • Patch near to being committed • Subtask jiras for porting each test or adding a new test • Hbase book documentation available in patch for HBASE-6302 • Candidate tests to be converted: • LoadTestTool • BigTopTestLoadAndVerify • Goraci (https://github.com/keith-turner/goraci) • TestAcidGuarantees / TestAtomicOperation • TestRegionBalancing (HBASE-6053) • TestFullLogReconstruction • TestMultiVersions / TestKeepDeletes • TestFromClientSide • TestShell and src/test/ruby • TestRollingRestart • TestMasterFailover • TestImportExport • Test**OnCluster • Balancer tests Architecting the Future of Big Data
HBASE-6241 Overview • New junit @Category(IntegrationTests.class) • Introduces HBaseCluster • public abstract ClusterStatusgetClusterStatus(); • public abstract void startRegionServer(String hostname); • Start/stop/kill RS / Master • MiniHBaseClusterand DistributedHBaseCluster implements HBaseCluster • Introduces DistributedHBaseCluster • Keeps state about a distributed cluster • Does not know about environment. Uses ClusterManager to interact with the cluster • ClusterManager is an Api to manage services • public abstract void start(ServiceType service, String hostname) • Pluggable, specific implementations for every environment • HBaseClusterManager extends ClusterManager • Knows about environment (HBASE_HOME, HBASE_CONF_DIR, HBASE_SSH_OPTS ) • Uses SSH and remotely executes ps, kill -9, bin/hbase-daemon.sh start/stop Architecting the Future of Big Data
HBASE-6241 Overview (cont.) • Introduces HBaseIntegrationTestingUtilityextends HBaseTestingUtility • Integration tests uses this • Introduces 2 ways to run integration tests • bin/hbaseo.a.h.h.IntegrationTestsDriver –config <hbase_conf_dir> • mvn verify (runs tests using mini cluster) • mvn verify –Dhbase.test.cluster.distributed=true –Dhbase.conf.dir=<hbase_conf_dir> (further patch) • Introduces ChaosMonkey • A utility to injects faults in a running cluster. • Does not know about environment, uses HBaseIntegrationTestingUtility • Actions (RestartActiveMaster, RestartRandomRs,RestartRsHoldingMeta, BatchRestartRs, RollingBatchRestartRs, etc ) • Policies (PeriodicRandomActionPolicy), predefined named policies (EVERY_MINUTE_RANDOM_ACTION_POLICY) • Introduces IntegrationTestDataIngestWithChaosMonkey • Runs LoadTestTool and ChaosMonkey for configurable running time • Adjusts data load based on the cluster size (also works with mini cluster) Architecting the Future of Big Data
Open Issues • 0.94 porting? • Hadoop NN HA testing • BIGTOP-635 • Community interest • Different environments (no SSH, etc) • ??? Architecting the Future of Big Data
HBase on Windows • WHY? • Because we can : ) • More platforms = more users + more development • WHO • MS + HW • WHEN? • Patches will start rolling shortly • WHERE? • Apache HBase Trunk • 0.94? 0.92? • WHAT • Hadoopon windows (HADOOP-8079) • Test fixes, file path name related fixes, timing issues, network related changes, build changes, bin scripts, etc Architecting the Future of Big Data
Thanks Go bananas! Taken from http://99designs.com/t-shirt-design/contests/design-chaos- monkey-t-shirt-70909/entries?entriespage=1#contest-header Architecting the Future of Big Data