NotServingRegionException: 区域 hbase:meta
NotServingRegionException: Region hbase:meta
我们通过 Cloudera Manager (5.4) 安装了 HBase 服务，但出于某种原因，我们决定删除该服务并重新部署 HBase 服务。但是在重新部署 HBase master 后无法以以下
启动
2015-05-13 14:17:56,242 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 73755 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-05-13 14:17:56,809 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:56,815 INFO org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-13 14:17:56,844 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 74358 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-05-13 14:17:58,349 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for region servers count to settle; checked in 1, slept for 75863 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-05-13 14:17:58,355 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://magnetic-hadoop/hbase/WALs/hdp015.abc.com,60020,1431541075296 belongs to an existing region server
2015-05-13 14:17:58,462 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=hdp015.abc.com,60020,1431540681831, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hdp015.abc.com,60020,1431541075296
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2761)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4256)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3623)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116)
at org.apache.hadoop.hbase.ipc.RpcExecutor.run(RpcExecutor.java:96)
at java.lang.Thread.run(Thread.java:745)
2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log dir for server hdp015.abc.com,60020,1431540681831 does not exist
2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [hdp015.abc.com,60020,1431540681831]
2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 0 logs in []
2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [] in 0ms
2015-05-13 14:17:58,469 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in ZooKeeper
2015-05-13 14:17:58,531 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,532 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OFFLINE, ts=1431541078483, server=null} to {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,571 INFO org.apache.hadoop.hbase.master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-05-13 14:17:58,588 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,865 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPEN, ts=1431541078865, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,868 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED of 1588230740 from hdp015.abc.com,60020,1431541075296; deleting unassigned node
2015-05-13 14:17:58,882 INFO org.apache.hadoop.hbase.master.RegionStates: Onlined 1588230740 on hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,884 INFO org.apache.hadoop.hbase.master.HMaster: hbase:meta assigned=1, rit=false, location=hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-05-13 14:17:58,987 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
2015-05-13 14:17:58,996 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Start to scan the hbase:meta for the current region assignment snappshot
2015-05-13 14:17:59,004 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Finished to scan the hbase:meta for the current region assignmentsnapshot
2015-05-13 14:17:59,023 INFO org.apache.hadoop.hbase.master.TableNamespaceManager: Namespace table not found. Creating...
2015-05-13 14:17:59,063 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-13 14:17:59,064 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1069)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:942)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:613)
at java.lang.Thread.run(Thread.java:745)
2015-05-13 14:17:59,076 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: Stopping server on 60000
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.CatalogJanitor: CatalogJanitor-hnn002:60000 exiting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopped
2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopping
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.ClusterStatusChore: hnn002.abc.com,60000,1431540999332-ClusterStatusChore exiting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.BalancerChore: hnn002.abc.com,60000,1431540999332-BalancerChore exiting
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.HFileCleaner: master:hnn002:60000.archivedHFileCleaner exiting
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.LogCleaner: master:hnn002:60000.oldLogCleaner exiting
2015-05-13 14:17:59,081 INFO org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x44d20db9b2a4642, quorum=hdp015.abc.com:2181,hdp005.abc.com:2181,hdp010.abc.com:2181, baseZNode=/hbase
2015-05-13 14:17:59,083 INFO org.mortbay.log: Stopped HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ZooKeeper: Session: 0x44d20db9b2a4642 closed
2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,189 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14d20db9a0f470c
2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470c closed
2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,248 INFO org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor: hnn002.abc.com,60000,1431540999332.splitLogManagerTimeoutMonitor exiting
2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470b closed
2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,305 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2015-05-13 14:17:59,306 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2822)
尝试删除 zookeeper 中的 /hbase，同时确保 HBase Cluster 已关闭：
echo "rmr /hbase" | zookeeper-client
然后尝试启动HBase。
我们通过 Cloudera Manager (5.4) 安装了 HBase 服务，但出于某种原因，我们决定删除该服务并重新部署 HBase 服务。但是在重新部署 HBase master 后无法以以下
启动2015-05-13 14:17:56,242 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 0, slept for 73755 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-05-13 14:17:56,809 INFO org.apache.hadoop.hbase.master.ServerManager: Registering server=hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:56,815 INFO org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-05-13 14:17:56,844 INFO org.apache.hadoop.hbase.master.ServerManager: Waiting for region servers count to settle; currently checked in 1, slept for 74358 ms, expecting minimum of 1, maximum of 2147483647, timeout of 4500 ms, interval of 1500 ms.
2015-05-13 14:17:58,349 INFO org.apache.hadoop.hbase.master.ServerManager: Finished waiting for region servers count to settle; checked in 1, slept for 75863 ms, expecting minimum of 1, maximum of 2147483647, master is running.
2015-05-13 14:17:58,355 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs://magnetic-hadoop/hbase/WALs/hdp015.abc.com,60020,1431541075296 belongs to an existing region server
2015-05-13 14:17:58,462 INFO org.apache.hadoop.hbase.catalog.CatalogTracker: Failed verification of hbase:meta,,1 at address=hdp015.abc.com,60020,1431540681831, exception=org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hdp015.abc.com,60020,1431541075296
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2761)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4256)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:3623)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService.callBlockingMethod(AdminProtos.java:20158)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116)
at org.apache.hadoop.hbase.ipc.RpcExecutor.run(RpcExecutor.java:96)
at java.lang.Thread.run(Thread.java:745)
2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.MasterFileSystem: Log dir for server hdp015.abc.com,60020,1431540681831 does not exist
2015-05-13 14:17:58,467 INFO org.apache.hadoop.hbase.master.SplitLogManager: dead splitlog workers [hdp015.abc.com,60020,1431540681831]
2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: started splitting 0 logs in []
2015-05-13 14:17:58,468 INFO org.apache.hadoop.hbase.master.SplitLogManager: finished splitting (more than or equal to) 0 bytes in 0 log files in [] in 0ms
2015-05-13 14:17:58,469 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker: Unsetting hbase:meta region location in ZooKeeper
2015-05-13 14:17:58,531 INFO org.apache.hadoop.hbase.master.AssignmentManager: Assigning hbase:meta,,1.1588230740 to hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,532 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OFFLINE, ts=1431541078483, server=null} to {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,571 INFO org.apache.hadoop.hbase.master.ServerManager: AssignmentManager hasn't finished failover cleanup; waiting
2015-05-13 14:17:58,588 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=PENDING_OPEN, ts=1431541078532, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,865 INFO org.apache.hadoop.hbase.master.RegionStates: Transitioned {1588230740 state=OPENING, ts=1431541078588, server=hdp015.abc.com,60020,1431541075296} to {1588230740 state=OPEN, ts=1431541078865, server=hdp015.abc.com,60020,1431541075296}
2015-05-13 14:17:58,868 INFO org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED of 1588230740 from hdp015.abc.com,60020,1431541075296; deleting unassigned node
2015-05-13 14:17:58,882 INFO org.apache.hadoop.hbase.master.RegionStates: Onlined 1588230740 on hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,884 INFO org.apache.hadoop.hbase.master.HMaster: hbase:meta assigned=1, rit=false, location=hdp015.abc.com,60020,1431541075296
2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: hbase:meta doesn't have any entries to update.
2015-05-13 14:17:58,970 INFO org.apache.hadoop.hbase.catalog.MetaMigrationConvertingToPB: META already up-to date with PB serialization
2015-05-13 14:17:58,987 INFO org.apache.hadoop.hbase.master.AssignmentManager: Clean cluster startup. Assigning userregions
2015-05-13 14:17:58,996 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Start to scan the hbase:meta for the current region assignment snappshot
2015-05-13 14:17:59,004 INFO org.apache.hadoop.hbase.master.SnapshotOfRegionAssignmentFromMeta: Finished to scan the hbase:meta for the current region assignmentsnapshot
2015-05-13 14:17:59,023 INFO org.apache.hadoop.hbase.master.TableNamespaceManager: Namespace table not found. Creating...
2015-05-13 14:17:59,063 FATAL org.apache.hadoop.hbase.master.HMaster: Master server abort: loaded coprocessors are: []
2015-05-13 14:17:59,064 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.org.apache.hadoop.hbase.TableExistsException: hbase:namespace
at org.apache.hadoop.hbase.master.handler.CreateTableHandler.prepare(CreateTableHandler.java:133)
at org.apache.hadoop.hbase.master.TableNamespaceManager.createNamespaceTable(TableNamespaceManager.java:232)
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:86)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1069)
at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:942)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:613)
at java.lang.Thread.run(Thread.java:745)
2015-05-13 14:17:59,076 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: Stopping server on 60000
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.CatalogJanitor: CatalogJanitor-hnn002:60000 exiting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=60000: stopping
2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopped
2015-05-13 14:17:59,078 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.responder: stopping
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.ClusterStatusChore: hnn002.abc.com,60000,1431540999332-ClusterStatusChore exiting
2015-05-13 14:17:59,077 INFO org.apache.hadoop.hbase.master.balancer.BalancerChore: hnn002.abc.com,60000,1431540999332-BalancerChore exiting
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.HMaster: Stopping infoServer
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.HFileCleaner: master:hnn002:60000.archivedHFileCleaner exiting
2015-05-13 14:17:59,079 INFO org.apache.hadoop.hbase.master.cleaner.LogCleaner: master:hnn002:60000.oldLogCleaner exiting
2015-05-13 14:17:59,081 INFO org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner: Stopping replicationLogCleaner-0x44d20db9b2a4642, quorum=hdp015.abc.com:2181,hdp005.abc.com:2181,hdp010.abc.com:2181, baseZNode=/hbase
2015-05-13 14:17:59,083 INFO org.mortbay.log: Stopped HttpServer$SelectChannelConnectorWithSafeStartup@0.0.0.0:60010
2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ZooKeeper: Session: 0x44d20db9b2a4642 closed
2015-05-13 14:17:59,088 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,189 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x14d20db9a0f470c
2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470c closed
2015-05-13 14:17:59,197 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,248 INFO org.apache.hadoop.hbase.master.SplitLogManager$TimeoutMonitor: hnn002.abc.com,60000,1431540999332.splitLogManagerTimeoutMonitor exiting
2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14d20db9a0f470b closed
2015-05-13 14:17:59,305 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2015-05-13 14:17:59,305 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2015-05-13 14:17:59,306 ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Master exiting java.lang.RuntimeException: HMaster Aborted at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:194)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2822)
尝试删除 zookeeper 中的 /hbase，同时确保 HBase Cluster 已关闭：
echo "rmr /hbase" | zookeeper-client
然后尝试启动HBase。