解决大批量往hbase写数据会导致RegionServer 挂掉的问题

优化Hbase,加大内存都解决不了这个问题

 

解决方法:

rowkey里加个随机整数

  FLOOR(1 + (RAND() * 10))

 

select concat(FLOOR(1 + (RAND() * 10)),(case when A.user_id is null then B.user_id else A.user_id end ),'realName',A.order_time) as rowkey,
(case when A.user_id is null then B.user_id else A.user_id end ) as userId,
'realName' as verifyType,
A.order_id as billNo,
'' as cardNo,
A.card_id as certNo,
A.realname as realName,
A.status as verifyStatus,
'' as add_product,
A.channel as verifyChannel,
A.order_time as verifyTime,
from_unixtime(A.order_time) as createTime,
from_unixtime(A.order_time) as updateTime
from yyd_realname_identify A left join yyd_approve_realname B on A.card_id = B.card_id where (case when A.user_id is null then B.user_id else A.user_id end ) is not null

 

创建Hbase表时 指定分区

 

create 'USER_INFO:unit_verify_log', {NAME => 'UnitCreditVerifyLogEntity', VERSIONS => 1, DATA_BLOCK_ENCODING => 'PREFIX_TREE',BLOOMFILTER=>'ROW',BLOCKCACHE=>true}, SPLITS => ['1', '2', '3', '4', '5', '6', '7', '8', '9']

 

 

 

image

您可以选择一种方式赞助本站