HBase is by design a columnar store, that is optimized for random reads.
\nYou just ask for a row using rowId as an identifier and you get your
\ndata instantaneously.<\/p>\n
Performing a scan on part or whole table is a completely different thing.
\nFirst of all, it is sequential. Meaning it is rather slow, because it
\ndoesn’t use all the RegionServers at the same time. It is implemented
\nthat way to realize the contract of Scan command – which has to return
\nresults sorted by key.<\/p>\n
So, how to do this efficiently?<\/p>\n
<\/p>\n
The usual way of getting data from HBase is with the help of its API,
\nmainly Scan objects. To accomplish the task I’ll use just them. I’ll
\nspecify startRow and stopRow, so that each Scan request will be looking
\nthrough only part of the key space.<\/p>\n
It is crucial to note, that this whole solution works because of key
\nsorting properties in HBase. So, HBase scans a table according to ascending key
\nvalues. Since keys are of String type, key with value “1” is smaller
\nthan “2”, because they are sorted lexicographicly. So, also key with value “12345” is smaller than “2”. I’ve
\nleveraged this property so that I’ve partitioned my whole key space according to
\nthe first character of the key. In my case keys contain only digits. So I
\nhave 10 ranges:<\/p>\n
null-1\r\n1-2\r\n2-3\r\n3-4\r\n4-5\r\n5-6\r\n6-7\r\n7-8\r\n8-9\r\n9-null<\/pre>\nThe speedup comes from the fact, that each range resides in its own
\npartition. That’s right, I’ve presplit the table to have 10 partitions.
\nThis corresponds rather nicely with my cluster’s setup, because I have
\nmore than 10 RegionServers. So every partition should be on different
\nRegionServer. It will allow the code to do the requested scan operations
\nin parallel – giving me this exact performance boost.<\/p>\nHow I’ve created the input table:<\/p>\n
$ create 'tariff_changes', { NAME => 'cf', SPLITS_FILE => 'splits.txt', VERSIONS => 50, MAX_FILESIZE => 1073741824 }\r\n\r\n$ alter 'tariff_changes', { NAME => 'cf', SPLITS_FILE => 'splits.txt', VERSIONS => 50, MAX_FILESIZE => 1073741824 }<\/pre>\nSplit file is just something along this lines:<\/p>\n
1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n8\r\n9\r\n0\r\n<\/pre>\nThis tells HBase what are the rowKeys starting and ending each of the
\npartitions.<\/p>\nOk, so after this rather lengthy introduction, what the actual code
\ndoes? It just spins of a few threads – one for each partition – and runs
\na Scan request tailored to that partitions key space. This way, I got a
\n10x speedup for this particular scan. The execution time went down from
\n30 minutes to 3 for my use case.<\/p>\n