{"id":10656,"date":"2012-10-30T12:40:00","date_gmt":"2012-10-30T11:40:00","guid":{"rendered":"http:\/\/zygm0nt.github.com\/blog\/2012\/10\/30\/hadoop-ha"},"modified":"2023-03-22T16:34:23","modified_gmt":"2023-03-22T15:34:23","slug":"hadoop-ha-setup","status":"publish","type":"post","link":"https:\/\/touk.pl\/blog\/2012\/10\/30\/hadoop-ha-setup\/","title":{"rendered":"Hadoop HA setup"},"content":{"rendered":"

With the advent of Hadoop’s 2.x version, there finally is a working
\nHigh-Availability solution. Even two of those. Now it really is easy to
\nconfigure and use those solutions. It no longer require external
\ncomponents, like
\nDRBD<\/a>.
\nIt all is just neatly packed into Cloudera Hadoop distribution – the
\nprecursor of this solution.<\/p>\n

Read on to find out how to use it.<\/p>\n

<\/p>\n

The most important weakness of previous Hadoop releases was the
\nsingle-point-of-failure, which happend to be NameNode. NameNode as a key
\ncomponent of every Hadoop cluster, is responsible for managing
\nfilesystem namespace information and block location. Loosing its data results in loosing all the data
\nstored on DataNodes. HDFS is no longer able to reach for specific files,
\nor its blocks. This renders your cluster inoperable.<\/p>\n

So it is crucial to be able to detect and counter problems with NameNode.
\nThe most desirable behavior is to have a hot backup, that would ensure
\na no-downtime cluster operation. To achieve this, the second NameNode
\nneed to have up-to-date information on filesystem metadata and it needs
\nto be also up and running. Starting NameNode with existing set of data
\nmay easily take many minutes to parse the actual filesystem state.<\/p>\n

Previously used solution – depoying SecondaryNameNode – was somewhat
\nflawed. It took long time to recover after failure. It was not a
\nhot-backup solution, which also added to the problem. Some other
\nsolution was required.<\/p>\n

So, what needed to be made redundant is the edits dir contents and
\nsending block location maps from each of the DataNodes to NameNodes –
\nin case of HA deployment – to both NameNodes. This was accomplished in
\ntwo steps. The first one with the release of CDH 4 beta – solution based
\non sharing edits directory. Than, with CDH 4.1 came quorum based solution.<\/p>\n

Find out how to configure those on your cluster.<\/p>\n

Shared edits directory solution<\/h2>\n

For this kind of setup, there is an assumption, that in a cluster exists
\na shared storage directory. It should be deployed using some kind of
\nnetwork-based filesystem. You could try with NFS or GlusterFS.<\/p>\n

<property>\r\n  <name>fs.default.name\/name>\r\n  <value>hdfs:\/\/example-cluster<\/value>\r\n<\/property>\r\n<\/pre>\n
<!-- common server name -->\r\n<property>\r\n  <name>dfs.nameservices<\/name>\r\n  <value>example-cluster<\/value>\r\n<\/property>\r\n\r\n<!-- HA configuration -->\r\n<property>\r\n  <name>dfs.ha.namenodes.example-cluster<\/name>\r\n  <value>nn1,nn2<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.rpc-address.example-cluster.nn1<\/name>\r\n  <value>master1:8020<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.rpc-address.example-cluster.nn2<\/name>\r\n  <value>master2:8020<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.http-address.example-cluster.nn1<\/name>\r\n  <value>0.0.0.0:50070<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.http-address.example-cluster.nn2<\/name>\r\n  <value>0.0.0.0:50070<\/value>\r\n<\/property>\r\n\r\n<!-- Storage for edits' files -->\r\n<property>\r\n  <name>dfs.namenode.shared.edits.dir<\/name>\r\n  <value>file:\/\/\/mnt\/filer1\/dfs\/ha-name-dir-shared<\/value>\r\n<\/property>\r\n\r\n<!-- Client failover -->\r\n<property>\r\n  <name>dfs.client.failover.proxy.provider.example-cluster<\/name>\r\n  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<\/value>\r\n<\/property>\r\n\r\n<!-- Fencing configuration -->\r\n<property>\r\n  <name>dfs.ha.fencing.methods<\/name>\r\n  <value>sshfence<\/value>\r\n<\/property>\r\n <property>\r\n  <name>dfs.ha.fencing.ssh.private-key-files<\/name>\r\n  <value>\/home\/user\/.ssh\/id_dsa<\/value>\r\n<\/property>\r\n\r\n\r\n<!-- Automatic failover configuration -->\r\n<property>\r\n  <name>dfs.ha.automatic-failover.enabled<\/name>\r\n  <value>true<\/value>\r\n<\/property>\r\n<property>\r\n  <name>ha.zookeeper.quorum<\/name>\r\n  <value>zk1:2181,zk2:2181,zk3:2181<\/value>\r\n<\/property>\r\n<\/pre>\n
This setup is quite OK, as long as you’re comfortable with maintaining a
\nseparate service (network storage) for handling the HA state. It seems
\nerror prone to me, because it adds another service which high
\navailability should be ensured. NFS seems to be a bad choice here,
\nbecause AFAIK it does not offer HA out of the box.<\/p>\n
On the other hand, we have GlusterFS, which is a distributed filesystem,
\nyou can deploy on multiple bricks and increase the replication level.<\/p>\n
Nevertheless, it still brings additional burden of another service to
\nmaintain.<\/p>\n
Quorum based solution<\/h2>\n
With the release of CDH 4.1.0 we are now able to use a much better
\nintegrated solution called JournalNode. Now all the updates are
\nsynchronized through a JournalNode. Each JournalNode have the same data
\nand all the NameNodes are able to recive filesystem state updates from
\nthat daemons.<\/p>\n
This solution is much more consistent with Hadoop ecosystem.<\/p>\n
Please note, that the config is almost identical to the one needed for
\nshared edits directory solution. The only difference is the value for
\ndfs.namenode.shared.edits.dir<\/em>. This now points to all the journal
\nnodes deployed in our cluster.<\/p>\n
<property>\r\n  <name>fs.default.name\/name>\r\n  <value>hdfs:\/\/example-cluster<\/value>\r\n<\/property><\/pre>\n<\/div>\n
\n
<!-- common server name -->\r\n<property>\r\n  <name>dfs.nameservices<\/name>\r\n  <value>example-cluster<\/value>\r\n<\/property>\r\n\r\n<!-- HA configuration -->\r\n<property>\r\n  <name>dfs.ha.namenodes.example-cluster<\/name>\r\n  <value>nn1,nn2<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.rpc-address.example-cluster.nn1<\/name>\r\n  <value>master1:8020<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.rpc-address.example-cluster.nn2<\/name>\r\n  <value>master2:8020<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.http-address.example-cluster.nn1<\/name>\r\n  <value>0.0.0.0:50070<\/value>\r\n<\/property>\r\n<property>\r\n  <name>dfs.namenode.http-address.example-cluster.nn2<\/name>\r\n  <value>0.0.0.0:50070<\/value>\r\n<\/property>\r\n\r\n<!-- Storage for edits' files -->\r\n<property>\r\n  <name>dfs.namenode.shared.edits.dir<\/name>\r\n  <value>qjournal:\/\/node1:8485;node2:8485;node3:8485\/example-cluster<\/value>\r\n<\/property>\r\n\r\n<!-- Client failover -->\r\n<property>\r\n  <name>dfs.client.failover.proxy.provider.example-cluster<\/name>\r\n  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider<\/value>\r\n<\/property>\r\n\r\n<!-- Fencing configuration -->\r\n<property>\r\n  <name>dfs.ha.fencing.methods<\/name>\r\n  <value>sshfence<\/value>\r\n<\/property>\r\n <property>\r\n  <name>dfs.ha.fencing.ssh.private-key-files<\/name>\r\n  <value>\/home\/user\/.ssh\/id_dsa<\/value>\r\n<\/property>\r\n\r\n\r\n<!-- Automatic failover configuration -->\r\n<property>\r\n  <name>dfs.ha.automatic-failover.enabled<\/name>\r\n  <value>true<\/value>\r\n<\/property>\r\n<property>\r\n  <name>ha.zookeeper.quorum<\/name>\r\n  <value>zk1:2181,zk2:2181,zk3:2181<\/value>\r\n<\/property><\/pre>\n<\/div>\n
Infrastructure<\/h1>\n
In both cases you need to run Zookeeper-based Failover Controller
\n(hadoop-hdfs-zkfc<\/em>). This daemon negotiates which NameNode should
\nbecome active and which standby.<\/p>\n
But that’s not all. Depending on the way you’ve choosen to deploy HA you
\nneed to do some other things:<\/p>\n
Shared edits dir<\/h2>\n
With shared edits dir you need to deploy networked filesystem, and mount
\nit on your NameNodes. After that you can run your cluster and be happy
\nwith your new HA.<\/p>\n
Quroum based<\/h2>\n
For QJournal to operate you need to install one new package called
\nhadoop-hdfs-journalnode<\/em>. This provides startup scripts for Journal
\nNode daemons. Choose at least three nodes that will be responsible for
\nhandling edits state and deploy journal nodes on them.<\/p>\n
Conclusion<\/h1>\n
Thanks to guys from Cloudera we now can use an enterprise grade High
\nAvailability features for Hadoop. Eliminating the single point of
\nfailure in your cluster is essential for easy maintainability of your
\ninfrastructure.<\/p>\n
Given the above choices, I’d suggest using QJournal setup, becasue of
\nits relatively small impact on the overal cluster architecture. It’s
\ngood performance and fairly simple setup enable the users to easily
\nstart using Hadoop in HA setup.<\/p>\n
Are you using Hadoop with HA? What are your impressions?<\/p>\n","protected":false},"excerpt":{"rendered":"With the advent of Hadoop’s 2.x version, there finally is a working High-Availability solution. Even two of those.…\n","protected":false},"author":11,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11],"tags":[],"_links":{"self":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/10656"}],"collection":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/comments?post=10656"}],"version-history":[{"count":9,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/10656\/revisions"}],"predecessor-version":[{"id":15544,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/posts\/10656\/revisions\/15544"}],"wp:attachment":[{"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/media?parent=10656"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/categories?post=10656"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/touk.pl\/blog\/wp-json\/wp\/v2\/tags?post=10656"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}