HDFS Name Node High Availability

-

NameNode is the heart of HDFS. It stores the namespace for the filesystem and also tracks the location of the blocks in the the cluster. The location of the blocks are not persisted in the NameNode, but the DataNodes report the blocks it has to the NameNode when the DataNode starts. If an instance of NameNode is not available, then HDFS is not accessible till it's back running.

Hadoop 0.23 release introduced HDFS federation where it is possible to have multiple independent NameNodes in a cluster, where in a particular DataNode can have blocks for more than one Name Node. Federation provides horizontal scalability, better performance and isolation.

HDFS NN HA (NameNode High Availability) is an area where active work is happening. Here are the JIRA, Presentation and Video for the same. HDFS NN HA was not cut into 0.23 release and will be part of later releases. Changes are going in the HDFS-1623 branch, if someone is interested in the code.


Edit (11th March, 2012) : Detailed blog entry from Cloudera on HA.

Comments

Post a Comment

thank you for your feedback

Popular posts from this blog

Cloudera Data Hub: Where Agility Meets Control

-
Image
Cloudera’s new Data Hub cloud service, powered by  Cloudera Data Platform , enables users to seamlessly migrate on-premises data management and analytics workloads to the cloud as well as implement new cloud workloads in pursuit of your cloud-first data management strategy. On August 22 nd ,  Cloudera demonstrated its Data Hub  service during a webinar highlighting key business benefits, use cases, and product capabilities. Below is a brief overview of the topics covered and some of the most frequently asked questions from attendees. What is Cloudera Data Hub? Cloudera Data Hub  is a powerful cloud service on Cloudera Data Platform (CDP) that makes it easier, safer, and faster to build modern, mission-critical, data-driven applications with enterprise security, governance, scale, and control. The cloud-native service is powered by a suite of integrated open source technologies that delivers the widest range of analytical workloads such as data marts and data...
See more »

3X FASTER INTERACTIVE QUERY WITH APACHE HIVE LLAP

-
Image
Thanks to Carter Shanklin & Nita Dembla from Hortonworks for valuable post. One of the most exciting new features of HDP 2.6 from Hortonworks was the general availability of Apache Hive with LLAP. If you missed DataWorks Summit you’ll want to look at some of the great LLAP experiences our users shared, including Geisinger who found that  Hive LLAP outperforms their traditional EDW  for most of their queries, and Comcast who found  Hive LLAP is faster than Presto for 75% of benchmark queries . These great results are thanks to performance and stability improvements Hortonworks made to Hive LLAP resulting in 3x faster interactive query in HDP 2.6. This blog dives into the reasons HDP 2.6 is so much faster. We’ll also take a look at the massive step forward Hive has made in SQL compliance with HDP 2.6, enabling Hive to run all 99 TPC-DS queries with only trivial modifications to the original source queries. STARTING OFF: 3X PERFORMANCE GAINS IN HDP 2.6 WITH HIVE ...
See more »

Introduction to HDFS Erasure Coding in Apache Hadoop

-
Image
Thanks to blog contributors from Cloudera Erasure coding, a new feature in HDFS, can reduce storage overhead by approximately 50% compared to replication while maintaining the same durability guarantees. This post explains how it works. HDFS by default replicates each block three times. Replication provides a simple and robust form of redundancy to shield against most failure scenarios. It also eases scheduling compute tasks on locally stored data blocks by providing multiple replicas of each block to choose from. However, replication is expensive: the default 3x replication scheme incurs a 200% overhead in storage space and other resources (e.g., network bandwidth when writing the data). For datasets with relatively low I/O activity, the additional block replicas are rarely accessed during normal operations, but still consume the same amount of storage space. Therefore, a natural improvement is to use erasure coding (EC) in place of replication, which uses far less storage s...
See more »

Big Data Trendz

Loading...