Big Data Trendz

Very often I get the query `What is a cluster?` when discussing about Hadoop and Big Data. To keep it simple ` A cluster is a group or a network of machines wired together acting a single entity to work on a task which when run on a single machine takes much more longer time. ` The given task is split and processed by multiple machines in parallel and so that the task gets completed faster. Jesse Johnson puts it in simple and clear terms what a cluster is all about and how to design distributed algorithms here .                                                 In a Big Data cluster, the machines (or nodes) are neither as powerful as a server grade machine nor as dumb as a desktop machine. Having multiple (like in thousands) server grade ma...

Setting up a Hadoop cluster is all easy with a bit of familiarity with system and network administration. It's all interesting, the only frustrating thing is the downloading of the patches after the installation of the OS and the downloading of the packages for the softwares on top of OS. The downloads can go to all the way close to a GB also, which might take a couple of minutes to hours based on the internet bandwidth. Here is where caching tools really help. They will cache the downloaded packages to one of the designated local machine (lets call it the cache server) and the other machines can point to the cache server to get the packages. This way the packages are downloaded from the internet for the first time and from then on the local cache server will be used for getting the packages. This approach will not only save the network bandwidth, but will also make the whole installation process faster. For debian systems, apt-cacher-ng is designed to cache the ...

Pre-requisites Ubuntu / CentOS Hadoop 1.x/ 2.x , I prefer to install with 2.x Step –> 1: Download and Install Download the Hive from the Apache Download Mirror and i place it in /home/bigdata/Installations/ d irectory. $ cd /home/bigdata/Installation $ wget http://redrockdigimark.com/apachemirror/hive/stable/apache-hive-1.2.1-bin.tar.gz  ( i preferred to download hive-1.2.1 .tar.gz, as it is stable version) $ sudo tar xzf hive-1.2.1.tar.gz Step –> 2: After downloading and installation. Now we are moving to edit hive-env.sh file for Configuration. To configure hive, there I have installed and give permission to bigdata. In $HIVE_HOME/conf/hive-env.sh export JAVA_HOME=/opt/jdk1.80_10  Step –> 3: add hbase path to bashrc $ gedit .bashrc and add following lines to it #HIVE export HIVE_HOME=/home/bigdata/Installations/hive-1.2.1/ export PATH=$PATH:$HIVE_HOME/bin Step –> 4: Restart the terminal and start hadoop, th...

To Download and Install Hadoop, the prerequisites are 1. Linux based OS 64-bit OS like             Ubuntu             CentOS             Fedora ... etc I preferred to use Ubuntu 12.04LTS, later 14.04 LTS(upcomming version) 2. JAVA 1.6 or 1.7 JDK Go to Downloads folder > cd Downloads Un-zip the hadoop tar file > sudo tar xzf hadoop-1.1.2.tar.gz I created a folder in /home/hduser/ >mkdir Installations Move the Hadoop Un-Zip folder to Installations Directory, pointing as Hadoop >sudo mv /home/hduser/Downloads/hadoop-1.2.1 hadoop Giving some permissions to hadoop folder >sudo addgroup hadoop >sudo chown -R hduser:hadoop hadoop Restart the terminal inorder to get .bashrc file with some content

Many a times there will be a requirement of running a group of dependent data processing jobs. Also, we might want to run some of them at regular intervals of time. This is where Apache Oozie fits the picture. Here are some nice articles ( 1 , 2 , 3 , 4 ) on how to use Oozie. Apache Oozie has three components which are a work flow engine to run a DAG of actions, a coordinator (similar to a cron job or a scheduler) and a bundle to batch a group of coordinators. Azkaban from LinkedIn is similar to Oozie, here are the articles ( 1 , 2 ) comparing both of them. Installing and configuring Oozie is not straight forward, not only because of the documentation, but also because the release includes only the source code and not the binaries. The code has to be got, the dependencies installed and then the binaries built. It's a bit tedious process, so this blog with an assumption that Hadoop has been already installed and configured. Here is the official documentation o...