Big Data Trendz

MRv2  is a revamp of the MapReduce engine for making Hadoop reliable, available, scalable and also for better cluster utilization. MRv2 development had been under active development for some time and alpha are being  released  now. The Cloudera  article  is very useful for getting the code from SVN, building, deploying and finally running a Job to make sure Hadoop has been setup properly. Here are some additional steps -  Protocol Buffers  is used as an RPC protocol between different daemons. The  recomendation  is to use Protocol Buffer version 2.4.1+. Some of the Linux releases don't have this version. So, Protocol Buffers code has to be downloaded, built and installed using the `configure, make, make install` command. `make install` will require administrative privileges. `protoc --version` will give the version number. - In Ubuntu 11.10, g++ was not installed by default. `sudo apt-get install g++` installed the required binaries....

Distributed applications are by nature difficult to debug, Hadoop is no exception. This blog entry will try to explain how to put break points and debug a user defined Java MapReduce program in Eclipse. Hadoop support executing a MapReduce job in  Standalone ,  Pseudo-Distributed  and  Fully-Distributed Mode . As we move from one more to another in the same order, the debugging becomes harder and new bugs are found on the way. Standalone mode with the default Hadoop configuration properties allows MapReduce programs to be debugged in Eclipse. Step 1:  Create a Java Project in Eclipse. Step 2:  For the Java project created in the earlier step, add the following dependencies (commons-configuration-1.6.jar, commons-httpclient-3.0.1.jar, commons-lang-2.4.jar, commons-logging-1.1.1.jar, commons-logging-api-1.0.4.jar, hadoop-core-1.0.3.jar, jackson-core-asl-1.8.8.jar, jackson-mapper-asl-1.8.8.jar and log4j-1.2.15.jar) in Eclipse. The dependencies a...