Big Data Trendz

Source: Cloudera The following post was originally published by the Sentry community at apache.org. We re-publish it here for your convenience. We are very excited to announce that  Apache Sentry  has graduated out of Incubator and is now an Apache Top-Level Project! Sentry, which provides centralized fine-grained access control on metadata and data stored in Apache Hadoop clusters, was introduced as an Apache Incubator project back in August 2013. In the past two and a half years, the development community grew significantly to a large number of contributors from various organizations. Upon graduation, there were more than 50 contributors, 31 of whom had become committers. What’s Sentry? While Hadoop has strong security at the filesystem level, it lacked the granular support needed to adequately secure access to data by users and BI applications. This problem forces users to make a choice: either leave data unprotected or lock out users entirely. Most of the time, the...

Thanks to Lisa Sensmeier and hortonworks Link Bit Refinery is a Hortonworks Technical Partner and recently certified with HDP. Bit Refinery is a VMware© Cloud Infrastructure-as-a-Service (IaaS) provider featuring virtualization technology hosted within their fully redundant virtual data centers. Bit Refinery offers a  hosted Hortonworks Sandbox  providing an easy way to experience and learn Hadoop with ease. All the tutorials available from the Hortonworks Sandbox work just as if you were running a localized version of the Sandbox. Brandon Hieb, Managing Partner at Bit Refinery, is our guest blogger, and in this blog, he provides insight to virtualizing Hadoop infrastructures. Here at Bit Refinery we provide infrastructure for companies large and small which includes a variety of big data applications running on both bare-metal and VMware servers. With this new technology constantly changing, it’s hard to keep up with the different required resources needed which co...

Thanks to Cloudera(Source)                                 ---         Get this Copy I Every time follow Cloudera blog and updating their information to share for all technocrats There’s an important new addition coming to the Apache Hadoop book ecosystem. It’s now in early release! We are very happy to announce that the new Apache Hadoop book we have been writing for O’Reilly Media,  Hadoop Application Architectures , is now available as an early release! It contains the first two chapters and can be found in O’Reilly’s Catalog  and via  Safari .         The goal of this book is to give developers and architects guidance on architecting end-to-end solutions using Hadoop and tools in the ecosystem. We have split the book into two broad sections: the first section discusses various considerations for designing applications, and the second sect...

Thanks to  Charles Lamb, Source: Cloudera Link: here Prefer  IntelliJ IDEA over Eclipse? We’ve got you covered: learn how to get ready to contribute to Apache Hadoop via an IntelliJ project. It’s generally useful to have an IDE at your disposal when you’re developing and debugging code. When I first started working on HDFS,  I used Eclipse , but I’ve recently switched to JetBrains’  IntelliJ IDEA  (specifically, version 13.1 Community Edition). My main motivation was the ease of project setup in the face of Maven and  Google Protocol Buffers  (used in HDFS). The latter is an issue because the code generated by  protoc  ends up in one of the target subdirectories, which can be a configuration headache. The problem is not that Eclipse can’t handle getting these files into the classpath — it’s that in my personal experience, configuration is cumbersome and it takes more time to set up a new project whenever I make a new clone. Conversely...

To Download and Install Hadoop, the prerequisites are 1. Linux based OS 64-bit OS like             Ubuntu             CentOS             Fedora ... etc I preferred to use Ubuntu 12.04LTS, later 14.04 LTS(upcomming version) 2. JAVA 1.6 or 1.7 JDK Go to Downloads folder > cd Downloads Un-zip the hadoop tar file > sudo tar xzf hadoop-1.1.2.tar.gz I created a folder in /home/hduser/ >mkdir Installations Move the Hadoop Un-Zip folder to Installations Directory, pointing as Hadoop >sudo mv /home/hduser/Downloads/hadoop-1.2.1 hadoop Giving some permissions to hadoop folder >sudo addgroup hadoop >sudo chown -R hduser:hadoop hadoop Restart the terminal inorder to get .bashrc file with some content

As with any software (open source or proprietary) active development ceases after a period of time. This may be due to lack of intrest in the developers/users of that particular software or due to availability of a better software. Same is the case with Apache also. All the projects start at  incubator  and based on the progress they make and intrest shown by the community will get promoted to the main. But, before using a particular software or getting involved in it's development, it's important if the development of the software is happening actively and also the bugs are getting fixed on a regular basis. Here are a couple of ways to make sure that the project is active - Check the  mail groups  if the user and the dev groups are active. - Check when was the last, the code has been updated in SVN ( 1 ,  2 ). - Check the open  JIRAs  and how quickly they are getting closed. Apache has got a lot of interesting projects covering a wide area of int...

Google has unique requirements with respective to data processing and storage which no one has. According to the WikiPedia , Google has to process about 24 Peta Bytes of data per day which be a bit outdated and Google might be processing more data per day. So, they need to continuously innovate to address the unique requirements. Soon they outgrow the innovation and they come up with some new innovation. The good thing is that Google had been continuously  releasing  these innovations  as papers once they have it refined and there is a solid internal implementation of it. These Google Papers have been implemented by the  ASF (Apache Software Foundation) and others. It's taking some time for the ASF frameworks like Hadoop and others to production ready. There is a catchup between Google papers and the ASF on a continuous basis. Google Paper Apache Frameworks The Google File System  (October, 2003) HDFS  (2008 became Apache TLP) MapReduce:...