Big Data Trendz

Learn how to use OCR tools, Apache Spark, and other Apache Hadoop components to process PDF images at scale. Optical character recognition (OCR) technologies have advanced significantly over the last 20 years. However, during that time, there has been little or no effort to marry OCR with distributed architectures such as Apache Hadoop to process large numbers of images in near-real time. In this post, you will learn how to use standard open source tools along with Hadoop components such as Apache Spark, Apache Solr, and Apache HBase to do just that for a medical device information use case. Specifically, you will use a public  dataset  to convert narrative text into searchable fields. Although this example concentrates on medical device information, it can be applied in many other scenarios where processing and persisting images is required. Insurance companies, for example, can make all their scanned documents in claims files searchable for better claim resolu...

From Cloudera Blog, thanks to    Justin Kestelyn   for valuable post in cloudera blog. The super-active Apache Spark community is exerting a strong gravitational pull within the Apache Hadoop ecosystem. I recently had that opportunity to ask Cloudera’s Apache Spark committers (Sean Owen, Imran Rashid [PMC], Sandy Ryza, and Marcelo Vanzin) for their perspectives about how the Spark community has worked and is working together, and the work to be done via the  One Platform initiative  to make the Spark stack enterprise-ready. Recently, Apache Spark has become the most currently active project in the Apache Hadoop ecosystem (measured by number of contributors/commits over time), if not the entire ASF. Why do you think that is? Owen: Partly because of scope: Apache Spark has been many sub-projects under an umbrella from the start, some large and complex in their own right, and has tacked on several more in just the last six months. Culture is anothe...

Thanks to  Michael Yoder The best data protection strategy is to remove sensitive information from everyplace it’s not needed Have you ever wondered what sort of “sensitive” information might wind up in Apache Hadoop log files? For example, if you’re storing credit card numbers inside HDFS, might they ever “leak” into a log file outside of HDFS? What about SQL queries? If you have a query like  select * from table where creditcard = '1234-5678-9012-3456' , where is that query information ultimately stored? This concern affects anyone managing a Hadoop cluster containing sensitive information. At Cloudera, we set out to address this problem through a new feature called  Sensitive Data Redaction , and it’s now available starting in Cloudera Manager 5.4.0 when operating on a CDH 5.4.0 cluster. Specifically, this feature addresses the “leakage” of sensitive information into channels unrelated to the flow of data–not the data stream itself. So, for example, Sensitive...

this post is from cloudera blog, thanks to IIya Ganelin Our thanks to Ilya Ganelin, Senior Data Engineer at Capital One Labs, for the guest post below about his hard-earned lessons from using Spark. I started using Apache Spark in late 2014, learning it at the same time as I learned Scala, so I had to wrap my head around the various complexities of a new language as well as a new computational framework. This process was a great in-depth introduction to the world of Big Data (I previously worked as an electrical engineer for Boeing), and I very quickly found myself deep in the guts of Spark. The hands-on experience paid off; I now feel extremely comfortable with Spark as my go-to tool for a wide variety of data analytics tasks, but my journey here was no cakewalk. Capital One’s original use case for Spark was to surface product recommendations for a set of 25 million users and 10 million products, one of the largest datasets available for this type of modeling. Moreover, we had ...

Thanks to Lisa Sensmeier and hortonworks Link Bit Refinery is a Hortonworks Technical Partner and recently certified with HDP. Bit Refinery is a VMware© Cloud Infrastructure-as-a-Service (IaaS) provider featuring virtualization technology hosted within their fully redundant virtual data centers. Bit Refinery offers a  hosted Hortonworks Sandbox  providing an easy way to experience and learn Hadoop with ease. All the tutorials available from the Hortonworks Sandbox work just as if you were running a localized version of the Sandbox. Brandon Hieb, Managing Partner at Bit Refinery, is our guest blogger, and in this blog, he provides insight to virtualizing Hadoop infrastructures. Here at Bit Refinery we provide infrastructure for companies large and small which includes a variety of big data applications running on both bare-metal and VMware servers. With this new technology constantly changing, it’s hard to keep up with the different required resources needed which co...

for more info/Source : Click Here Hortonworks is excited to announce that our first hands-on, performance based certification exam is now available! The HDP Certified Developer (HDPCD) exam is designed for Hadoop developers working with frameworks like  Pig ,  Hive ,  Sqoop  and  Flume . This new approach to Hadoop certification is designed to allow individuals an opportunity to prove their Hadoop skills in a way that is recognized in the industry as meaningful and relevant to on-the-job performance. Instead of multiple-choice questions, the exam consists of tasks executed on a live, three-node Hortonworks Data Platform cluster: Exam Objectives The exam has three main categories of tasks that involve: Data ingestion Data transformation Data analysis Click  here  to view a detailed list of objectives for the HDPCD exam.

source: Thanks to CLOUDERA for technical updates Hadoop Security  is the latest book from Cloudera engineers in the  Hadoop ecosystem books canon . We are thrilled to announce the availability of the early release of  Hadoop Security , a new book about security in the Apache Hadoop ecosystem published by O’Reilly Media. The early release contains two chapters on System Architecture and Securing Data Ingest and is available in  O’Reilly’s catalog  and in  Safari Books . The goal of the book is to serve the experienced security architect that has been tasked with integrating Hadoop into a larger enterprise security context. System and application administrators also benefit from a thorough treatment of the risks inherent in deploying Hadoop in production and the associated how and why of Hadoop security. As Hadoop continues to mature and become ever more widely adopted, material must become specialized for the security architects tasked with ensur...

What does a “Big Data engineer” do, and what does “Big Data architecture” look like? In this post, you’ll get answers to both questions. Apache Hadoop has come a long way in its relatively short lifespan. From its beginnings as a reliable storage pool with integrated batch processing using the scalable, parallelizable (though inherently sequential) MapReduce framework, we have witnessed the recent additions of real-time (interactive) components like Impala for interactive SQL queries and integration with Apache Solr as a search engine for free-form text exploration. Getting started is now also a lot easier: Just install CDH, and all the Hadoop ecosystem components are at your disposal. But after installation, where do you go from there? What is a good first use case? How do you ask those “bigger questions”? Having worked with more customers running Hadoop in production than any other vendor, Cloudera’s field technical services team has seen more than its fair share of these use...

Thanks to Cloudera(Source)                                 ---         Get this Copy I Every time follow Cloudera blog and updating their information to share for all technocrats There’s an important new addition coming to the Apache Hadoop book ecosystem. It’s now in early release! We are very happy to announce that the new Apache Hadoop book we have been writing for O’Reilly Media,  Hadoop Application Architectures , is now available as an early release! It contains the first two chapters and can be found in O’Reilly’s Catalog  and via  Safari .         The goal of this book is to give developers and architects guidance on architecting end-to-end solutions using Hadoop and tools in the ecosystem. We have split the book into two broad sections: the first section discusses various considerations for designing applications, and the second sect...

Source: cloudera , Thanks to Cloudera Cloudera Live  is a new way to get started with Apache Hadoop, online. No downloads, no installations, no waiting. Watch tutorial videos and work with real-world examples of the complete Hadoop stack included with CDH, Cloudera’s completely open source Hadoop platform, to: Learn Hue, the Hadoop User Interface developed by Cloudera Query data using popular projects like Apache Hive, Apache Pig, Impala, Apache Solr, and Apache Spark (new!) Develop workflows using Apache Oozie

I am big fan of automation and recently was looking for a way to automatically tweet when I publish a new blog. Found  ifttt.com  which allows to create recipes like these and share with others. The recipes run every 15 minutes, so there is a delay of maximum 15 minutes between publishing a new blog and getting it posted into Twitter. The acronym IFTTT is a bit cryptic to remember and expands to ` IF   T his  T hen  T hat`. The service had been running for almost 4 years, but had been a bit flaky when using it. Was not able to create recipes for the first time and was also not able to add LinkedIn as a channel. Also, it's not possible to create multiple channels of the same type. For example, the new blog event cannot be send to multiple Twitter account. I had to create multiple accounts with IFTTT so as to add multiple Twitter channels. Also, creating multiple triggers for a single recipe is not possible for now. Also, it would be nice to have some co...

There are a lot of nice videos in YouTube from  tops for kids  to  machine learning . Some of these videos are so interesting, feel like viewing them again and again. When you find this pattern, it's better to download the videos. Not only does this allow for offline view, but also save the bandwidth.  Bandwidth cap  makes this even more useful. `youtube-dl` is a very useful command to download files from YouTube in Ubuntu. `youtube-dl`has got a lot of nice options, here are some of the options I use youtube-dl -c -t -f 5 --batch-file=files.txt -c -> resume partially downloaded file -t -> Use the title of the video in the file name used to download the video -f -> Specify the video format (quality) in which to download the video. --batch-file -> Specify the name of a file containing URLs of videos to download from youtube in batch mode. The file must contain one URL per line.

Setting up a Hadoop cluster is all easy with a bit of familiarity with system and network administration. It's all interesting, the only frustrating thing is the downloading of the patches after the installation of the OS and the downloading of the packages for the softwares on top of OS. The downloads can go to all the way close to a GB also, which might take a couple of minutes to hours based on the internet bandwidth. Here is where caching tools really help. They will cache the downloaded packages to one of the designated local machine (lets call it the cache server) and the other machines can point to the cache server to get the packages. This way the packages are downloaded from the internet for the first time and from then on the local cache server will be used for getting the packages. This approach will not only save the network bandwidth, but will also make the whole installation process faster. For debian systems, apt-cacher-ng is designed to cache the ...

Pre-requisites Ubuntu / CentOS Hadoop 1.x/ 2.x , I prefer to install with 2.x Step –> 1: Download and Install Download the Hive from the Apache Download Mirror and i place it in /home/bigdata/Installations/ d irectory. $ cd /home/bigdata/Installation $ wget http://redrockdigimark.com/apachemirror/hive/stable/apache-hive-1.2.1-bin.tar.gz  ( i preferred to download hive-1.2.1 .tar.gz, as it is stable version) $ sudo tar xzf hive-1.2.1.tar.gz Step –> 2: After downloading and installation. Now we are moving to edit hive-env.sh file for Configuration. To configure hive, there I have installed and give permission to bigdata. In $HIVE_HOME/conf/hive-env.sh export JAVA_HOME=/opt/jdk1.80_10  Step –> 3: add hbase path to bashrc $ gedit .bashrc and add following lines to it #HIVE export HIVE_HOME=/home/bigdata/Installations/hive-1.2.1/ export PATH=$PATH:$HIVE_HOME/bin Step –> 4: Restart the terminal and start hadoop, th...

Source from Cloudera Also,  Reintroducing Cloudera Enterprise, now with Apache Spark - See more at: cloudera spark Cloudera is announcing the general availability of support for Spark, bringing interactive machine learning and stream processing to enterprise data hubs. Cloudera is pleased to announce the immediate availability of its first release of  Apache Spark  for Cloudera Enterprise (comprising CDH and Cloudera Manager). Spark was created and contributed to the Apache Software Foundation by UC Berkeley, and it has quickly gained adoption for machine learning, interactive analytics, and streaming analytics over large datasets. It features a general programming model for writing applications by composing arbitrary operators, such as mappers, reducers, joins, group-bys, and filters. Spark keeps track of the data that each of the operators produces, enabling applications to reliably store this data in memory, which makes it ideal for low-latency computation...

Those who are interested to work on Hadoop, One commonly asked question that I got from these people  is  what Hadoop feature can I work on? Here are some items that I have in mind that are good topics for students to attempt if they want to work in Hadoop. Ability to make Hadoop scheduler resource aware, especially CPU, memory and IO resources. The current implementation is based on statically configured  slots. Abilty to make a map-reduce job take new input splits even after a map-reduce job has already started. Ability to dynamically increase replicas of data in HDFS based on access patterns. This is needed to handle hot-spots of data. Ability to extend the map-reduce framework to be able to process data that resides partly in memory. One assumption of the current implementation is that the map-reduce framework is used to scan data that resides on disk devices. But memory on commodity machines is becoming larger and larger. A cluster of 3000 machines with ...

To Download and Install Hadoop, the prerequisites are 1. Linux based OS 64-bit OS like             Ubuntu             CentOS             Fedora ... etc I preferred to use Ubuntu 12.04LTS, later 14.04 LTS(upcomming version) 2. JAVA 1.6 or 1.7 JDK Go to Downloads folder > cd Downloads Un-zip the hadoop tar file > sudo tar xzf hadoop-1.1.2.tar.gz I created a folder in /home/hduser/ >mkdir Installations Move the Hadoop Un-Zip folder to Installations Directory, pointing as Hadoop >sudo mv /home/hduser/Downloads/hadoop-1.2.1 hadoop Giving some permissions to hadoop folder >sudo addgroup hadoop >sudo chown -R hduser:hadoop hadoop Restart the terminal inorder to get .bashrc file with some content

                                                   I have a 64 bit version of Ubuntu 12.04 LTS installed, so the instructions below only apply to this OS. Download the Java JDK from http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase6-419409.html#jdk-6u45-oth-JPR . 1.            Click  Accept License Agreement 2.            Click  jdk-6u45-linux-x64.bin 3.            Login to Oracle.com with your Oracle account 4.            Download the JDK to your  ~/Downloads  directory 5.            After downloading, open a terminal, then enter ...

I am happy to announce that Splunk Hadoop Connect 1.1 is now available. This version of Hadoop Connect rounds out Splunk’s integration with the Hadoop distributions by becoming certified on  MapR .  Cloudera , Hortonworks , and  Apache  Hadoop distributions also have the ability to benefit from the power of Splunk. Splunk Hadoop Connect provides bi-directional integration to easily and reliably move data between Splunk and Hadoop. It provides Hadoop users the ability to gain real-time analysis, visualization and role based access control for a stream of machine-generated data. It delivers three core capacities: Export data from Splunk to Hadoop, Explore Hadoop directories and Import data from Hadoop to Splunk. The most significant new feature added to version 1.1 is the ability to select whether you want to map to a remote HDFS cluster or to a mounted file system. The option to map to a mounted file system enables Splunk to integrate with MapR Hadoop distr...

Many a times there will be a requirement of running a group of dependent data processing jobs. Also, we might want to run some of them at regular intervals of time. This is where Apache Oozie fits the picture. Here are some nice articles ( 1 , 2 , 3 , 4 ) on how to use Oozie. Apache Oozie has three components which are a work flow engine to run a DAG of actions, a coordinator (similar to a cron job or a scheduler) and a bundle to batch a group of coordinators. Azkaban from LinkedIn is similar to Oozie, here are the articles ( 1 , 2 ) comparing both of them. Installing and configuring Oozie is not straight forward, not only because of the documentation, but also because the release includes only the source code and not the binaries. The code has to be got, the dependencies installed and then the binaries built. It's a bit tedious process, so this blog with an assumption that Hadoop has been already installed and configured. Here is the official documentation o...