Big Data Trendz

Cloudera’s new Data Hub cloud service, powered by  Cloudera Data Platform , enables users to seamlessly migrate on-premises data management and analytics workloads to the cloud as well as implement new cloud workloads in pursuit of your cloud-first data management strategy. On August 22 nd ,  Cloudera demonstrated its Data Hub  service during a webinar highlighting key business benefits, use cases, and product capabilities. Below is a brief overview of the topics covered and some of the most frequently asked questions from attendees. What is Cloudera Data Hub? Cloudera Data Hub  is a powerful cloud service on Cloudera Data Platform (CDP) that makes it easier, safer, and faster to build modern, mission-critical, data-driven applications with enterprise security, governance, scale, and control. The cloud-native service is powered by a suite of integrated open source technologies that delivers the widest range of analytical workloads such as data marts and data...

Thanks to Cloudera Blog HBase Customers upgrading to CDH 6 from CDH 5, will also get an HBase upgrade moving from HBase1 to HBase2. Performance is an important aspect customers consider.   We measured performance of CDH 5 HBase1 vs CDH 6 HBase2 using YCSB workloads to understand the performance implications of the upgrade on customers doing in-place upgrades (no changes to hardware).  About YCSB For our testing we used the  Yahoo! Cloud Serving Benchmark  (YCSB). YCSB is an open-source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. It is often used to compare relative performance of  NoSQL  database management systems. The original benchmark was developed by workers in the research division of  Yahoo!  who released it in 2010.  More info on YCSB at  https://github.com/brianfrankcooper/YCSB In our test environment YCSB @1TB data scale was used, and run workloads in...

This case study is an instructive example of how performance analysis is a multi-faceted process that often leads one in surprising directions.  Apache Solr Near Real Time (NRT)  Search  allows Solr users to search documents indexed just seconds ago. It’s a critical feature in many real-time analytics applications. As Solr indexes more and more documents in near real time, end-user expectations for performance get higher and higher. However, recently the  Cloudera Search  team found that Solr NRT indexing throughput often hit a bottleneck even when there are plenty of CPU, disk, and network resources available. Latency was average, in the hundreds of milliseconds range. Considering that Solr NRT indexing is a mainly machine-to-machine operation, without a human waiting for indexing to complete, that latency range was actually fairly good. Furthermore, some customers reported other issues under heavy Solr NRT indexing workloads, such as connection resets,...

                                                                                                                                 Source: Cloudera Blog The recently-released Apache Hive 2.0 contains some exciting improvements, many of which are already available in CDH 5.x. Recently, the Apache Hive community announced Hive 2.0.0. This is a larger release...

From Cloudera Blog, thanks to    Justin Kestelyn   for valuable post in cloudera blog. The super-active Apache Spark community is exerting a strong gravitational pull within the Apache Hadoop ecosystem. I recently had that opportunity to ask Cloudera’s Apache Spark committers (Sean Owen, Imran Rashid [PMC], Sandy Ryza, and Marcelo Vanzin) for their perspectives about how the Spark community has worked and is working together, and the work to be done via the  One Platform initiative  to make the Spark stack enterprise-ready. Recently, Apache Spark has become the most currently active project in the Apache Hadoop ecosystem (measured by number of contributors/commits over time), if not the entire ASF. Why do you think that is? Owen: Partly because of scope: Apache Spark has been many sub-projects under an umbrella from the start, some large and complex in their own right, and has tacked on several more in just the last six months. Culture is anothe...

From Clodera Blog: Thanks to  Yanpei Chen, Alan Choi, Dileep Kumar, David Rorke, Silvius Rus, and Devadutta Ghat Impala , the open source MPP query engine designed for high-concurrency SQL over Apache Hadoop, has seen tremendous  adoption  across enterprises in industries such as financial services, telecom, healthcare, retail, gaming, government, and advertising. Impala has unlocked the ability to use business intelligence (BI) applications on Hadoop; these applications support critical business needs such as data discovery, operational dashboards, and reporting. For example, one customer has proven that Impala scales to 80 queries/second, supporting 1,000+ web dashboard end-users with sub-second response time. Clearly, BI applications represent a good fit for Impala, and customers can support more users simply by enlarging their clusters. Cloudera’s previous testing already established that Impala is the  clear winner  among analytic SQL-on-Hadoop alternat...

Thanks to  Michael Yoder The best data protection strategy is to remove sensitive information from everyplace it’s not needed Have you ever wondered what sort of “sensitive” information might wind up in Apache Hadoop log files? For example, if you’re storing credit card numbers inside HDFS, might they ever “leak” into a log file outside of HDFS? What about SQL queries? If you have a query like  select * from table where creditcard = '1234-5678-9012-3456' , where is that query information ultimately stored? This concern affects anyone managing a Hadoop cluster containing sensitive information. At Cloudera, we set out to address this problem through a new feature called  Sensitive Data Redaction , and it’s now available starting in Cloudera Manager 5.4.0 when operating on a CDH 5.4.0 cluster. Specifically, this feature addresses the “leakage” of sensitive information into channels unrelated to the flow of data–not the data stream itself. So, for example, Sensitive...

this post is from cloudera blog, thanks to IIya Ganelin Our thanks to Ilya Ganelin, Senior Data Engineer at Capital One Labs, for the guest post below about his hard-earned lessons from using Spark. I started using Apache Spark in late 2014, learning it at the same time as I learned Scala, so I had to wrap my head around the various complexities of a new language as well as a new computational framework. This process was a great in-depth introduction to the world of Big Data (I previously worked as an electrical engineer for Boeing), and I very quickly found myself deep in the guts of Spark. The hands-on experience paid off; I now feel extremely comfortable with Spark as my go-to tool for a wide variety of data analytics tasks, but my journey here was no cakewalk. Capital One’s original use case for Spark was to surface product recommendations for a set of 25 million users and 10 million products, one of the largest datasets available for this type of modeling. Moreover, we had ...

Apache Hive 1.2.0, although not a major release, contains significant improvements. Recently, the Apache Hive community moved to a more frequent, incremental release schedule. So, a little while ago, we  covered the Apache Hive 1.0.0 release  and explained how it was renamed from 0.14.1 with only minor feature additions since 0.14.0. Shortly thereafter,  Apache Hive 1.1.0  was released (renamed from Apache Hive 0.15.0), which included more significant features—including  Hive-on-Spark . Last week, the community released  Apache Hive 1.2.0 . Although a more narrow release than Hive 1.1.0, it nevertheless contains improvements in the following areas: New Functionality Support for Apache Spark 1.3 ( HIVE-9726 ), enabling dynamic executor allocation and impersonation Support for integration of Hive-on-Spark with Apache HBase ( HIVE-10073 ) Support for numeric partition columns with literals ( HIVE-10313 ,  HIVE-10307 ) Support for Union Dist...

Thanks to Cloudera for article and updated Version of CDH 5.3 Support for transparent, end-to-end encryption in HDFS is now available and production-ready (and shipping inside  CDH 5.3  and later). Here’s how it works. Apache Hadoop 2.6 adds support for transparent encryption to HDFS. Once configured, data read from and written to specified HDFS directories will be transparently encrypted and decrypted, without requiring any changes to user application code. This encryption is also end-to-end, meaning that data can only be encrypted and decrypted by the client. HDFS itself never handles unencrypted data or data encryption keys. All these characteristics improve security, and HDFS encryption can be an important part of an organization-wide data protection story. Cloudera’s HDFS and Cloudera Navigator Key Trustee (formerly Gazzang zTrustee) engineering teams did this work under  HDFS-6134  in collaboration with engineers at Intel as an extension of earlier...

What does a “Big Data engineer” do, and what does “Big Data architecture” look like? In this post, you’ll get answers to both questions. Apache Hadoop has come a long way in its relatively short lifespan. From its beginnings as a reliable storage pool with integrated batch processing using the scalable, parallelizable (though inherently sequential) MapReduce framework, we have witnessed the recent additions of real-time (interactive) components like Impala for interactive SQL queries and integration with Apache Solr as a search engine for free-form text exploration. Getting started is now also a lot easier: Just install CDH, and all the Hadoop ecosystem components are at your disposal. But after installation, where do you go from there? What is a good first use case? How do you ask those “bigger questions”? Having worked with more customers running Hadoop in production than any other vendor, Cloudera’s field technical services team has seen more than its fair share of these use...

Thanks to Cloudera(Source)                                 ---         Get this Copy I Every time follow Cloudera blog and updating their information to share for all technocrats There’s an important new addition coming to the Apache Hadoop book ecosystem. It’s now in early release! We are very happy to announce that the new Apache Hadoop book we have been writing for O’Reilly Media,  Hadoop Application Architectures , is now available as an early release! It contains the first two chapters and can be found in O’Reilly’s Catalog  and via  Safari .         The goal of this book is to give developers and architects guidance on architecting end-to-end solutions using Hadoop and tools in the ecosystem. We have split the book into two broad sections: the first section discusses various considerations for designing applications, and the second sect...

Source: cloudera , Thanks to Cloudera Cloudera Live  is a new way to get started with Apache Hadoop, online. No downloads, no installations, no waiting. Watch tutorial videos and work with real-world examples of the complete Hadoop stack included with CDH, Cloudera’s completely open source Hadoop platform, to: Learn Hue, the Hadoop User Interface developed by Cloudera Query data using popular projects like Apache Hive, Apache Pig, Impala, Apache Solr, and Apache Spark (new!) Develop workflows using Apache Oozie

source: Cloudera blog;  Thanks to Alexander Rubin of Percona Apache Hadoop is commonly used for data analysis. It is fast for data loads and scalable. In a previous post I showed how to  integrate MySQL with Hadoop . In this post I will show how to export a table from  MySQL to Hadoop, load the data to  Cloudera Impala  (columnar format), and run reporting on top of that. For the examples below, I will use the “ontime flight performance” data from my  previous post . I’ve used  Cloudera Manager  to install Hadoop and Impala. For this test I’ve (intentionally) used an old hardware (servers from 2006) to show that Hadoop can utilize the old hardware and still scale. The test cluster consists of 6 datanodes. Below are the specs: Purpose Server specs Namenode, Hive metastore, etc + Datanodes 2x PowerEdge 2950, 2x L5335 CPU @ 2.00GHz, 8 cores, 16GB RAM, RAID 10 with 8 SAS drives Datanodes only 4x PowerEdge SC1425, 2x Xeon ...

Source from Cloudera Also,  Reintroducing Cloudera Enterprise, now with Apache Spark - See more at: cloudera spark Cloudera is announcing the general availability of support for Spark, bringing interactive machine learning and stream processing to enterprise data hubs. Cloudera is pleased to announce the immediate availability of its first release of  Apache Spark  for Cloudera Enterprise (comprising CDH and Cloudera Manager). Spark was created and contributed to the Apache Software Foundation by UC Berkeley, and it has quickly gained adoption for machine learning, interactive analytics, and streaming analytics over large datasets. It features a general programming model for writing applications by composing arbitrary operators, such as mappers, reducers, joins, group-bys, and filters. Spark keeps track of the data that each of the operators produces, enabling applications to reliably store this data in memory, which makes it ideal for low-latency computation...

As a technology geek, I am not sure why I do something :) This experiment falls under the same category. Wanted to setup a Hadoop cluster on my Notebook. I have a  HP 430 Notebook  with a Core i5 processor and 4 GB RAM. I choose to use  Cloudera Manager  for installation of the cluster as it automates most of the installation and configuration required for a Hadoop cluster. Below is how the configuration looks like. On the Laptop Ubuntu 12.04 Desktop has been installed (host OS), which is the OS I use most of the time. On top of it  Oracle VirtualBox  had been installed, so as to enable running one OS (guest) on top of another OS (host). On top of VirtualBox, two instances of Ubuntu 12.04 Server have been installed (Guest OS). There is no need to have a full fledged desktop as nodes. Not only it is unnecessary, but Desktop versions of the OS make the whole thing slower. One of the Guest OS has been configured as a master/slave and other as a sla...