Big Data Trendz

Cloudera’s new Data Hub cloud service, powered by  Cloudera Data Platform , enables users to seamlessly migrate on-premises data management and analytics workloads to the cloud as well as implement new cloud workloads in pursuit of your cloud-first data management strategy. On August 22 nd ,  Cloudera demonstrated its Data Hub  service during a webinar highlighting key business benefits, use cases, and product capabilities. Below is a brief overview of the topics covered and some of the most frequently asked questions from attendees. What is Cloudera Data Hub? Cloudera Data Hub  is a powerful cloud service on Cloudera Data Platform (CDP) that makes it easier, safer, and faster to build modern, mission-critical, data-driven applications with enterprise security, governance, scale, and control. The cloud-native service is powered by a suite of integrated open source technologies that delivers the widest range of analytical workloads such as data marts and data...

Thanks to Carter Shanklin & Nita Dembla from Hortonworks for valuable post. One of the most exciting new features of HDP 2.6 from Hortonworks was the general availability of Apache Hive with LLAP. If you missed DataWorks Summit you’ll want to look at some of the great LLAP experiences our users shared, including Geisinger who found that  Hive LLAP outperforms their traditional EDW  for most of their queries, and Comcast who found  Hive LLAP is faster than Presto for 75% of benchmark queries . These great results are thanks to performance and stability improvements Hortonworks made to Hive LLAP resulting in 3x faster interactive query in HDP 2.6. This blog dives into the reasons HDP 2.6 is so much faster. We’ll also take a look at the massive step forward Hive has made in SQL compliance with HDP 2.6, enabling Hive to run all 99 TPC-DS queries with only trivial modifications to the original source queries. STARTING OFF: 3X PERFORMANCE GAINS IN HDP 2.6 WITH HIVE ...

Thanks to Vinay Shukla( Leading Data Science Product Management at Hortonworks) Our customers increasingly leverage Data Science, and Machine Learning to solve complex predictive analytics problem. A few examples of these problems are churn prediction, predictive maintenance, image classification, and entity matching. While everyone wants to predict the future, truly leveraging Data Science for Predictive Analytics remains the domain of a select few. To expand the reach of Data Science, the Modern Data Architecture (MDA) needs to address the following 4 requirements: Enable Apps to consume predictions and become smarter Bring predictive analytics to the IOT Edge Become easier, more accurate & faster to deploy and manage Fully support  data science life cycle The below diagram represents where Data science fits in the MDA. DATA SMART APPLICATIONS The end-users consumes data, analytics and the results of Data Science analytics via data centric applications (or...

source: Thanks to CLOUDERA for technical updates Hadoop Security  is the latest book from Cloudera engineers in the  Hadoop ecosystem books canon . We are thrilled to announce the availability of the early release of  Hadoop Security , a new book about security in the Apache Hadoop ecosystem published by O’Reilly Media. The early release contains two chapters on System Architecture and Securing Data Ingest and is available in  O’Reilly’s catalog  and in  Safari Books . The goal of the book is to serve the experienced security architect that has been tasked with integrating Hadoop into a larger enterprise security context. System and application administrators also benefit from a thorough treatment of the risks inherent in deploying Hadoop in production and the associated how and why of Hadoop security. As Hadoop continues to mature and become ever more widely adopted, material must become specialized for the security architects tasked with ensur...

What does a “Big Data engineer” do, and what does “Big Data architecture” look like? In this post, you’ll get answers to both questions. Apache Hadoop has come a long way in its relatively short lifespan. From its beginnings as a reliable storage pool with integrated batch processing using the scalable, parallelizable (though inherently sequential) MapReduce framework, we have witnessed the recent additions of real-time (interactive) components like Impala for interactive SQL queries and integration with Apache Solr as a search engine for free-form text exploration. Getting started is now also a lot easier: Just install CDH, and all the Hadoop ecosystem components are at your disposal. But after installation, where do you go from there? What is a good first use case? How do you ask those “bigger questions”? Having worked with more customers running Hadoop in production than any other vendor, Cloudera’s field technical services team has seen more than its fair share of these use...

Source: cloudera , Thanks to Cloudera Cloudera Live  is a new way to get started with Apache Hadoop, online. No downloads, no installations, no waiting. Watch tutorial videos and work with real-world examples of the complete Hadoop stack included with CDH, Cloudera’s completely open source Hadoop platform, to: Learn Hue, the Hadoop User Interface developed by Cloudera Query data using popular projects like Apache Hive, Apache Pig, Impala, Apache Solr, and Apache Spark (new!) Develop workflows using Apache Oozie

source: Cloudera blog;  Thanks to Alexander Rubin of Percona Apache Hadoop is commonly used for data analysis. It is fast for data loads and scalable. In a previous post I showed how to  integrate MySQL with Hadoop . In this post I will show how to export a table from  MySQL to Hadoop, load the data to  Cloudera Impala  (columnar format), and run reporting on top of that. For the examples below, I will use the “ontime flight performance” data from my  previous post . I’ve used  Cloudera Manager  to install Hadoop and Impala. For this test I’ve (intentionally) used an old hardware (servers from 2006) to show that Hadoop can utilize the old hardware and still scale. The test cluster consists of 6 datanodes. Below are the specs: Purpose Server specs Namenode, Hive metastore, etc + Datanodes 2x PowerEdge 2950, 2x L5335 CPU @ 2.00GHz, 8 cores, 16GB RAM, RAID 10 with 8 SAS drives Datanodes only 4x PowerEdge SC1425, 2x Xeon ...

Very often I get the query `What is a cluster?` when discussing about Hadoop and Big Data. To keep it simple ` A cluster is a group or a network of machines wired together acting a single entity to work on a task which when run on a single machine takes much more longer time. ` The given task is split and processed by multiple machines in parallel and so that the task gets completed faster. Jesse Johnson puts it in simple and clear terms what a cluster is all about and how to design distributed algorithms here .                                                 In a Big Data cluster, the machines (or nodes) are neither as powerful as a server grade machine nor as dumb as a desktop machine. Having multiple (like in thousands) server grade ma...

Source from Cloudera Also,  Reintroducing Cloudera Enterprise, now with Apache Spark - See more at: cloudera spark Cloudera is announcing the general availability of support for Spark, bringing interactive machine learning and stream processing to enterprise data hubs. Cloudera is pleased to announce the immediate availability of its first release of  Apache Spark  for Cloudera Enterprise (comprising CDH and Cloudera Manager). Spark was created and contributed to the Apache Software Foundation by UC Berkeley, and it has quickly gained adoption for machine learning, interactive analytics, and streaming analytics over large datasets. It features a general programming model for writing applications by composing arbitrary operators, such as mappers, reducers, joins, group-bys, and filters. Spark keeps track of the data that each of the operators produces, enabling applications to reliably store this data in memory, which makes it ideal for low-latency computation...

Google has unique requirements with respective to data processing and storage which no one has. According to the WikiPedia , Google has to process about 24 Peta Bytes of data per day which be a bit outdated and Google might be processing more data per day. So, they need to continuously innovate to address the unique requirements. Soon they outgrow the innovation and they come up with some new innovation. The good thing is that Google had been continuously  releasing  these innovations  as papers once they have it refined and there is a solid internal implementation of it. These Google Papers have been implemented by the  ASF (Apache Software Foundation) and others. It's taking some time for the ASF frameworks like Hadoop and others to production ready. There is a catchup between Google papers and the ASF on a continuous basis. Google Paper Apache Frameworks The Google File System  (October, 2003) HDFS  (2008 became Apache TLP) MapReduce:...