Big Data Trendz

Thanks to Carter Shanklin & Nita Dembla from Hortonworks for valuable post. One of the most exciting new features of HDP 2.6 from Hortonworks was the general availability of Apache Hive with LLAP. If you missed DataWorks Summit you’ll want to look at some of the great LLAP experiences our users shared, including Geisinger who found that  Hive LLAP outperforms their traditional EDW  for most of their queries, and Comcast who found  Hive LLAP is faster than Presto for 75% of benchmark queries . These great results are thanks to performance and stability improvements Hortonworks made to Hive LLAP resulting in 3x faster interactive query in HDP 2.6. This blog dives into the reasons HDP 2.6 is so much faster. We’ll also take a look at the massive step forward Hive has made in SQL compliance with HDP 2.6, enabling Hive to run all 99 TPC-DS queries with only trivial modifications to the original source queries. STARTING OFF: 3X PERFORMANCE GAINS IN HDP 2.6 WITH HIVE ...

Thanks to Hortonworks blog resource --  Carter Shanklin && Nita Dembla Apache Hive(™)  is the most complete SQL on Hadoop system, supporting comprehensive SQL, a sophisticated cost-based optimizer, ACID transactions and fine-grained dynamic security. Though Hive has proven itself on multi-petabyte datasets spanning thousands of nodes many interesting use cases demand more interactive performance on smaller datasets, requiring a shift to in-memory. Hive 2 marks the beginning of Hive’s journey from a disk-centric architecture to a memory-centric architecture through  Hive LLAP (Live Long and Process). Since memory costs about 100x as much as disk, memory-centric architectures demand a careful design that makes the most of available resources. In this blog, we’ll update benchmark results from our earlier blog, “ Announcing Apache Hive 2.1: 25x Faster Queries and Much More. ” NOT ALL MEMORY ARCHITECTURES ARE EQUAL

                                                                                                                                 Source: Cloudera Blog The recently-released Apache Hive 2.0 contains some exciting improvements, many of which are already available in CDH 5.x. Recently, the Apache Hive community announced Hive 2.0.0. This is a larger release...

Apache Hive 1.2.0, although not a major release, contains significant improvements. Recently, the Apache Hive community moved to a more frequent, incremental release schedule. So, a little while ago, we  covered the Apache Hive 1.0.0 release  and explained how it was renamed from 0.14.1 with only minor feature additions since 0.14.0. Shortly thereafter,  Apache Hive 1.1.0  was released (renamed from Apache Hive 0.15.0), which included more significant features—including  Hive-on-Spark . Last week, the community released  Apache Hive 1.2.0 . Although a more narrow release than Hive 1.1.0, it nevertheless contains improvements in the following areas: New Functionality Support for Apache Spark 1.3 ( HIVE-9726 ), enabling dynamic executor allocation and impersonation Support for integration of Hive-on-Spark with Apache HBase ( HIVE-10073 ) Support for numeric partition columns with literals ( HIVE-10313 ,  HIVE-10307 ) Support for Union Dist...

Pre-requisites Ubuntu / CentOS Hadoop 1.x/ 2.x , I prefer to install with 2.x Step –> 1: Download and Install Download the Hive from the Apache Download Mirror and i place it in /home/bigdata/Installations/ d irectory. $ cd /home/bigdata/Installation $ wget http://redrockdigimark.com/apachemirror/hive/stable/apache-hive-1.2.1-bin.tar.gz  ( i preferred to download hive-1.2.1 .tar.gz, as it is stable version) $ sudo tar xzf hive-1.2.1.tar.gz Step –> 2: After downloading and installation. Now we are moving to edit hive-env.sh file for Configuration. To configure hive, there I have installed and give permission to bigdata. In $HIVE_HOME/conf/hive-env.sh export JAVA_HOME=/opt/jdk1.80_10  Step –> 3: add hbase path to bashrc $ gedit .bashrc and add following lines to it #HIVE export HIVE_HOME=/home/bigdata/Installations/hive-1.2.1/ export PATH=$PATH:$HIVE_HOME/bin Step –> 4: Restart the terminal and start hadoop, th...

Apache Hive is a framework that sits on top of Hadoop for doing ad-hoc queries on data in Hadoop. Hive supports HiveQL which is similar to SQL, but doesn't support the complete constructs of SQL. Hive coverts the HiveQL query into Java MapReduce program and then submits it to the Hadoop cluster. The same outcome can be achieved using HiveQL and Java MapReduce, but using Java  MapReduce will required a lot of code to be written/debugged compared to HiveQL. So, it increases the developer productivity to use Hive. To summarize, Hive through HiveQL language provides a higher level abstraction over Java MapReduce programming. As with any other high level abstraction, there is a bit of performance overhead using HiveQL when compared to Java MapReduce. But the Hive community is working to narrow down this gap for most of the commonly used scenarios. Along the same line Pig provides a higher level abstraction over MapReduce. Pig supports PigLatin constructs, which is ...