Big Data Trendz

Source: Cloudera The following post was originally published by the Sentry community at apache.org. We re-publish it here for your convenience. We are very excited to announce that  Apache Sentry  has graduated out of Incubator and is now an Apache Top-Level Project! Sentry, which provides centralized fine-grained access control on metadata and data stored in Apache Hadoop clusters, was introduced as an Apache Incubator project back in August 2013. In the past two and a half years, the development community grew significantly to a large number of contributors from various organizations. Upon graduation, there were more than 50 contributors, 31 of whom had become committers. What’s Sentry? While Hadoop has strong security at the filesystem level, it lacked the granular support needed to adequately secure access to data by users and BI applications. This problem forces users to make a choice: either leave data unprotected or lock out users entirely. Most of the time, the...

                                                                                                                                 Source: Cloudera Blog The recently-released Apache Hive 2.0 contains some exciting improvements, many of which are already available in CDH 5.x. Recently, the Apache Hive community announced Hive 2.0.0. This is a larger release...

Via a combination of beta functionality in CDH 5.5 and new Cloudera Labs packages, you now have access to Apache HTrace for doing performance tracing of your HDFS-based applications. HTrace  is a new Apache incubator project that provides a bird’s-eye view of the performance of a distributed system. While log files can provide a peek into important events on a specific node, and metrics can answer questions about aggregate performance, HTrace can follow specific requests all the way through the cluster. HTrace breaks down requests into sets of  trace spans . Each trace span represents a length of time. A single request, such as an HDFS  copyToLocal  command, will generate many different trace spans. Each trace span has a list of parents that allow you to figure out why it was created and in which larger operation it is involved. Trace spans also have a “TracerId” that identifies which service and process they came from. Processes like the NameNode, Dat...

From Cloudera Blog, thanks to    Justin Kestelyn   for valuable post in cloudera blog. The super-active Apache Spark community is exerting a strong gravitational pull within the Apache Hadoop ecosystem. I recently had that opportunity to ask Cloudera’s Apache Spark committers (Sean Owen, Imran Rashid [PMC], Sandy Ryza, and Marcelo Vanzin) for their perspectives about how the Spark community has worked and is working together, and the work to be done via the  One Platform initiative  to make the Spark stack enterprise-ready. Recently, Apache Spark has become the most currently active project in the Apache Hadoop ecosystem (measured by number of contributors/commits over time), if not the entire ASF. Why do you think that is? Owen: Partly because of scope: Apache Spark has been many sub-projects under an umbrella from the start, some large and complex in their own right, and has tacked on several more in just the last six months. Culture is anothe...

From Clodera Blog: Thanks to  Yanpei Chen, Alan Choi, Dileep Kumar, David Rorke, Silvius Rus, and Devadutta Ghat Impala , the open source MPP query engine designed for high-concurrency SQL over Apache Hadoop, has seen tremendous  adoption  across enterprises in industries such as financial services, telecom, healthcare, retail, gaming, government, and advertising. Impala has unlocked the ability to use business intelligence (BI) applications on Hadoop; these applications support critical business needs such as data discovery, operational dashboards, and reporting. For example, one customer has proven that Impala scales to 80 queries/second, supporting 1,000+ web dashboard end-users with sub-second response time. Clearly, BI applications represent a good fit for Impala, and customers can support more users simply by enlarging their clusters. Cloudera’s previous testing already established that Impala is the  clear winner  among analytic SQL-on-Hadoop alternat...

Drink from Elephant’s Well Of Knowledge Developer success starts with open and reusable code, and a community that allows for both consumption of code and contribution of updates to the code base. This success engenders a thriving and evolving community. To that end, today we are announcing the  Hortonworks Gallery  for developers. Located on  GitHub , the Gallery brings together the Hortonworks’ Apache Hadoop code, Apache Ambari Views and extensions, as well as related resources into a single view for developers to use within the familiar context of Git and open source software. The Hortonworks Gallery brings together all of the code, tutorials and sample apps that help new and experienced Hadoop developers reduce the time to success, whether it’s learning about Hadoop, Spark, Storm or other HDP components, or delivering apps for Internet of Things, predictive analytics or data warehousing solutions. Additionally, over the coming weeks we will be moving the so...

Thanks to Dhruv Kumar, as he introduces Cascading, an open source application development framework that allows Java developers to build applications on top of Hadoop through its Java API. Now, Currently Dhruv Kumar is at partner solution, Hortonworks .