The case for Apache Spark

Tue, 23 Sep 2014 10:00:05 -0700

I have been following Big Data, Hadoop in particular, for the past four years and a lot has changed since then. At the time I was looking for the next big wave of technologies, especially on the back end of development as that was my forte at the time. After our initial introduction, the more I learned the more I felt Hadoop was the right tool to get the job done. Focusing upon the development aspects of job vs actual maintenance of the servers/cluster has always been a no-brainer for me. Now that Hadoop has been adopted by a wide variety of industries and has grown quite popular, it has become much more apparent that there are indeed areas where Hadoop is lacking. Security and real-time analytics are two of these concerns which both affect real-time processing and reporting, but fortunately there is a lot of current work happening to address these limitations. Consequently, there is a group of big data vendors who are investing a large number of resources as a remedy, bringing us newer tools such as Yarn, Tez, Storm, etc. Also, acquisitions of XA Secure by Hortonworks and Gazzang by Cloudera are contributing to the solving these issues around security.

Subash D'Souza on DataScience.LA

The case for Apache Spark