SIGN UP HERE!
Video Available Now:
▶ PyDSLA The Nitty Gritty of Advanced Analytics Using Apache Spark in Python May 5, 2016 Talk by Miklos Christine, solutions engineer at Databricks
With so much hype about “big data” and the industry pushing for “big data” analytical tools for everyone, the question arises how many people have big data (for analytics) and how many of them really need these tools (which are more complex and often more immature compared to the traditional tools for analytics).
I currently have the need for machine learning tools that can deal with observations of the order of 10 millions in the context of binary classification. That kind of data is a few GBs in size and it fits comfortably nowadays in the RAM of a decent single machine. It is a trivial task for linear models, as there are plenty of open source tools that can train a logistic regression with this amount of data on a single machine in a few seconds, even while using only 1 processor core (many of these tools are single-threaded). Linear models are also the gold standard of large-scale machine learning that can run on clusters, processing very large distributed datasets.
I have been following Big Data, Hadoop in particular, for the past four years and a lot has changed since then. At the time I was looking for the next big wave of technologies, especially on the back end of development as that was my forte at the time. After our initial introduction, the more I learned the more I felt Hadoop was the right tool to get the job done. Focusing upon the development aspects of job vs actual maintenance of the servers/cluster has always been a no-brainer for me. Now that Hadoop has been adopted by a wide variety of industries and has grown quite popular, it has become much more apparent that there are indeed areas where Hadoop is lacking. Security and real-time analytics are two of these concerns which both affect real-time processing and reporting, but fortunately there is a lot of current work happening to address these limitations. Consequently, there is a group of big data vendors who are investing a large number of resources as a remedy, bringing us newer tools such as Yarn, Tez, Storm, etc. Also, acquisitions of XA Secure by Hortonworks and Gazzang by Cloudera are contributing to the solving these issues around security.
Today I am happy to announce that we are expanding our partnership on several fronts with my friend Subash D’Souza, a leader of the Big Data community in LA.
During the past few years I have been organizing several data science meetups in Los Angeles (LA R, DataVis LA, Data Science/Machine Learning) and Subash has provided the same kind of leadership on the big data front (LA Hadoop, HBase, LA Big Data). While I think of big data as primarily technology and of data science as a blend of statistics and technology (and not only for large data sets), there is clearly a significant overlap between these two fields. Considering this overlap, we have already been collaborating for some time regarding these various meetups, e.g. organizing joint events or speaking at one another’s meetups.
