[{"content":"","date":"July 1, 2021","externalUrl":null,"permalink":"/authors/","section":"Authors","summary":"","title":"Authors","type":"authors"},{"content":"From 2014 to 2021 DataScience.LA was the blog of the Los Angeles data science community: summaries and videos of LA R, PyData, DataVis and Machine Learning meetup talks, interviews from the useR! 2014 conference at UCLA, benchmarks, and essays. This is a read-only archive of the original site. Comments are preserved but closed.\n","date":"July 1, 2021","externalUrl":null,"permalink":"/","section":"Data Science LA","summary":"From 2014 to 2021 DataScience.LA was the blog of the Los Angeles data science community: summaries and videos of LA R, PyData, DataVis and Machine Learning meetup talks, interviews from the useR! 2014 conference at UCLA, benchmarks, and essays. This is a read-only archive of the original site. Comments are preserved but closed.\n","title":"Data Science LA","type":"page"},{"content":"I (Szilard) started organizing meetups more than a decade ago (the LA R Users Group est. 2009 was the very first data meetup in the Los Angeles area). A few years later, a couple of us (Amelia, Leigh, Eduardo and I, and also several contributors) started a few more meetup groups (e.g. Data Science) and this very website DataScience.LA. However, in recent years my DataScience.LA Leadership Teammates relocated and now it\u0026rsquo;s my turn, I\u0026rsquo;m the last one to leave California (I\u0026rsquo;m very excited to move to Texas!). While we are ceasing all activities in LA now and this is the last update/post on this website, we will keep the existing/old content accessible for a while. It was a great pleasure running these events/this site and getting to know so many of you. LA has so many great data science folks! However, now it\u0026rsquo;s time to say goodbye, goodbye LA and California, and good luck to you all.\n","date":"July 1, 2021","externalUrl":null,"permalink":"/goodbye-datascience-la-goodbye-california/","section":"Posts","summary":"I (Szilard) started organizing meetups more than a decade ago (the LA R Users Group est. 2009 was the very first data meetup in the Los Angeles area). A few years later, a couple of us (Amelia, Leigh, Eduardo and I, and also several contributors) started a few more meetup groups (e.g. Data Science) and this very website DataScience.LA. However, in recent years my DataScience.LA Leadership Teammates relocated and now it’s my turn, I’m the last one to leave California (I’m very excited to move to Texas!). While we are ceasing all activities in LA now and this is the last update/post on this website, we will keep the existing/old content accessible for a while. It was a great pleasure running these events/this site and getting to know so many of you. LA has so many great data science folks! However, now it’s time to say goodbye, goodbye LA and California, and good luck to you all.\n","title":"Goodbye DataScience.LA, goodbye California!","type":"posts"},{"content":"","date":"July 1, 2021","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"July 1, 2021","externalUrl":null,"permalink":"/authors/szilard/","section":"Authors","summary":"","title":"Szilard Pafka","type":"authors"},{"content":"Update: Video Recording:\n▶ LA Data Science Meetup, June 8, 2021 - Michal Kurka: H2O\u0026#39;s GBM Slides: here\nAfter our last 3 meetups with core developers of XGBoost, LightGBM and Catboost, respectively, it is now H2O\u0026rsquo;s turn (with H2O\u0026rsquo;s Director of Engineering speaking)! Just as the last few times, we\u0026rsquo;ll fit in a 1-hour slot, talk 35 minutes + Q\u0026amp;A 20 minutes (10:00-10:55am Pacific Time).\nOverview of H2O GBM implementation\nby Michal Kurka\nH2O started its mission to let data scientists train their models on datasets of any size in 2011 with just a few machine learning algorithms, Gradient Boosting Machine (GBM) being one of them. To this day H2O GBM is one of our most popular and full featured algorithms in what grew out to be a well-rounded and diverse machine learning framework. In this talk we will briefly introduce H2O’s ML framework and discuss how it differs from its competitors. Then we will dive into the internals and discover how GBM implementation leverages H2O\u0026rsquo;s MapReduce framework to train models fast in both single-node and distributed environments. Furthermore, we will take a look at the ecosystem that we built to support data scientists to inspect, debug and learn from their models (tree visualization, Shapley, feature interactions, …). We will touch on some of the more nuanced features of H2O GBM - applying monotonic constraints and implementing a custom loss function. We will explore options H2O provides for deploying models in production. Finally, we will take a look at our roadmap and discuss how users can contribute to make H2O GBM better.\nSpeaker Bio:\nMichal holds a Masters degree in Mathematical Optimization. For the last 10 years he was involved in implementing scalable ML algos using frameworks like Hadoop MapReduce, Spark and H2O. As a Director of Engineering in H2O.ai he is responsible for development of H2O\u0026rsquo;s open source machine learning framework H2O-3. He is passionate about performance of distributed systems and he contributed to performance improvements of the overall platform as well as GBM specifically. He implemented a variety of GBM features including TreeSHAP and monotonicity constraints. He also maintains H2O’s implementation of PSVM, CoxPH, word2vec and was responsible for H2O\u0026rsquo;s XGBoost integration.\nDate/Time: Tuesday, June 8, 10:00-10:55am Pacific Time\n**Venue:**online (zoom)\nRSVP: here on meetup\n**Note:**The zoom link will be posted in comments on meetup (link above) at 9:55am and due to our zoom’s 100-attendee limit, the first 100 people will be able to join the zoom call.\n","date":"May 10, 2021","externalUrl":null,"permalink":"/overview-of-h2os-gbm-implementation/","section":"Posts","summary":"Update: Video Recording:\n▶ LA Data Science Meetup, June 8, 2021 - Michal Kurka: H2O's GBM Slides: here\n","title":"Overview of H2O's GBM implementation","type":"posts"},{"content":"Update: Video Recording:\n▶ LA Data Science Meetup, March 30, 2021 - Stanislav Kirillov: CatBoost Slides: here\nAfter our last 2 meetups with core developers of XGBoost and LightGBM, respectively, it is now CatBoost\u0026rsquo;s turn (with the head of the CatBoost dev team speaking)! Just as last time, we\u0026rsquo;ll fit in a 1-hour slot, talk 35 minutes + Q\u0026amp;A 20 minutes (10:00-10:55am Pacific Time).\nThe zoom link will be posted in comments below at 9:55am and due to our zoom\u0026rsquo;s 100-attendee limit, the first 100 people will be able to join the zoom call.\nCatBoost: Distributed Training, Uncertainty Estimation and Other News\nby Stanislav Kirillov\nCatBoost is a popular open-source library for training gradient boosting models, with built-in categorical, text, and embedding features support.\nIn this talk, we will discuss major updates and recall the main features of CatBoost, including:\nCatBoost for Spark release Object embeddings and text features support Uncertainty estimation GPU training support Dataset prequantization support Fast inference (both CPU and GPU) We will show a brief demo of CatBoost PySpark training and present plans for CatBoost development.\nSpeaker Bio:\nStanislav Kirillov is the head of CatBoost development team at Yandex. He develops machine learning tools, supporting and developing infrastructure for them. Stanislav is a big fan of distributed training and low-level software optimizations.\nDate/Time: Tuesday, March 30, 10:00-10:55am Pacific Time\n**Venue:**online (zoom)\nRSVP: here on meetup\n**Note:**The zoom link will be posted in comments on meetup (link above) at 9:55am and due to our zoom’s 100-attendee limit, the first 100 people will be able to join the zoom call.\n","date":"March 10, 2021","externalUrl":null,"permalink":"/catboost-distributed-training-uncertainty-estimation-and-other-news/","section":"Posts","summary":"Update: Video Recording:\n▶ LA Data Science Meetup, March 30, 2021 - Stanislav Kirillov: CatBoost Slides: here\n","title":"CatBoost: Distributed Training, Uncertainty Estimation and Other News","type":"posts"},{"content":"","date":"March 10, 2021","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"March 10, 2021","externalUrl":null,"permalink":"/category/machine-learning-data-science/","section":"Categories","summary":"","title":"Machine Learning / Data Science","type":"categories"},{"content":"","date":"March 10, 2021","externalUrl":null,"permalink":"/category/meetups/","section":"Categories","summary":"","title":"Meetups","type":"categories"},{"content":"Update: Video recording:\n▶ LA Data Science Meetup, Jan 12, 2021 - James Lamb: Recent Developments in LightGBM Slides and code: here.\nAfter our last meetup with an overview and benchmark of the most popular GBM implementations and then a separate talk on XGBoost, we are back with a talk about LightGBM this time, again right from one of its core developers and maintainers. This time we\u0026rsquo;ll fit in a 1-hour slot, talk 35 minutes + Q\u0026amp;A 20 minutes (10:00-10:55am Pacific Time).\nRecent Developments in LightGBM\nby James Lamb\nIn this talk, attendees will learn about recent developments in LightGBM, a popular open source gradient boosting library. The talk\u0026rsquo;s primary goal is to educate attendees about recent work that has been done to make LightGBM easier to install and use effectively.\nThe talk will describe in detail a few recent additions to the library, including:\nCUDA-based GPU acceleration new options to make installation of a GPU-enabled Python package easier availability of the R package on CRAN new detailed documentation on hyperparameter tuning Dask integration for distributed training on large datasets Demos of the new (experimental) Dask integration will be shown. The talk will conclude with a summary of new features that are coming in the future, and a call for attendees who are interested to contribute their ideas, documentation, and code to LightGBM.\nSpeaker Bio:\nJames Lamb is a software engineer at Saturn Cloud, where he works on a managed data science platform built on Dask. Before Saturn Cloud, James worked on industrial internet of things (IIoT) problems as a data scientist at AWS and Chicago-based Uptake. He is a core maintainer on LightGBM, and has contributed on other open source data science and data engineering projects such as XGBoost and prefect. James holds Masters degrees in Applied Economics (Marquette University) and Data Science (University of California, Berkeley).\nDate/Time: Tuesday, January 12, 10:00-10:55am Pacific Time\nVenue: online (zoom)\nRSVP: here on meetup\nNote: The zoom link will be posted in comments on meetup (link above) at 9:55am and due to our zoom\u0026rsquo;s 100-attendee limit, the first 100 people will be able to join the zoom call.\nDon\u0026rsquo;t miss the chance to interact with a core developer of one of the most popular GBM libraries.\n","date":"December 15, 2020","externalUrl":null,"permalink":"/recent-developments-in-lightgbm/","section":"Posts","summary":"Update: Video recording:\n▶ LA Data Science Meetup, Jan 12, 2021 - James Lamb: Recent Developments in LightGBM Slides and code: here.\n","title":"Recent Developments in LightGBM","type":"posts"},{"content":"Update: Video recordings:\nTalk #1 (Szilard Pafka: GBM Benchmarks)\n▶ LA Data Science Meetup, Nov. 10, 2020 - Talk #1 - Szilard Pafka: GBM Benchmarks Slides here\nTalk #2 (Hyunsu Cho: The State of XGBoost)\n▶ LA Data Science Meetup, Nov. 10, 2020 - Talk #2 - Hyunsu Cho: The State of XGBoost Slides: here\nIn this first ONLINE event we\u0026rsquo;ll have 2 fantastic talks from 2 gradient boosting experts/open source contributors. Szilard Pafka is the author of benchm-ml and GBM-perf benchmarks and in this talk he will go in-depth like never before. Hyunsu Cho is the lead maintainer of XGBoost and he will give us a firsthand on the project\u0026rsquo;s history, major milestones and its roadmap for the future.\nAnd the Winner Is\u0026hellip;: Insights from a Gradient Boosting (GBM) Benchmark\nSzilard Pafka, PhD\nChief Scientist, Epoch\nWith all the hype about deep learning and \u0026ldquo;AI\u0026rdquo;, it is not well publicized that for structured/tabular data widely encountered in business applications it is actually another machine learning algorithm, the gradient boosting machine (GBM) that most often achieves the highest accuracy in supervised learning/prediction tasks. In this talk we\u0026rsquo;ll review some of the main open source GBM implementations such as xgboost, h2o, lightgbm, catboost, Spark MLlib and we\u0026rsquo;ll discuss some of their main performance characteristics. We\u0026rsquo;ll go more in-depth vs all my previous talks on the topic, and we\u0026rsquo;ll discuss in details training speed, memory footprint, scaling to multiple CPU cores, performance degradation on hyperthreaded cores and multi-socket CPUs, performance on latest Intel and AMD CPUs, GPU implementations, GPU utilization patterns etc. and also several 2020 recent updates such as improved multi-core performance in xgboost and speedups in catboost.\nSzilard studied Physics in the 90s and obtained a PhD by using statistical methods to analyze the risk of financial portfolios. He worked in finance, then more than a decade ago moved to become the Chief Scientist of a tech company in Santa Monica, California doing \u0026ldquo;everything data\u0026rdquo;. He is the founder/organizer of several meetups in the Los Angeles area (R, data science etc) and the data science community website datascience.la. He is the author of a well-known machine learning benchmark on github (1000+ stars), a frequent speaker at conferences (keynote/invited at KDD, R-finance, Crunch, eRum etc.), and he has developed and taught graduate data science courses at two universities (UCLA and CEU in Europe).\nThe State of XGBoost: history and community overview\nHyunsu Cho\nSenior Systems Software Engineer, NVIDIA\nWhat used to be a research prototype to process several GBs of data on a single workstation, XGBoost has now grown to be a production-quality software that can process hundreds of GBs of data in a cluster. In the last few years, XGBoost has added multiple major features, such as support for NVIDIA GPUs as a hardware accelerator and distributed computing platforms including Apache Spark and Dask. This talk will provide a short tour of the history of XGBoost, recognizing its major milestones. We will particularly pay attention to how XGBoost has integrated with major data science packages and frameworks, such as scikit-learn, R, Apache Spark, Dask, and RAPIDS AI. We will also share current efforts to grow the community, to put the open source project on a sustainable path. Finally, we will share the future roadmap of the project and list of wish items.\nAfter receiving Masters degree from University of Washington, Hyunsu went to join Amazon Web Services (AWS) to develop AI as a service. In April 2020, Hyunsu joined the RAPIDS AI team in NVIDIA. He is now focusing on improving end-to-end data science pipelines using NVIDIA GPUs. Hyunsu has been in charge of maintaining the XGBoost project since November 2017. In addition to triaging bug reports and reviewing pull requests, Hyunsu also maintains the test farm for continuous integration (CI), which ensures that all incoming pull requests meet the quality bar.\nDate/Time: November 10, 2020, 10am Pacific\nLocation: ONLINE (zoom)\nRSVP and zoom link: https://www.meetup.com/Los-Angeles-Machine-Learning-Data-Science/events/273958512/\nNOTE: The event will be limited to the first 100 attendees joining the zoom meeting.\n","date":"October 15, 2020","externalUrl":null,"permalink":"/2-talks-gbm-benchmark-in-depth-the-state-of-xgboost/","section":"Posts","summary":"Update: Video recordings:\nTalk #1 (Szilard Pafka: GBM Benchmarks)\n▶ LA Data Science Meetup, Nov. 10, 2020 - Talk #1 - Szilard Pafka: GBM Benchmarks Slides here\n","title":"2 talks: GBM Benchmark In-Depth | The State of XGBoost","type":"posts"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/authors/amy-chen/","section":"Authors","summary":"","title":"Amy Tzu-Yu Chen","type":"authors"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tag/data-science-2/","section":"Tags","summary":"","title":"data science","type":"tags"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/category/education/","section":"Categories","summary":"","title":"Education","type":"categories"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tag/machine-learning/","section":"Tags","summary":"","title":"machine learning","type":"tags"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tag/meetups-2/","section":"Tags","summary":"","title":"meetups","type":"tags"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/category/r/","section":"Categories","summary":"","title":"R","type":"categories"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tag/r/","section":"Tags","summary":"","title":"R","type":"tags"},{"content":"Come and have fun with local useRs at SatRday LA on April 6, 2019. This is the first SatRday in LA, and the second in the states. If you have not heard of SatRday, it is a one-day affordable, inclusive, non-profit R conference organized by local R users.\nImportant Details # Who: All levels of R users\nWhen: April 6, 2019\nWhere: UCLA James West Alumni Center\nPrice:$35 for professionals, $15 for students\nCall for Presentations: now ~ Feb 10, 2019\nHow Can I participate? # SatRday LA is accepting submissions (30/45min talk, poster, and 5min lightning talk) until Feb 10. This is a good chance to share and connect with LA R community. You can find submission guidelines or buy a ticket at losangeles2019.satrdays.org/\nTo sponsor, follow, or connect with SatRday LA # @satRdays_org on Twitter\nsatrdays.losangeles@gmail.com\n","date":"January 28, 2019","externalUrl":null,"permalink":"/satrday-la-r-conference-announcement/","section":"Posts","summary":"Come and have fun with local useRs at SatRday LA on April 6, 2019. This is the first SatRday in LA, and the second in the states. If you have not heard of SatRday, it is a one-day affordable, inclusive, non-profit R conference organized by local R users.\n","title":"SatRday LA - R Conference Announcement","type":"posts"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"January 28, 2019","externalUrl":null,"permalink":"/tag/ucla/","section":"Tags","summary":"","title":"UCLA","type":"tags"},{"content":"","date":"July 20, 2018","externalUrl":null,"permalink":"/tag/gradient-boosting/","section":"Tags","summary":"","title":"gradient boosting","type":"tags"},{"content":"Update_1: Slides for the 4 talks added below (at each talk section separately).\nUpdate_2: Photos from the meetup and a short description of the event added below (at the bottom of the post).\nUpdate_3: Video recording of the meetup here:\nThis video is no longer available on YouTube (original link).\nIn this event we had a main talk (30 mins) and 3 excellent lightning talks about gradient boosting machines (GBMs). GBM is the machine learning algorithm that usually achieves the best accuracy on structured/tabular data beating other algorithms such as the much hyped deep neural networks (deep learning).\nBetter than Deep Learning: Gradient Boosting Machines (GBM)\nby Szilard Pafka, PhD\nChief Scientist, Epoch\nAbstract: With all the hype about deep learning and \u0026ldquo;AI\u0026rdquo;, it is not well publicized that for structured/tabular data widely encountered in business applications it is actually another machine learning algorithm, the gradient boosting machine (GBM) that most often achieves the highest accuracy in supervised learning tasks. In this talk we\u0026rsquo;ll review some of the main GBM implementations available as R and Python packages such as xgboost, h2o, lightgbm etc, we\u0026rsquo;ll discuss some of their main features and characteristics, and we\u0026rsquo;ll see how tuning GBMs and creating ensembles of the best models can achieve the best prediction accuracy for many business problems.\nSlides here.\nBio: Szilard studied Physics in the 90s and obtained a PhD by using statistical methods to analyze the risk of financial portfolios. He worked in finance, then more than a decade ago moved to become the Chief Scientist of a tech company in Santa Monica doing everything data (analysis, modeling, data visualization, machine learning, data infrastructure etc). He is the founder/organizer of several meetups in the Los Angeles area (R, data science etc) and the data science community website datascience.la. He is the author of a well-known machine learning benchmark on github (1000+ stars), a frequent speaker at conferences (keynote/invited at KDD, R-finance, Crunch, eRum and contributed at useR!, PAW, EARL etc.), and he has developed and taught graduate data science and machine learning courses as a visiting professor at two universities (UCLA and CEU in Europe).\nLighting talks:\n1. Not your father’s objective function - weird things you can do with xgboost\nby Peter Foley\nVice President, Analytics at 605\nXGBoost makes it easy to predict a variety of different outcome types \u0026ndash; binary, continuous, ranked, categorical, count, and also custom objective functions for specialized needs. Those custom objective functions let you do some even weirder stuff that you might not expect. I’ll give examples of hacking custom objectives to fit vector-valued outcomes and local linear regressions, and give tips on how to plug your own weird functions into xgboost to take advantage of it’s powerful and fast tree construction.\nBio: Peter leads the analytics and data science team, and manages 605’s behavioral modeling and experimentation. 605 offers unique, independent television audience measurement and analytics to build better marketing and programming initiatives within the media and entertainment industries. Most of Peter’s work is management, but he maintains involvement in the group’s experimental method development, machine learning techniques, and scalable modeling infrastructure.\nSlides here.\n2. GBM model interpretability\nby Michael Tiernay\nSenior Data Scientist at Netflix\nSlides here.\n3. Why lightGBM become my 1st choice in Kaggle?\nby Hang Li\nHulu\nXgboost has become the most popular algorithm in Kaggle competitions 3 or 4 years ago. Recently, another open source GBM implementation (lightGBM) was introduced to the Kaggle community by Microsoft. Due to its good performance, it has become my 1st choice for Kaggles. In this talk will briefly introduce some of the nice features of lightGBM.\nBio: Hang is a Competition Master at Kaggle. He participated in over 30 Data Science Competitions. He has a strong passion for using machine learning techniques to solve real-world problems.\nSlides here.\nThis meetup was pretty wild. The meetup sold out in 56 minutes from announcement. Another 200+ people signed up for the waitlist. Then, my prediction about the show-up rate was way off (isn\u0026rsquo;t that ironic for a machine learning meetup?), so the room reached the max capacity just by too many people with RSVPs showing up (never had that before, usually about 50-60% show up). Thankfully, our hosts at Hulu figured it out and set up an overflow room and streamed the show to it. The rest of the event was smooth, lots of people, great talks, lots of good questions and discussions. It was a great event after all! Thanks again Hulu for hosting us. An special thanks to Carl Mullins for recording the meetup and making the video available (embedded at the top of this post) to all of you who could not attend (either being on the waiting list or not living in LA).\n","date":"July 20, 2018","externalUrl":null,"permalink":"/meetup-machine-learning-best-algorithms-gradient-boosting-machines-gbm/","section":"Posts","summary":"Update_1: Slides for the 4 talks added below (at each talk section separately).\nUpdate_2: Photos from the meetup and a short description of the event added below (at the bottom of the post).\nUpdate_3: Video recording of the meetup here:\n","title":"Meetup: Machine Learning Best Algorithms: Gradient Boosting Machines (GBM)","type":"posts"},{"content":"","date":"October 28, 2017","externalUrl":null,"permalink":"/tag/hadley/","section":"Tags","summary":"","title":"hadley","type":"tags"},{"content":"Update: Video of the 2nd half of the talk is here (1st half was not recorded, sorry):\nThis video is no longer available on YouTube (original link).\nIn our next meetup, we\u0026rsquo;ll have Hadley (I think no introduction needed).\nPackages made easy\nby Hadley Wickham\nIn this talk I\u0026rsquo;ll show you how easy it to make an R package by live coding one in front of you! You\u0026rsquo;ll learn about three useful packages:\nusethis: automating the setup of key package components devtools: for automating many parts of the package development workflow testthat: to write unit tests that make you confident your code is correct Timeline:\n6:30pm arrival, drinks and networking 7:00pm talk You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP here on Eventbrite.\nVenue: Edmunds, 2401 Colorado Avenue (this is Edmunds\u0026rsquo; NEW LOCATION, don\u0026rsquo;t go to the old one)\nPark underneath the building (Colorado Center), Edmunds will validate.\n","date":"October 28, 2017","externalUrl":null,"permalink":"/r-meetup-with-hadley-wickham/","section":"Posts","summary":"Update: Video of the 2nd half of the talk is here (1st half was not recorded, sorry):\nThis video is no longer available on YouTube (original link).\nIn our next meetup, we’ll have Hadley (I think no introduction needed).\nPackages made easy\nby Hadley Wickham\n","title":"R Meetup with Hadley Wickham","type":"posts"},{"content":"Update: Video recording of the meetup here:\n▶ Machine Learning in Production with Szilard Pafka Slides here:\nOpening the Black Box: Attempts to Understand the Results of Machine Learning Models - Michael Tiernay - LA Data Science Meetup - May 2017 · Speaker Deck Machine Learning in Production - LA Data Science/Machine Learning Meetup - May 2017 · Speaker Deck Machine Learning in Production\nby Szilard Pafka\nIn this talk I will discuss the main steps involved in having machine learning models in production. No hype (\u0026ldquo;artificial intelligence\u0026rdquo;), no bullshitting and no marketing of expensive products (the best tools are all open source and free, you just need to know what you are doing).\nBio: Szilard has trained, deployed and maintained machine learning models in production (starting with random forests) since 2007. He\u0026rsquo;s a Chief Scientist and a Physics PhD with 20+ years experience with data, computing and modeling. He founded the first data meetup in LA in 2009 (by chance) and he has organized several data science events since.\nWe\u0026rsquo;ll also have a lightning talk (10-mins) before the main talk:\nOpening the black box: Attempts to understand the results of machine learning models\nby Michael Tiernay, Data Scientist at Edmunds\nTimeline:\n6:30pm arrival, food/drinks and networking 7:30pm talks 9 pm more networking You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\n**Venue:**Edmunds, 2401 Colorado Avenue (this is Edmunds\u0026rsquo; NEW LOCATION, don\u0026rsquo;t go to the old one)\nPark underneath the building (Colorado Center), Edmunds will validate.\n","date":"April 28, 2017","externalUrl":null,"permalink":"/meetup-machine-learning-in-production-with-szilard-pafka/","section":"Posts","summary":"Update: Video recording of the meetup here:\n▶ Machine Learning in Production with Szilard Pafka Slides here:\n","title":"Meetup: Machine Learning in Production with Szilard Pafka","type":"posts"},{"content":"","date":"April 28, 2017","externalUrl":null,"permalink":"/tag/python/","section":"Tags","summary":"","title":"python","type":"tags"},{"content":"","date":"April 28, 2017","externalUrl":null,"permalink":"/tag/statistics/","section":"Tags","summary":"","title":"statistics","type":"tags"},{"content":"","date":"March 8, 2017","externalUrl":null,"permalink":"/tag/big-data/","section":"Tags","summary":"","title":"big data","type":"tags"},{"content":"Update: Slides click here.\nVideo recording here:\nThis video is no longer available on YouTube (original link).\nStream processing with R in AWS\nby Gergely Daróczi\nAbstract: R is rarely mentioned among the big data tools, although it\u0026rsquo;s fairly well scalable for most data science problems and ETL tasks. This talk presents an open-source R package to interact with Amazon Kinesis via the MultiLangDaemon bundled with the Amazon KCL to start multiple R sessions on a machine or cluster of nodes to process data from theoretically any number of Kinesis shards. Besides the technical background and a quick introduction on how Kinesis works, this talk will feature some stream processing use-cases at CARD.com, and will also provide an overview and hands-on demos on the related data infrastructure built on the top of Docker, Amazon ECS, ECR, KMS, Redshift and a bunch of third-party APIs.\nBio: Gergely Daróczi is an enthusiast R user and package developer, founder of an R-based web application at rapporter.net, Ph.D. candidate in Sociology, Director of Analytics at CARD.com with a strong interest in designing a scalable data platform built on the top of R, AWS and a dozen APIs. He maintains some CRAN packages mainly dealing with reporting and API integrations, co-authored a number of journal articles in social and medical sciences, and recently published the \u0026ldquo;Mastering Data Analysis with R\u0026rdquo; book at Packt.\nTimeline:\n6:30pm arrival, food/drinks and networking 7:30pm talks 9 pm more networking You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\nVenue: Edmunds, 2401 Colorado Avenue (this is Edmunds\u0026rsquo; NEW LOCATION, don\u0026rsquo;t go to the old one)\nPark underneath the building (Colorado Center), Edmunds will validate.\n","date":"March 8, 2017","externalUrl":null,"permalink":"/meetup-stream-processing-with-r-in-aws/","section":"Posts","summary":"Update: Slides click here.\nVideo recording here:\nThis video is no longer available on YouTube (original link).\nStream processing with R in AWS\nby Gergely Daróczi\nAbstract: R is rarely mentioned among the big data tools, although it’s fairly well scalable for most data science problems and ETL tasks. This talk presents an open-source R package to interact with Amazon Kinesis via the MultiLangDaemon bundled with the Amazon KCL to start multiple R sessions on a machine or cluster of nodes to process data from theoretically any number of Kinesis shards. Besides the technical background and a quick introduction on how Kinesis works, this talk will feature some stream processing use-cases at CARD.com, and will also provide an overview and hands-on demos on the related data infrastructure built on the top of Docker, Amazon ECS, ECR, KMS, Redshift and a bunch of third-party APIs.\n","title":"Meetup: Stream processing with R in AWS","type":"posts"},{"content":"Update: Slides here:\nSlides: https://speakerdeck.com/szilard/machine-learning-meetup-febr-2017\nOur next meetup:\nNo-Bullshit Data Science\nby Szilard Pafka\nAbstract: While extracting business value from data has been performed by practitioners for decades, the last several years have seen an unprecedented amount of hype in this field. This hype has created not only unrealistic expectations in results, but also glamour in the usage of the newest tools assumably capable of extraordinary feats. In this talk I will apply the much needed methods of critical thinking and quantitative measurements (that data scientists are supposed to use daily in solving problems for their companies) to assess the capabilities of the most widely used software tools for data science. I will discuss in details two such analyses, one concerning the size of datasets used for analytics and the other one regarding the performance of machine learning software used for supervised learning.\nBio: Szilard studied Physics in the 90s and has obtained a PhD by using statistical methods to investigate the risk of financial portfolios. Next he has worked in a bank quantifying and managing market risk. About a decade ago he moved to Santa Monica, California to become the Chief Scientist of a credit card processing company doing everything data (ETL, analysis, modeling, visualization, machine learning etc). He is also the founder/organizer of several data science related meetups in Santa Monica (LA R, Data Science/Machine Learning, DataVis LA etc.) and the data community website datascience.la. LinkedIn: szilard Twitter: @DataScienceLA Github: @szilard\nTimeline:\n6:30pm arrival, food/drinks and networking 7:30pm talk starts 8:30 pm more networking You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\nVenue: Headspace, 2415 Michigan Avenue, Santa Monica. Thanks for hosting and providing food/drinks!\n","date":"February 17, 2017","externalUrl":null,"permalink":"/meetup-no-bullshit-data-science/","section":"Posts","summary":"Update: Slides here:\nSlides: https://speakerdeck.com/szilard/machine-learning-meetup-febr-2017\nOur next meetup:\nNo-Bullshit Data Science\nby Szilard Pafka\nAbstract: While extracting business value from data has been performed by practitioners for decades, the last several years have seen an unprecedented amount of hype in this field. This hype has created not only unrealistic expectations in results, but also glamour in the usage of the newest tools assumably capable of extraordinary feats. In this talk I will apply the much needed methods of critical thinking and quantitative measurements (that data scientists are supposed to use daily in solving problems for their companies) to assess the capabilities of the most widely used software tools for data science. I will discuss in details two such analyses, one concerning the size of datasets used for analytics and the other one regarding the performance of machine learning software used for supervised learning.\n","title":"Meetup: No-Bullshit Data Science","type":"posts"},{"content":"Contributor\n","date":"January 19, 2017","externalUrl":null,"permalink":"/authors/eduardo/","section":"Authors","summary":"Contributor\n","title":"Eduardo Ariño de la Rubia","type":"authors"},{"content":"","date":"January 19, 2017","externalUrl":null,"permalink":"/category/pydsla/","section":"Categories","summary":"","title":"PyDSLA","type":"categories"},{"content":"Update: Video recording of the meet up here:\nThis video is no longer available on YouTube (original link).\nCome to the PyDSLA January Meetup to hear talks about PyMC3 and jobs.py!\nREGISTER HERE\nFor this event we are incredibly fortunate to have two incredibly speakers give really interesting talks.\nOur first speaker is Dr. Josiah Carlson. He lives and works in Los Angeles, currently as VP of Technology at OpenMail.\nTitle: jobs.py\nTalk Abstract\nOne of the difficulties with distributed concurrency is caused by concurrent data access. While transactional databases in the SQL and NoSQL varieties abound, transactions and ACID compliance may not be enough to prevent data consistency, availability, and deadlock failures in a concurrent environment - even when you are only accessing a single database on a single server\nAfter surveying Luigi, Airflow, and various Jenkins plugins intended to solve this particular problem, we realized that OpenMail needed something that could be used from more than just our ETL flows, and we couldn\u0026rsquo;t spend months rewriting everything to use it. Enter jobs.py, developed to be easily added to an existing Python codebase, meant to handle locking around named \u0026ldquo;input\u0026rdquo; and \u0026ldquo;output\u0026rdquo; resources with a decorator or context manager.\nAt the talk, we will be discussing what jobs.py does, why we explicitly limited the scope of jobs.py the way we did, how OpenMail has used it to ensure success in our processes, and how you might want to use jobs.py.\nOur second speaker is Dr. Chris Fonnesbeck. Chris is an Assistant Professor in the Department of Biostatistics at the Vanderbilt University Medical Center in Nashville, TN. He specializes in computational statistics, Bayesian methods, meta-analysis, and applied decision analysis. Chris started the PyMC project, a package for Bayesian statistical analysis in Python, and continues as a PyMC developer today. He originally hails from Vancouver, BC and received his Ph.D. from the University of Georgia.\nTitle: Building Bayesian Models in Python with PyMC3\nTalk Abstract\nComputationally-intensive statistical algorithms, such as newer gradient-based MCMC methods, reveal computing tradeoffs inherent in making useful statistical software. On one hand, acceptable performance typically requires a compiled language; on the other, the flexibility to easily implement a variety of models, custom statistical distributions and algorithms are best satisfied by high-level scripting languages. We will describe how Python in general and PyMC3 in particular offer statisticians the sweet spot in this tradeoff. PyMC3 includes several newer computational methods for fitting Bayesian models, including Hamiltonian Monte Carlo (HMC) and automatic differentiation variational inference (ADVI). Python\u0026rsquo;s intuitive syntax is helpful for new users, and has allowed developers to keep the PyMC3 code base simple, making it easy to extend the software to meet analytic needs. PyMC3 itself extends Python\u0026rsquo;s powerful \u0026ldquo;scientific stack\u0026rdquo; of development tools, which provide automatic differentiation, fast and efficient data structures, parallel processing, and interfaces for describing statistical models.\nTimeline:\n6:30pm arrival, food/drinks and networking 7:15pm introductions, main talks start 8:30pm or 9:30 p.m. - networking and conversation 10:00pm closing You must have a confirmed RSVP and please arrive by 7:15pm the latest.\nPlease RSVP here on Eventbrite.\nVenue: Venice Arts, 1702 Lincoln Boulevard\nThanks OpenMail for being our amazing sponsors!\n","date":"January 19, 2017","externalUrl":null,"permalink":"/pydsla-pymc3-and-jobs-py/","section":"Posts","summary":"Update: Video recording of the meet up here:\nThis video is no longer available on YouTube (original link).\nCome to the PyDSLA January Meetup to hear talks about PyMC3 and jobs.py!\nREGISTER HERE\nFor this event we are incredibly fortunate to have two incredibly speakers give really interesting talks.\n","title":"PyDSLA: PyMC3 and jobs.py","type":"posts"},{"content":"","date":"November 24, 2016","externalUrl":null,"permalink":"/tag/google/","section":"Tags","summary":"","title":"Google","type":"tags"},{"content":"UPDATE: The meetup \u0026ldquo;sold out\u0026rdquo; in record time (35 minutes!) from announcing it on our mailing list. There are more than 250 people on the waiting list now.\nMachine Learning at Google # Come and hear about a range of the work that is being done in Google’s Los Angeles office in the area of machine learning:\nScot Fang, Google Engineer\nScot works on classification of mobile apps into taxonomies of categories. He will discuss aspects of his work related to machine learning, software engineering, and product applications.\nPayam Refael, Tech Lead, Google\nPayam works in the signals group at Google LA. The Signals Group builds systems that input text (web page, keyword, creative, query) and output signals for various applications. He will give an overview of these signals and their applications.\nWe will also host a brief panel discussion with Q\u0026amp;A to follow.\nWhen: Thursday December 1st, 2016\nTimeline:\n6.30: Doors open, pizza and drinks served.\n7.30: Talks start.\n8.30: Talks end. Car parking attendants will be there until 9.00.\nRSVP here on Eventbrite.\nWhere: Google Venice, 340 Main Street, Venice 90291*\nParking, food and drinks will be provided.\n*Use the entrance at 275 Sunset Avenue.\nThanks Google’s Tech Outreach Committee for selecting the speakers and hosting.\n","date":"November 24, 2016","externalUrl":null,"permalink":"/meetup-machine-learning-at-google/","section":"Posts","summary":"UPDATE: The meetup “sold out” in record time (35 minutes!) from announcing it on our mailing list. There are more than 250 people on the waiting list now.\nMachine Learning at Google # Come and hear about a range of the work that is being done in Google’s Los Angeles office in the area of machine learning:\n","title":"Meetup: Machine Learning at Google","type":"posts"},{"content":"Happy R dev; Linux \u0026amp; Emacs user; org of the Hungarian R User Group, the first satRday conference (2016 in Budapest), the European R Users Meeting (2018 in Budapest); ex CARD.com, rapporter.net and stats teacher, currently managing data ops at System1.com; father of 2 wonderful kids.\n","date":"November 18, 2016","externalUrl":null,"permalink":"/authors/gergely-daroczi/","section":"Authors","summary":"Happy R dev; Linux \u0026 Emacs user; org of the Hungarian R User Group, the first satRday conference (2016 in Budapest), the European R Users Meeting (2018 in Budapest); ex CARD.com, rapporter.net and stats teacher, currently managing data ops at System1.com; father of 2 wonderful kids.\n","title":"Gergely Daroczi","type":"authors"},{"content":"It has been over 2 months since the first satRday conference ended and I shared some quick impressions on that, where I also promised a follow-up post on my experiences as ending up being the main organizer of the event \u0026ndash; this is what you are reading now. Although the idea was born at the EARL Boston conference last year, and I decided to take part in the project while living in Los Angeles \u0026ndash; the conference took place in Budapest after all as logistics seemed to be easier there thanks to my closer relationship to the Hungarian R User Group.\nAnyway, the location does not matter that much: if you have the chance (and resources) to organize a satRday event anywhere around the world, please do \u0026ndash; as it\u0026rsquo;s great and extremely useful for the local community, and it\u0026rsquo;s a very rewarding (although often also demanding) experience for yourself :)\nDo it only if you are passionate! # As said it\u0026rsquo;s fun to take an active part in the project, but it\u0026rsquo;s a full-time job for at least a month.\nI\u0026rsquo;ve started the actual first steps (looking for venue, picking the date etc) at the beginning of June, and I thought it takes a few hours a week until September to get ready for the event. Actually, it became 2-3 hours a day fun project of my life for 3 months \u0026ndash; including weekends, which was sometimes pretty cumbersome besides doing a full-time job. So be prepared for that :)\nFor example, I did not even think about how many hours will I spend with writing raw HTML:\nSo it takes around 4 months and 100 commits to organize a #satRdays conference https://t.co/oVKe9Cn9H0 pic.twitter.com/V3HB5SiBVJ — Gergely Daróczi (@daroczig) · September 19, 2016 (archived tweet) Build a team # I was lucky to have Denes Tóth (co-organizer of the event) who volunteered for the project and took care most of the venue, catering, accommodation, and a whole bunch of other minor things that came up as we progressed with the organization. Also, I received many useful tips and connections from Bence Arato (organizer of BI and DW conferences in Hungary). Still, as the main organizer, I should have built a larger team right at the start and delegate better the tasks, because each task has so many sub-tasks, for example:\nDecide on the program (workshops, regular and lightning talks, posters) Choose and invite speakers Promote the event, be active on social media Keep track of the CfP, collect presentation materials from speakers Create and maintain a homepage Find sponsors, make and sign contracts, keep in touch, collect their roll-up banners Bookkeeping (invoices, receiving payments, taxes, flight tickets etc) Stationery (badge, booklet, rollup banner etc) Venue (rooms, seats, set up and test technology, IT guy) Catering (coffee breaks, lunch, pizza, beer, coke etc) Think about and organize social event(s) before or after the conference Recruit volunteers, plan the registration process Record the talks, take photos on the event Organize social events before and after the conference Reply e-mails, pick up the phone literally in 24/7 All these might sound pretty easy, but believe me: each item can take a lot of time, and you realize this only when you are eg tweaking the CSS of the conference homepage (on your way to a sponsor to chit chat a bit) you have manually put together a few days ago, as that seems to be the quickest way to provide some usable input (as a png export from the browser to be edited with Gimp that you never really used before and your designs skills suck anyway) for the press working on the roll-up banner to be printed in time.\nPick a date and venue # Although this is a community-driven event, but sometimes you just have to make some decisions on your own, and it\u0026rsquo;s not a good idea to wait for feedback from the community if you want to (or have to) get things done in time.\nThere were three important things we took into account when deciding the date of the conference:\nNo other data R, Data Science or BI conference should take place in Europe withing 2 weeks (eg late September wasn\u0026rsquo;t an option due to EARL London, October thanks to ERUM and the Budapest BI Forum) Pretty good weather (it\u0026rsquo;s not that fun to spend a few days in the snowy Budapest compared to late summer) August is the main holiday season in Hungary, so that\u0026rsquo;s not a good option, school starts in early/mid September, so we picked the last weekend of the summer.\nFunding # We were very fortunate to have the R Consortium as a strong sponsor even without a preliminary conference program \u0026ndash; it helped a lot and provided confidence when looking for further sponsors!\nOn the other hand, keeping the budget of the conference at a very low level still required us to raise at least $10K, knowing that we won\u0026rsquo;t make a lot by selling most of the conference tickets for $10.\nSo I started with a spreadsheet trying to list the expected expenses (which first estimates almost doubled until September) and also reached out to potential sponsors in my networks. Fortunately, the feedback was very positive in most cases, and I was soon pretty sure that organizing the conference will not cost a lot of money to my own small business, although I was prepared for that.\nThe most important learnings from managing the budget are:\nFood is expensive! It took ~50% of the overall budget to have 3 coffee breaks and a lunch On the other hand, ordering a few dozens of pizzas and buying a decent number of six packs in the local store can provide a light dinner for the attendees for the price of a coffee break, and it\u0026rsquo;s usually accepted well at meetups and RUGs anyway, so no need to stick to fancy things (eg at Budapest we paid around $20 for catering despite the fact that most attendees registered for $10) Conference swag is relatively cheap but works well when it comes to building community, eg I\u0026rsquo;ve found a place printing OK t-shirts for $4 and now I regret we didn\u0026rsquo;t print a conference booklet Try to dedicate some funds to help speakers with travel/accommodation costs \u0026ndash; it\u0026rsquo;s relatively cheap (compared to catering), but can boost willingness of speakers when it comes to giving up their free time to travel and spend time with the community Know (someone who knows) your tax system! It might sound silly, but I was shocked when it turned out that if my company pays for the lunch of the attendees, then I have to pay an extra ~50% \u0026ldquo;entertainment allowance\u0026rdquo;. In simple words, this minor thing added a mere 25% to the expenses. Timeline # One day is relatively short to do a bunch of different things: you cannot host workshops, multiple keynotes, dozens of regular talks (but you should do lightning talks!), a poster session and a data visualization contest! Well, you can on paper, but participants in real-life seemed to be pretty exhausted spending more than 12 hours at the conference venue \u0026ndash; although I was very happy to see the majority of the attendees joining the unofficial after-party in a nearby pub until late hours :)\nBut besides kidding, you really have to know the limits. Either do multiple tracks, or plan to host a reasonable number of talks. Eg having 25 speakers at Budapest was pretty crazy and the schedule did not allow much time for decent networking in coffee breaks. Especially if you (and you will) start with 15 mins late due to something going wrong at the registration.\nLet others know about the event! # Yeah, you have to active on Twitter, which seems to be an active channel among the R folks. I\u0026rsquo;m not on Facebook, so I cannot share much experience on that, but probably wouldn\u0026rsquo;t hurt to promote the event there as well :) And besides these formal channels, let your friends and network know about the event \u0026ndash; I\u0026rsquo;ve sent a number of newsletters and personal e-mails to promote the event for those who are not active on Twitter. And sending reminder e-mails to group before approaching deadlines (eg CfP, early-bird period) is also very important.\nWhat could go wrong? # Basically anything! Would you believe if I say:\nSpeakers cancel their talks a few days/hours before the event (that\u0026rsquo;s pretty normal and no personal offense at all, I\u0026rsquo;m just saying that you should be prepared for such things) The ho(s)tel cancels all speaker\u0026rsquo;s reservations a few weeks before the event without any explanation, and all other places are fully booked in the area by that time You get a phone call 12 hours before the start of the event from a guy, who turns out to be a pizza delivery person standing in front of the conference venue holding 30 pizzas, and not being happy when he realizes that they missed the date. Mixed feelings: you feel a bit proud that you ordered 2x30 pizzas from two different places hoping that at least some pizzas arrive on time, but starting to get worried if the very same 30 pizzas get delivered the next day On the other hand, you have no updates from the t-shirt webshop and you cannot reach them via phone at 5pm Friday, so you start drawing alternate plans on what you will do with the 200 satRday t-shirts on Monday when the late delivery arrives The official photographer of the event did not show up after all and you cannot reach him on phone or e-mail, so you end up taking photos with your mobile phone of the event you prepared for a few months The good news: all turned out to be absolutely OK and fun, no worries at all \u0026ndash; just be prepared that things might not be on track all the time.\nAnd most importantly: Is it worth it? # ▶ A brief visual summary on the first satRday conference in Budapest 2016 Every single minute (and penny) you spent on the project! As said in the first paragraph, you make good for the community, and the community is very grateful for your efforts. At the end of the (satR)day, although you feel physically exhausted, but it\u0026rsquo;s inspiring and rewarding to hear such feedbacks when reading the conference evaluation forms:\n\u0026ldquo;I expected a lot from the event, but it went beyond my expectations. Congratulations and thanks!\u0026rdquo;\nIn short, I hope to see and attend many future satRday events all around the world! Next in Cape Town and Puerto Rico.\n","date":"November 18, 2016","externalUrl":null,"permalink":"/how-to-organize-a-satrday-conference-v0-1/","section":"Posts","summary":"It has been over 2 months since the first satRday conference ended and I shared some quick impressions on that, where I also promised a follow-up post on my experiences as ending up being the main organizer of the event – this is what you are reading now. Although the idea was born at the EARL Boston conference last year, and I decided to take part in the project while living in Los Angeles – the conference took place in Budapest after all as logistics seemed to be easier there thanks to my closer relationship to the Hungarian R User Group.\n","title":"How-to organize a satRday conference (v0.1)","type":"posts"},{"content":"","date":"November 18, 2016","externalUrl":null,"permalink":"/tag/user/","section":"Tags","summary":"","title":"useR","type":"tags"},{"content":"","date":"October 14, 2016","externalUrl":null,"permalink":"/tag/meetup/","section":"Tags","summary":"","title":"meetup","type":"tags"},{"content":"REGISTER HERE ON EVENTBRITE\nDATE AND TIME\nWed, November 2, 2016 6:30 PM – 9:30 PM PDT Come to the PyDSLA November Meetup to hear talks from the community! In the last month, the organizers of the PyDSLA Meetup sent out a call for speakers, and the community answered with a number of great members coming forward willing and able to talk about some of the cool ways they are using the Python Data Science ecosystem to do cool stuff.\nTalks are still being finalized, but be prepared for an evening with at least 4-5 speakers from companies ranging from OpenMail to Domino Data Lab. Keep your eyes on this event as speakers will be finalized. The current list of speakers includes (this may change before the day of the event):\nEduardo Ariño de la Rubia - Chief Data Scientist at Domino Data Lab speaking about Keras, deep learning made kinda easy I guess Josiah Carlson - VP of Technology at OpenMail talking his recently open sourced data pipeline tool jobs.py Neal Fultz - Principal Data Scientist at OpenMail Roozbeh Davari - Data Scientist at The Honest Company speaking about Embracing the Noise: Using Bootstrapping For Improving ML Model Predictions Albert Hamood - Data Scientist at The Honest Company speaking about Predicting User Behavior Using Ensemble Learning Annie Flippo - Sr. Data Scientist at AwesomenessTV speaking about natural language processing in python Joshua Cook - Senior Python Engineer at ParkMe talking about a system for distributed processing in Python The talks will be short punchy talks, designed to get good information disseminated in the community, and get you more productive and excited about new tools and techniques! Full speaker bios to follow!\nTimeline:\n6:30pm arrival, food/drinks and networking 7:30pm introductions, main talk 8:30pm or 9 p.m. - networking and conversation 10:00pm closing You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\nVenue: Venice Arts, 1702 Lincoln Boulevard\nThanks OpenMail for being our amazing sponsors! ","date":"October 14, 2016","externalUrl":null,"permalink":"/pydsla-talks-from-the-la-data-science-community/","section":"Posts","summary":"REGISTER HERE ON EVENTBRITE\nDATE AND TIME\nWed, November 2, 2016 6:30 PM – 9:30 PM PDT Come to the PyDSLA November Meetup to hear talks from the community! In the last month, the organizers of the PyDSLA Meetup sent out a call for speakers, and the community answered with a number of great members coming forward willing and able to talk about some of the cool ways they are using the Python Data Science ecosystem to do cool stuff.\n","title":"PyDSLA: Talks from the LA Data Science Community","type":"posts"},{"content":"","date":"October 14, 2016","externalUrl":null,"permalink":"/tag/amazon/","section":"Tags","summary":"","title":"amazon","type":"tags"},{"content":"","date":"October 14, 2016","externalUrl":null,"permalink":"/tag/deep-learning/","section":"Tags","summary":"","title":"deep learning","type":"tags"},{"content":" Update: This event has ended. Slides are available here. # Update: Video recording by Carl Mullins: # This video is no longer available on YouTube (original link).\nREGISTER HERE ON EVENTBRITE\nDATE AND TIME\nThu, October 27, 2016 6:30 PM – 10:00 PM PDT Speaker: Scott Le Grand\nThis talk is a gentle introduction to Amazon open sourced deep learning library DSSTNE. The creation of DSSTNE is motivated by the massive scale and low latency computing needed for Amazon scale catalog and recommendation production problem. It is also motivated by the inherent extreme sparseness of recommendation or data sets in Amazon. In this talk Scott will walk through such requirements for a new (better) deep learning frame suitable for large scale production use. He will delineate a few considerations and solutions of how DSSTNE address those challenges. He will also briefly introduce DSSTNE’s API and how it can be used to easily reproduce the state of art deep nets topologies e.g. Alex Net. This talk also includes DSSTNE’s performance compared to TensorFlow and compared to other machine learning algorithms on benchmark datasets.\nBio: Scott Le Grand is currently a senior scientist in Tezza Technology. He is the lead author of DSSTNE (Deep Sparse Scalable Tensor Network Engine). DSSTNE is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models, and in one of forces powering the personalization of the gigantic Amazon retail business. Besides the creation of DSSTNE, Scott is also leading the effort of optimizing Deep Neural Network (DNN) performance on GPUs and GPU clusters and applying DSSTNE to general machine learning, deep learning and analytical applications.\nScott has been an expert in developing large scale distributed algorithms and systems. He developed life science services on Amazon\u0026rsquo;s Elastic Compute Cloud (EC2) as a principal engineer at Amazon AWS. Before Amazon he was a distinguished Principal Engineer and a CUDA Fellow in NVIDIA. There he was best known for his work porting the AMBER molecular dynamics package to CUDA, attaining record-breaking performance in the process. In a previous life, Scott picked up a B.S. in biology from Siena College and a Ph.D. in biochemistry from the Pennsylvania State University.\nDate: October 27, 2016\nTimeline:\n6:30pm arrival, food/drinks and networking 7:30pm introductions, main talk 8:30pm after party: more networking, drinks, games, demos etc 10:00pm closing You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\nVenue: Red Bull Media House, 1740 Stewart Street, Santa Monica\nParking: You can park for free in the gated Red Bull Media House parking lot (see picture below).\nThanks Red Bull Media House for hosting at this awesome venue.\n","date":"October 14, 2016","externalUrl":null,"permalink":"/dsstne-a-new-deep-learning-framework-for-large-sparse-datasets/","section":"Posts","summary":"Update: This event has ended. Slides are available here. # Update: Video recording by Carl Mullins: # This video is no longer available on YouTube (original link).\n","title":"DSSTNE: A New Deep Learning Framework for Large Sparse Datasets","type":"posts"},{"content":"","date":"October 14, 2016","externalUrl":null,"permalink":"/tag/neural-networks/","section":"Tags","summary":"","title":"neural networks","type":"tags"},{"content":"","date":"June 6, 2016","externalUrl":null,"permalink":"/tag/gbm/","section":"Tags","summary":"","title":"GBM","type":"tags"},{"content":"","date":"June 6, 2016","externalUrl":null,"permalink":"/tag/predictive-modeling/","section":"Tags","summary":"","title":"predictive modeling","type":"tags"},{"content":"","date":"June 6, 2016","externalUrl":null,"permalink":"/tag/production/","section":"Tags","summary":"","title":"production","type":"tags"},{"content":"XGBoost is a fantastic open source implementation of Gradient Boosting Machines, a general purpose supervised learning method that achieves the highest accuracy on a wide range of datasets in practical applications. Deep learning is all the hype now, but apart from specific domains such as images, speech or text (i.e. problems with higher-level abstractions to be learnt and/or perception problems, where deep learning achieved some remarkable results indeed), it is usually outperformed by Gradient Boosting in a majority of general business domains and supervised learning applications. Proof of this and also because XGBoost has an easy-to-use interface from both R and Python, XGBoost has become a favorite tool in Kaggle competitions. Besides feature engineering, cross-validation and ensembling, XGBoost is a key ingredient for achieving the highest accuracy in many data science competitions and more importantly in practical applications.\nWe were fortunate to recently host Tianqi Chen, the main author of XGBoost in a workshop and a meetup talk in Santa Monica, California.\nFirst, we started with an advanced workshop in the afternoon for which anyone could apply to participate but there were only a dozen spots available (which got us some expert users of XGBoost, but unfortunately we had to reject some good people too, sorry).\nThis advanced workshop had 2 sessions. In the first one Tianqi gave a talk touching on many system/implementation issues, slides are below:\nTianqi Chen - XGBoost: Implementation Details - LA Workshop Talk · Speaker Deck The second session was a Q\u0026amp;A and we discussed topics such as\ntuning the hyper-parameters fast real-time scoring (stay tuned for some news from Tianqi soon) XGBoost vs alternative GBM implementations random forests with XGBoost (yes, it\u0026rsquo;s possible with some undocumented options) various tricks that make XGBoost so fast (columnar store with sorted columns, CPU cache aware algorithm, excellent representation of sparse data etc.) details of how the out-of-core and the distributed implementations work and many other topics, see more in this github repo.\nThen, we had an evening meetup hosted by Red Bull in their amazing venue:\nThe evening talk covered a general overview of XGBoost and some of the latest developments. You can watch the video here\n▶ XGBoost A Scalable Tree Boosting System June 02, 2016 or browse the slides:\nTianqi Chen - XGBoost: Overview and Latest News - LA Meetup Talk · Speaker Deck One piece from the talk I\u0026rsquo;d like to single out is this: \u0026ldquo;17 out of 29 winning solutions in Kaggle last year used XGBoost\u0026rdquo;. I\u0026rsquo;d also like to add that Tianqi and XGBoost have received the John Chambers award from ASA and the HEP meets ML award from CERN.\nWe are really fortunate to have had Tianqi here, and even more so as this talk is one of the few of his talks on XGBoost that have been recorded and are available for everyone to view.\nI\u0026rsquo;d like to thank Tianqi for coming to LA and for the talks, Red Bull for hosting us and EA for sponsoring Tianqi\u0026rsquo;s trip to LA.\nSome further resources:\nXGBoost paper by Tianqi etal with a lot more details Winning Data Science competitions - a previous meetup talk by Jeong-Yoon Lee Bechmarking machine learning tools - a github repo by Szilard Pafka (that\u0026rsquo;s me :)) ","date":"June 6, 2016","externalUrl":null,"permalink":"/xgboost-workshop-and-meetup-talk-with-tianqi-chen/","section":"Posts","summary":"XGBoost is a fantastic open source implementation of Gradient Boosting Machines, a general purpose supervised learning method that achieves the highest accuracy on a wide range of datasets in practical applications. Deep learning is all the hype now, but apart from specific domains such as images, speech or text (i.e. problems with higher-level abstractions to be learnt and/or perception problems, where deep learning achieved some remarkable results indeed), it is usually outperformed by Gradient Boosting in a majority of general business domains and supervised learning applications. Proof of this and also because XGBoost has an easy-to-use interface from both R and Python, XGBoost has become a favorite tool in Kaggle competitions. Besides feature engineering, cross-validation and ensembling, XGBoost is a key ingredient for achieving the highest accuracy in many data science competitions and more importantly in practical applications.\n","title":"XGBoost workshop and meetup talk with Tianqi Chen","type":"posts"},{"content":"","date":"May 19, 2016","externalUrl":null,"permalink":"/tag/kaggle/","section":"Tags","summary":"","title":"kaggle","type":"tags"},{"content":"","date":"May 19, 2016","externalUrl":null,"permalink":"/tag/xgboost/","section":"Tags","summary":"","title":"xgboost","type":"tags"},{"content":"XGBoost is a fantastic open source implementation of Gradient Boosting Machines, one of the most accurate general purpose supervised learning algorithms. With interfaces from both R and Python among others, it has become a favorite tool in Kaggle competitions, and it\u0026rsquo;s also widely used in real-world applications. In this meetup we\u0026rsquo;ll have a talk from the best person on the topic,Tianqi Chen, main author of XGBoost.\nUPDATE: Video recording and slides are available now here.\nDate: June 2nd, 2016\n**Timeline:**– 7:00pm arrival, food/drinks and networking\n8:00pm introductions, main talk – 9:30pm after party: more networking, drinks, games, demos etc 11:30pm closing You must have a confirmed RSVP and please arrive by 7:55pm the latest. Please RSVP here on Eventbrite.\nVenue: Red Bull Media House, 1740 Stewart Street, Santa Monica\nParking: You can park for free in the gated Red Bull Media House parking lot.\nThanks Red Bull Media House for hosting at this awesome venue and **Electronic Arts (EA)**for sponsoring Tianqi\u0026rsquo;s trip to LA.\n","date":"May 19, 2016","externalUrl":null,"permalink":"/xgboost-a-scalable-tree-boosting-system/","section":"Posts","summary":"XGBoost is a fantastic open source implementation of Gradient Boosting Machines, one of the most accurate general purpose supervised learning algorithms. With interfaces from both R and Python among others, it has become a favorite tool in Kaggle competitions, and it’s also widely used in real-world applications. In this meetup we’ll have a talk from the best person on the topic,Tianqi Chen, main author of XGBoost.\n","title":"XGBoost: meetup announcement","type":"posts"},{"content":"Deep Learning @Whisper\ntalk by Nick Stucky-Mack\nVideo now available, thank you to community member Carl Mullins for providing video services:\n▶ Deep Learning @Whisper THU, MAY 12 AT 6 00 PM, VENICE, CA In this talk we will discuss how Whisper uses machine learning for natural language processing tasks. We will focus on how Whisper applies deep learning to moderate content created by its more than 20 million monthly active users. The talk will explore the problem of short text classification, and will cover Whisper\u0026rsquo;s choices in machine learning models, software development, and hardware.\nSpeaker bio: Nick works on machine learning at Whisper. Nick conceived of, built, and currently develops and maintains the arbiter project for deep-learning-based content moderation at Whisper. Nick joined as Whisper\u0026rsquo;s third data scientist. Before that, he earned a BA in Mathematics from Harvard, was a Blakemore Fellow at Tsinghua University in Beijing, and worked in adtech in Taipei.\nDate: May 12, 2016 (Thursday)\nTimeline:\n6:00pm food/bev \u0026amp; networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite.\nNo more free spots? To increase your chance to get in next time, consider signing up to theDataScience.LA mailing list (we announce all new meetups on that mailing list first).\nVenue: Whisper, 69 Windward Avenue, Venice, CA 90291\nThanks Whisper for hosting!\n(Notice: parking is tough/expensive)\n","date":"May 3, 2016","externalUrl":null,"permalink":"/deep-learning-whisper/","section":"Posts","summary":"Deep Learning @Whisper\ntalk by Nick Stucky-Mack\nVideo now available, thank you to community member Carl Mullins for providing video services:\n▶ Deep Learning @Whisper THU, MAY 12 AT 6 00 PM, VENICE, CA In this talk we will discuss how Whisper uses machine learning for natural language processing tasks. We will focus on how Whisper applies deep learning to moderate content created by its more than 20 million monthly active users. The talk will explore the problem of short text classification, and will cover Whisper’s choices in machine learning models, software development, and hardware.\n","title":"Deep Learning @Whisper","type":"posts"},{"content":"","date":"May 3, 2016","externalUrl":null,"permalink":"/tag/gpu/","section":"Tags","summary":"","title":"GPU","type":"tags"},{"content":"","date":"April 18, 2016","externalUrl":null,"permalink":"/tag/databricks/","section":"Tags","summary":"","title":"databricks","type":"tags"},{"content":"SIGN UP HERE!\nVideo Available Now:\n▶ PyDSLA The Nitty Gritty of Advanced Analytics Using Apache Spark in Python May 5, 2016 Talk by Miklos Christine, solutions engineer at Databricks\nApache Spark is the next big data processing tool for Data Scientist. As seen on the recent StackOverflow analysis, it\u0026rsquo;s the hottest big data technology on their site! In this talk, I\u0026rsquo;ll use the PySpark interface to leverage the speed and performance of Apache Spark. I\u0026rsquo;ll focus on the end to end workflow for getting data into a distributed platform, and leverage Spark to process the data for advanced analytics. I\u0026rsquo;ll discuss the popular Spark APIs used for data preparation, SQL analysis, and ML algorithms. I\u0026rsquo;ll explain the performance differences between Scala and Python, and how Spark has bridged the gap in performance. I\u0026rsquo;ll focus on PySpark as the interface to the platform, and walk through a demo to showcase the APIs.\nTalk Overview:\nSpark\u0026rsquo;s Architecture. What\u0026rsquo;s out now and what\u0026rsquo;s in Spark 2.0 Spark APIs: Most common APIs used by Spark Common misconceptions and proper techniques for using Spark. Demo:\nWalk through ETL of the Reddit dataset. SparkSQL Analytics + Visualizations of the Dataset using MatplotLib Sentiment Analysis on Reddit Comments Speaker:\nMiklos Christine is a solutions engineer for Databricks where he helps customers deploy and use Apache Spark to build batch and streaming applications. Miklos was previously a system engineer at Cloudera where he helped strategic customers deploy and use the Apache Hadoop ecosystem in production. He has contributed to several projects in the open source community and holds a BS in electrical engineering and computer sciences from the University of California-Berkeley.\nDate: May 5, 2016\nTimeline:\n6:30pm arrival, food/drinks and networking 7:0pm talk starts You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP here on Eventbrite and if the event is already full,sign up to the DataScience.LA priority mailing list to increase your change in getting in next time (we announce all events there first before publicizing on www.datascience.la or meetup.com).\nVenue: Venice Arts, 1702 Lincoln Boulevard, Los Angeles, CA 90291\nThanks OpenMail for hosting and for food/drinks!\n","date":"April 18, 2016","externalUrl":null,"permalink":"/pydsla-the-nitty-gritty-of-advanced-analytics-using-apache-spark-in-python/","section":"Posts","summary":"SIGN UP HERE!\nVideo Available Now:\n▶ PyDSLA The Nitty Gritty of Advanced Analytics Using Apache Spark in Python May 5, 2016 Talk by Miklos Christine, solutions engineer at Databricks\n","title":"PyDSLA: The Nitty Gritty of Advanced Analytics Using Apache Spark in Python","type":"posts"},{"content":"","date":"April 18, 2016","externalUrl":null,"permalink":"/tag/spark/","section":"Tags","summary":"","title":"spark","type":"tags"},{"content":"","date":"February 5, 2016","externalUrl":null,"permalink":"/tag/education-2/","section":"Tags","summary":"","title":"education","type":"tags"},{"content":"What has Kaggle learned from 2 million machine learning models?\nAnthony Goldbloom, founder and CEO of Kaggle\nUPDATE:Video recording of the talk available here\nKaggle is a community of almost 450K data scientists who have built almost 2MM machine learning models to participate in our competitions. Data scientists come to Kaggle to learn, collaborate and develop the state of the art in machine learning. This talk will cover some of the lessons on winning techniques we have learned from the Kaggle community.\nSpeaker bio: Anthony Goldbloom is the founder and CEO of Kaggle. In 2011 \u0026amp; 2012, Forbes Magazine named Anthony as one of the 30 under 30 in technology, in 2013 the MIT Tech Review named him one of top 35 innovators under the age of 35 and the University of Melbourne awarded him an Alumni of Distinction Award. He holds a first call honors degree in Econometrics from the University of Melbourne. Anthony has published in the The Economist and the Harvard Business Review.\nDate: February 4, 2016\nTimeline:\n6:00pm arrival, food/drinks and networking 6:30pm talk starts You must have a confirmed RSVP and please arrive by 6:25pm the latest. Please RSVP here on Eventbrite and if the event is already full,sign up to the DataScience.LA priority mailing list to increase your change in getting in next time (we announce all events there first before publicizing on www.datascience.la or meetup.com).\nVenue: Hulu, 2500 Broadway 2nd Floor, Santa Monica, CA 90404\nThanks Hulu for hosting and for food/drinks!\nParking: There is some limited street parking around. There is a parking garage underneath, but tickets will not be validated.\n","date":"February 5, 2016","externalUrl":null,"permalink":"/what-has-kaggle-learned-from-2-million-machine-learning-models/","section":"Posts","summary":"What has Kaggle learned from 2 million machine learning models?\nAnthony Goldbloom, founder and CEO of Kaggle\nUPDATE:Video recording of the talk available here\nKaggle is a community of almost 450K data scientists who have built almost 2MM machine learning models to participate in our competitions. Data scientists come to Kaggle to learn, collaborate and develop the state of the art in machine learning. This talk will cover some of the lessons on winning techniques we have learned from the Kaggle community.\n","title":"What has Kaggle learned from 2 million machine learning models?","type":"posts"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/tag/h2o/","section":"Tags","summary":"","title":"h2o","type":"tags"},{"content":"Join PyDSLA for a talk by Erin LeDell Intro to Machine Learning with H2O and Python\nGet your tickets HERE\nSummary:\nThis workshop will provide an overview of how to use H2O, the scalable open source machine learning library, from Python. The core algorithms of H2O are implemented in Java, however, fully-featured APIs are available in R, Python, Scala, and also through a web interface. The focus of this hands-on workshop will be the \u0026ldquo;h2o\u0026rdquo; Python module.\nAll of H2O\u0026rsquo;s algorithm implementations are distributed, which allows the software to scale to large datasets that may not fit into RAM on a single machine. H2O currently features distributed implementations of Generalized Linear Models, Gradient Boosting Machines, Random Forest and Deep Neural Nets. In this workshop, attendees will learn how to train machine learning models, perform grid search, cross-validation, and evaluate model performance using the H2O Python API.\nBio:\nErin LeDell is a Statistician and Machine Learning Scientist at H2O.ai. She is the main author of H2O Ensemble. Before joining H2O.ai, she was the Principal Data Scientist at Wise.io and Marvin Mobile Security (acquired by Veracode in 2012) and the founder of DataScientific, Inc.Erin received her Ph.D. in Biostatistics with a Designated Emphasis in Computational Science and Engineering from University of California, Berkeley. Her research focuses on ensemble machine learning, learning from imbalanced binary-outcome data, influence curve based variance estimation and statistical computing. She also holds a B.S. and M.A. in Mathematics.\nDate: January 19th, 2016 (Tuesday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:30pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite, as space is limited due to the capacity of the room. Make sure to RSVP now!\nVenue: Venice Arts, 1702 Lincoln Blvd, Venice, CA 90291\n","date":"January 5, 2016","externalUrl":null,"permalink":"/pydsla-erin-ledell-presents-intro-to-machine-learning-with-h2o-and-python/","section":"Posts","summary":"Join PyDSLA for a talk by Erin LeDell Intro to Machine Learning with H2O and Python\nGet your tickets HERE\nSummary:\nThis workshop will provide an overview of how to use H2O, the scalable open source machine learning library, from Python. The core algorithms of H2O are implemented in Java, however, fully-featured APIs are available in R, Python, Scala, and also through a web interface. The focus of this hands-on workshop will be the “h2o” Python module.\n","title":"PyDSLA: Erin LeDell, Presents Intro to Machine Learning with H2O and Python","type":"posts"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/tag/apache/","section":"Tags","summary":"","title":"apache","type":"tags"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/tag/data-warehouse/","section":"Tags","summary":"","title":"data warehouse","type":"tags"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/tag/hadoop/","section":"Tags","summary":"","title":"Hadoop","type":"tags"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/category/la_dw_bi_meetup/","section":"Categories","summary":"","title":"LA Data Warehouse, Business Intelligence \u0026 Analytics","type":"categories"},{"content":"Tickets available HERE\nTalk:\nJihoon Son Presents - Introduction to Apache Tajo: Future of Data Warehouse\nAbstract: Apache Tajo is a data warehouse system for Web-scale data. It provides virtual integration of a multitude of diverse data sources, thereby facilitating easy and rapid data integration which has been regarded as an essential, but heavy step in business intelligence. In addition, it has a fault-tolerable distributed query engine for accelerating query speed. With the “query federation” and “distributed processing” capacities, Tajo is capable of providing users with reliable and efficient analysis of Web-scale data spread on multiple sources. I will introduce Apache Tajo including its overall architecture, current state and challenges, and discuss advantages what Tajo can bring to users. In addition, I will give a brief demo of Tajo.\nDate: January 25th, 2016 (Monday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite as space is limited due to the capacity of the room.\nVenue: Edmunds, 1620 26th Street, Santa Monica, CA 90404.Thanks Edmunds for hosting and providing food/drinks!\n","date":"January 5, 2016","externalUrl":null,"permalink":"/la-data-warehouse-business-intelligence-and-analytics-jihoon-son/","section":"Posts","summary":"Tickets available HERE\nTalk:\nJihoon Son Presents - Introduction to Apache Tajo: Future of Data Warehouse\nAbstract: Apache Tajo is a data warehouse system for Web-scale data. It provides virtual integration of a multitude of diverse data sources, thereby facilitating easy and rapid data integration which has been regarded as an essential, but heavy step in business intelligence. In addition, it has a fault-tolerable distributed query engine for accelerating query speed. With the “query federation” and “distributed processing” capacities, Tajo is capable of providing users with reliable and efficient analysis of Web-scale data spread on multiple sources. I will introduce Apache Tajo including its overall architecture, current state and challenges, and discuss advantages what Tajo can bring to users. In addition, I will give a brief demo of Tajo.\n","title":"LA Data Warehouse, Business Intelligence and Analytics - Jihoon Son","type":"posts"},{"content":"","date":"January 5, 2016","externalUrl":null,"permalink":"/tag/tajo/","section":"Tags","summary":"","title":"tajo","type":"tags"},{"content":"With so much hype about \u0026ldquo;big data\u0026rdquo; and the industry pushing for \u0026ldquo;big data\u0026rdquo; analytical tools for everyone, the question arises how many people have big data (for analytics) and how many of them really need these tools (which are more complex and often more immature compared to the traditional tools for analytics).\nDuring the process of data analysis we typically start with some larger \u0026ldquo;raw\u0026rdquo; datasets, we transform/clean/prepare them for modeling (typically with SQL-like transformations), and then we use these refined and usually smaller datasets for modeling/machine learning.\nIn terms of computational resources needed I like to think in terms of the pyramid of analytical tasks. I\u0026rsquo;m mostly interested in tools for non-linear machine learning, the distribution of dataset sizes practitioners have to deal with in this area, and how all this is changing in time.\nSize of datasets in KDnuggets surveys # KDnuggets has conducted surveys of \u0026ldquo;the largest dataset you analyzed/data mined\u0026rdquo; (yearly since 2006). It surveys the largest dataset for a given practitioner (instead of the typical one), it measures size in bytes (rather than my preference for number of records), and it surveys raw data sizes (I would be more interested in the size of the refined datasets used for modeling). Nevertheless, it provides data points interesting to study. (One could also question the representativeness of the sample, changing respondents over the years etc.)\nThe annual polls are available on various URLs and I compiled the data into a csv file. The cumulative distribution of dataset sizes for a few select years is plotted below: The dataset sizes vary over many orders of magnitude with most users in the 10 Megabytes to 10 Terabytes range (a huge range), but furthermore with some users in the many Petabytes range.\nIt seems the cumulative distribution function in the 0.1-0.9 range (on the vertical axis) follows a linear dependency vs log(size): Fitting a linear regression lm(log10(size_GB) ~ cum_freq + year, ...) for that range, one gets coefficients year: 0.075 and cum_freq: 6.0. We can use this \u0026ldquo;model\u0026rdquo; as a smoother in the discussion below.\nThe above results imply an annual rate of increase of datasets of 10^0.075 ~ 1.2 that is 20%.\nThe median dataset size increases from 6 GB (2006) to 30 GB (2015). That\u0026rsquo;s all tiny, even more for raw datasets, and it implies that over 50% of analytics professionals work with datasets that (even in raw form) can fit in the memory of a single machine, therefore it can be definitely dealt with using simple analytical tools.\nOn the other hand, the dataset sizes are distributed over many orders of magnitude, e.g. the larger quantiles based on smoothing for 2015 are:\nquantile value 50% 30 GB 60% 120 GB 70% 0.5 TB 80% 2 TB 90% 8 TB The Terabyte range is the home turf of data warehouses, MPP/analytical databases and the like, but many organizations are using \u0026ldquo;big data\u0026rdquo; tools (Hadoop/Spark) for those sizes.\nAbout 5% of uses are in the Petabytes range and likely use Hadoop/Spark. While the hype around big data, \u0026ldquo;exponential growth\u0026rdquo; of sensors and Internet-of-Things (IoT) etc. suggests a more rapid growth rate than 20% yearly, the simple linear fit used above does not extend over the 90% percentile and it\u0026rsquo;s hard to tell any trends for these large sizes from this survey data.\nSize of datasets in other studies # A Microsoft research study has found that the median size of input jobs submitted to an analytic production Hadoop cluster at Microsoft in 2011 was 14 GB, and it infers from other studies that the median data size of input jobs in a Yahoo production cluster was 12 GB, while 90% of the inputs in an analytical production cluster at Facebook were of size less than 100 GB.\nSize of datasets for modeling # Unfortunately it is unclear from all this discussion above what\u0026rsquo;s the distribution of dataset sizes used for modeling/machine learning (my primary area of interest). Some informal surveys I have done at various meetups and conference talks suggest that for at least 90% of non-linear supervised learning use cases the data fits well in the RAM of a single machine and can be processed by high-performant tools like xgboost or H2O or in many cases (I estimate 60%) even by using R packages or Python sklearn (see this github repo for a benchmark of the most commonly used open source tools for non-linear supervised learning). Many of the \u0026ldquo;big data\u0026rdquo; tools in this domain (non-linear supervised learning) are clunky, slow, memory-inefficient and buggy (affecting predictive accuracy).\nSize of RAM of a single machine # The size of EC2 instances with largest RAM:\nyear type RAM (GB) 2007 m1.xlarge 15 2009 m2.4xlarge 68 2012 hs1.8xlarge 117 2014 r3.8xlarge 244 2016* x1 2 TB With different assumptions one can get yearly RAM increase rates of 50%, 60% or 70%:\nfrom_year from_GB to_year to_GB rate 2007 15 2014 244 50% 2007 15 2016 2000 70% 2009 68 2016 2000 60% Either way, the rate of increase of RAM of a single machine has been much higher than the rate of increase of the typical dataset used for analytics (20%). This has huge implications in terms of in-memory (distributed) processing (e.g. SQL) and single-machine processing (e.g. non-linear machine learning or even plain old R/Python). Big RAM is eating big data. For example, the fact that many datasets (already refined for modeling) now fit in the RAM of a single high-end server and one can train machine learning models on them without distributed computing has been noted by many top large scale machine learning experts.\nOf course, maybe data (useful for analytics) is increasing faster, and the slower 20% per yr increase based on the KDnuggets poll just shows our inability (or the inability of our tools) to deal with ever larger data or maybe there is some strong bias and non-representativeness in the KDnuggets survey etc. Maybe your data increases faster. Maybe you think data is bigger and increasing faster. But facts should trump opinions, so I\u0026rsquo;d love to see more data and analysis either supporting or contradicting the above results.\n","date":"November 18, 2015","externalUrl":null,"permalink":"/big-ram-is-eating-big-data-size-of-datasets-used-for-analytics/","section":"Posts","summary":"With so much hype about “big data” and the industry pushing for “big data” analytical tools for everyone, the question arises how many people have big data (for analytics) and how many of them really need these tools (which are more complex and often more immature compared to the traditional tools for analytics).\n","title":"Big RAM is eating big data - Size of datasets used for analytics","type":"posts"},{"content":"If you\u0026rsquo;ve never heard of kaggle, I guess the first question I have to ask is \u0026ldquo;how did you end up on DataScience.LA?\u0026rdquo; :)\nKaggle is one of the foremost machine learning competition websites on the internet. Companies from GE to Criteo and everything in between create competitions. These competitions are in the form of a dataset, ranging from small to relatively large (up to many many gigabytes), a stated goal (such as predicting an event or finding a numeric value) and an evaluation criteria. The Kaggle community has grown in the last few years, so that now competitions often host thousands of participants!\nOn October 27th, we were fortunate to have Jeong-Yoon Lee, Chief Data Scientist at Conversion Logic and Kaggle master, present to the DataScience.LA Data Science meetup. His presentation covered a wide range of topics, from the dynamics of team building on Kaggle, through advanced techniques such as how the winning team for the latest competition ensembled 64 different models to get their winning solution.\nCheck out his slides below, and his video above! It was a fantastic meetup and a great way to cap off the DataScience.LA 2015 Data Science meetup season. We hope to see more of you in 2016, hopefully not just in the audience, but on the podium speaking! The community works better when everyone contributes.\nJeong-Yoon Lee - Winning Data Science Competitions - Data Science Meetup - Oct 2015 · Speaker Deck We were very fortunate to have two members of the community shoot video for this event! If the above video from Jeff Weakley doesn\u0026rsquo;t show up, please try this alternate video from DSLA community member Carl Mullins.\nThis video is no longer available on YouTube (original link).\n","date":"November 13, 2015","externalUrl":null,"permalink":"/meetup-summary-winning-data-science-competitions/","section":"Posts","summary":"If you’ve never heard of kaggle, I guess the first question I have to ask is “how did you end up on DataScience.LA?” :)\nKaggle is one of the foremost machine learning competition websites on the internet. Companies from GE to Criteo and everything in between create competitions. These competitions are in the form of a dataset, ranging from small to relatively large (up to many many gigabytes), a stated goal (such as predicting an event or finding a numeric value) and an evaluation criteria. The Kaggle community has grown in the last few years, so that now competitions often host thousands of participants!\n","title":"Meetup Summary - Winning Data Science Competitions","type":"posts"},{"content":"","date":"October 29, 2015","externalUrl":null,"permalink":"/tag/enterprise/","section":"Tags","summary":"","title":"enterprise","type":"tags"},{"content":"One of the best things I like in working at CARD.com is that I am not only crunching R code in 24/7, but I also have the chance to interact with and improve the related data infrastructure with some interesting technologies.\nAfter joining the company in January, I soon realized that while Impala is a very powerful database for handling data that do not comfortably fit in MySQL, it\u0026rsquo;s still not as fast as one might expect when querying large amount of data from R. Sometimes I had to wait several minutes for a query to run! So I used this spare time to think about how to improve the workflow.\nInteracting with Impala from R is pretty straightforward: just install and load the RImpala package, which uses the JDBC driver to communicate with Impala. It does the job very well for fetching aggregated data form the database, but gets extremely slow when loading more than a thousand or so row \u0026ndash; that you cannot resolve buy throwing more hardware on the problem.\nSo when loading larger amount of data, the related R process is running with 100% CPU usage on one core, while doing the very same query from bash via impala-shell, the results are returned pretty fast. Why not exporting the data to a CSV file via impala-shell then?\nTL;DR: loading data into/from Impala via an intermediary CSV file may perform a lot better compared to using the JDBC driver.\nBenchmark # To compare the performance of the two approach in a reproducible way, I started an Elastic MapReduce cluster on AWS with a single m3.xlarge instance running the AMI version 3.8.0 with Impala 1.2.4 \u0026ndash; R already pre-installed. Then I downloaded the dbgen utility to generate some data for the benchmarks, as described in the Amazon EMR docs:\n$ dir test; cd test $ wget http://elasticmapreduce.s3.amazonaws.com/samples/impala/dbgen-1.0-jar-with-dependencies.jar $ java -cp dbgen-1.0-jar-with-dependencies.jar DBGen -p /tmp/dbgen -b 1 -c 0 -t 0 Then put the generated pipe-delimited data files on HDFS:\n$ hadoop fs -mkdir /data/ $ hadoop fs -put /tmp/dbgen/* /data/ $ hadoop fs -ls -h -R /data/ And load the data into Impala (not dealing with transforming the data into the Parquet File format or other tweaks as now we are comparing data transfer speed of the connectors):\ncreate EXTERNAL TABLE transactions(id BIGINT, customer_id BIGINT, book_id BIGINT, quantity INT, transaction_date TIMESTAMP) ROW FORMAT DELIMITED FIELDS TERMINATED BY \u0026#39;|\u0026#39; LOCATION \u0026#39;/data/transactions/\u0026#39;; Now we need to install the RImpala package, then download and extract the zip of JDBC jars as described on the package GitHub page:\n\u0026gt; install.packages(\u0026#39;RImpala\u0026#39;) \u0026gt; download.file(\u0026#39;https://github.com/Mu-Sigma/RImpala/blob/master/impala-jdbc-cdh5.zip?raw=true\u0026#39;, \u0026#39;jdbc.zip\u0026#39;, method = \u0026#39;curl\u0026#39;, extra = \u0026#39;-L\u0026#39;) \u0026gt; unzip(\u0026#39;jdbc.zip\u0026#39;) And all set to interact with the database after initializing the connection to Impala:\n\u0026gt; library(RImpala) Loading required package: rJava \u0026gt; rimpala.init(libs = \u0026#39;impala-jdbc-cdh5\u0026#39;) [1] \u0026#34;Classpath added successfully\u0026#34; \u0026gt; rimpala.connect() [1] TRUE \u0026gt; rimpala.query(\u0026#39;select count(1) from books\u0026#39;) count(1) 1 15969976 Now let\u0026rsquo;s see what happens when we load 10, 100, 1000 or let\u0026rsquo;s say 10K rows:\n\u0026gt; install.packages(\u0026#39;microbenchmark\u0026#39;); library(microbenchmark) \u0026gt; microbenchmark( + l1e1 = rimpala.query(\u0026#39;select * from books limit 10\u0026#39;), + l1e2 = rimpala.query(\u0026#39;select * from books limit 100\u0026#39;), + l1e3 = rimpala.query(\u0026#39;select * from books limit 1000\u0026#39;), + l1e4 = rimpala.query(\u0026#39;select * from books limit 10000\u0026#39;), times = 10) Unit: milliseconds expr min lq mean median uq max neval l1e1 373.8289 391.8877 393.5581 392.6247 400.7414 409.0315 10 l1e2 526.9520 534.7739 547.1202 543.5515 551.1544 578.5519 10 l1e3 1236.0779 1872.6034 1887.2793 1948.8703 2125.3348 2222.2258 10 l1e4 17801.2652 23253.7269 24784.2390 25836.3453 26850.4088 27717.6926 10 Almost 30 seconds to fetch 10K rows! Things getting very slow, right? So let\u0026rsquo;s create a minimal working R implementation of the above proposed method to export the results from Impala to a CSV file and load it via data.table::fread (due to read performance and I am using data.table in most of my R scripts anyway):\n\u0026gt; library(data.table) \u0026gt; query_impala \u0026lt;- function(query) { + + ## generate a temporary file name + fn \u0026lt;- tempfile() + + ## remove after read + on.exit(unlink(fn)) + + ## export results to this file + system(paste( + \u0026#39;impala-shell -B --quiet -q\u0026#39;, + shQuote(query), + \u0026#39;-o\u0026#39;, fn, + \u0026#39;\u0026#34;--output_delimiter=,\u0026#34;\u0026#39;, + \u0026#39;--print_header \u0026gt; /dev/null\u0026#39;)) + + ## read (and return) data like a pro + fread(fn) + + } Well, this function is extremely basic and can work only on localhost. For a more general approach with SSH access to remote databases, logging features and a bit of error handling, please see the updated query_impala function referenced at the end of this post.\nBut this simple function is fair enough to do some quick benchmarks on how JDBC and the CSV export/import hack performs with different number of rows fetched from Impala. Let\u0026rsquo;s run a loop to load 1, 10, 100, 1K, 10K and 100K values from the database via the two methods, each repeated by 10 times for future comparison:\n\u0026gt; benchmarks \u0026lt;- lapply(10^(0:5), function(limit) { + query \u0026lt;- paste(\u0026#39;select id from books limit\u0026#39;, limit + 1) + res \u0026lt;- microbenchmark( + rimpala = rimpala.query(query), + csv_hack = query_impala(query), + times = 10) + res$limit \u0026lt;- limit + res + }) And let\u0026rsquo;s transform the results of the benchmarks to an aggregated data.table, and plot the averages on a joint (log-scaled) graph:\n\u0026gt; df \u0026lt;- rbindlist(benchmarks)[, .(time = mean(time)), by = .(expr, limit)] \u0026gt; library(ggplot2) \u0026gt; ggplot(df, aes(x = limit, y = time/1e9, color = expr)) + geom_line() + + scale_y_log10(breaks = c(0.5, 1, 5, 15, 60, 60*5, 15*60)) + + scale_x_log10(breaks = 10^(0:5), + labels = c(1, 10, 100, \u0026#39;1K\u0026#39;, \u0026#39;10K\u0026#39;, \u0026#39;100K\u0026#39;)) + + xlab(\u0026#39;Number of rows\u0026#39;) + ylab(\u0026#39;Seconds\u0026#39;) + + theme_bw() + theme(\u0026#39;legend.position\u0026#39; = \u0026#39;top\u0026#39;) Unfortunately, I did not have patience to run this benchmark on more rows or columns, but this is already rather impressive in my (personal) opinion. In short, if you are querying more than a 100 rows from Impala and you have (SSH) console access to the server, you\u0026rsquo;d better use CSV export instead of waiting for the JDBC driver to deliver the data for you.\nQuick comparison of the CSV export and the RImpala approach # Please find this quick comparison of the discussed methods for fetching data from Impala to R:\nAdvantages Disadvantages RImpala - Can connect to remote database without SSH access - On CRAN - Slow when querying many rows - Java dependency on the client side - 20 megabytes of jar files for the driver CSV export - Scales nicely - No Java and jar dependencies, it\u0026rsquo;s damn simple - Needs SSH access for remote connections - Not on CRAN (yet) Second thoughts # So why is it happening? I was not 100% sure, but suspected it must be something with the jar files or how those are used in R. The query takes the very same amount of time inside of Impala as it does not matter if you export the data into a CSV file or pass it via the JDBC driver, but parsing and loading it takes extremely long with the latter.\nMentioning these interesting results to Szilard at a lunch, he suggested me to give a try directly querying Impala with the RJDBC package. It sounded pretty insane to use a different R wrapper to the very the same jar files of the RImpala package, but I decided to do this extra test after all to make sure it\u0026rsquo;s a Java and not an R implementation issue \u0026ndash; as per my proposal of keeping things simple (KISS) over using Java.\nSo I unzipped all the jar files used by the RImpala package above and created a new archive containing the merged content in a file named to impala-jdbc-driver.jar. Then loaded the RJDBC package and initialized a connection:\n\u0026gt; library(RJDBC) \u0026gt; drv \u0026lt;- JDBC(\u0026#34;org.apache.hive.jdbc.HiveDriver\u0026#34;, \u0026#34;impala-jdbc-driver.jar\u0026#34;, \u0026#34;\u0026#39;\u0026#34;) \u0026gt; conn \u0026lt;- dbConnect(drv, \u0026#34;jdbc:hive2://localhost:21050/;auth=noSasl\u0026#34;) Then we can use the very convenient dbGetQuery method from the DBI package to fetch rows from Impala, with the following impressive results:\nSo my hypothesis turned out to be wrong! The JDBC driver performs pretty well, even better compared to the CSV hack. I was even tempted to revert our production R scripts to use JDBC instead of the below function using temporary files to read data from Impala, but decided to keep the non-Java approach for multiple reasons after all:\nNo Java dependency on the server. Although storage is cheap nowadays, and I\u0026rsquo;ve already created a minimal Docker image including R, Java and rJava, but do we really need that 60% increase in the Docker image size? Compare the minimal R Docker image size of with the R+Java Docker image of No memory issues when loading large amount of data. By default, the rJava packages starts JVM with 512 MB of memory, which might not be enough for some queries, so you have to update this default value via eg options(java.parameters = '-Xmx1024m') before loading the rJava package. I prefer using SSH to access the data even if SSL encryption is available with the JDBC driver as well. This might sound silly, but managing users and authentication methods can be a lot easier via traditional Linux users/groups compared to Impala, especially with older CDH releases. Not speaking about in-database permissions here, of course. Although JDBC can be perfect for reading data from Impala, writing to the database might be a nightmare. I am not aware of any bulk import feature via the JDBC driver, and separate INSERT statements are extremely slow. So instead of preparing SQL statements, I prefer creating an intermediary dump file to be imported by Impala on the command line \u0026ndash; via a helper R function that does all these steps automatically. I did not prepare any benchmarks on this, but believe me, it\u0026rsquo;s a LOT faster. The same also stands for eg Redshift, where loading data from S3 or remote hosts via SSH and using COPY FROM instead of INSERT statements can result in multiple orders of magnitude speedup. This hack seems to be used by the Spark folks as well. Proof-of-concept demo function to use intermediary CSV files to export data from Impala # If you find this useful, please see the below function to automate the required steps of using an intermediary file instead of JDBC to load data from Impala :\nconnect to a remote host via SSH create a temporary CSV file on the remote host dump the results of the Impala query to the CSV file copy the CSV to a local file read the content of the CSV clean up the local and remote workspaces. Comments and any kind of feedback is highly welcomed!\n#\u0026#39; Connect and query Imapala via SSH and Impala-shell #\u0026#39; @param query SQL query to run #\u0026#39; @param host server hostname or IP #\u0026#39; @param log enable or disable logging of debug/trace messages #\u0026#39; @return data.table object #\u0026#39; @export query_impala \u0026lt;- function(query, host = \u0026#39;localhost\u0026#39;, log = require(futile.logger)) { ## measure time of query timer \u0026lt;- proc.time() ## make sure the query is a one-liner query \u0026lt;- gsub(\u0026#39;\\n\u0026#39;, \u0026#39; \u0026#39;, query) ## try to connect and create a remote temp file if (log) flog.trace(\u0026#39;Connecting to %s and creating tempfile.\u0026#39;, host) rfn \u0026lt;- system(paste0( \u0026#39;ssh \u0026#39;, host, \u0026#39; \u0026#34;mktemp --tmpdir=/tmp rimpala-XXXXXXXX.csv\u0026#34;\u0026#39; ), intern = TRUE) ## error handling if (length(rfn) == 0) { stop(\u0026#39;SSH access error.\u0026#39;) } if (log) flog.trace(\u0026#39;Remote tempfile: %s\u0026#39;, rfn) ## build impala-shell query cmd \u0026lt;- paste( \u0026#39;impala-shell -B --quiet -q\u0026#39;, shQuote(query), \u0026#39;-o\u0026#39;, rfn, \u0026#39;\u0026#34;--output_delimiter=,\u0026#34;\u0026#39;, \u0026#39;--print_header \u0026gt; /dev/null\u0026#39;) ## and log then run it if (log) flog.trace(paste0(\u0026#39;Querying impala-shell directly via: \u0026#39;, cmd)) system(paste(\u0026#39;ssh\u0026#39;, host, shQuote(cmd))) ## copy CSV file to localhost lfn \u0026lt;- tempfile(pattern = \u0026#39;rimpala-\u0026#39;, fileext = \u0026#39;.csv\u0026#39;) if (log) flog.trace(\u0026#39;Copying tempfile to temporary location on localhost: %s\u0026#39;, lfn) system(paste0( \u0026#39;scp -q -C \u0026#39;, host, \u0026#39;:\u0026#39;, rfn, \u0026#39; \u0026#39;, lfn)) ## read data like a pro if (log) flog.trace(\u0026#39;Reading %s bytes of data.\u0026#39;, file.info(lfn)$size) res \u0026lt;- tryCatch(fread(lfn, sep = \u0026#39;,\u0026#39;), error = function(e) e) ## if we cannot read data as a pro if (inherits(res, \u0026#39;error\u0026#39;)) { res \u0026lt;- data.table(read.csv(lfn)) } ## cleanup if (log) flog.trace(\u0026#39;Deleting temporary files on localhost and remote.\u0026#39;) unlink(lfn) system(paste(\u0026#39;ssh\u0026#39;, host, \u0026#39;rm\u0026#39;, rfn)) ## log query time if (log) flog.trace(\u0026#39;Query finished with %s rows after %s seconds.\u0026#39;, nrow(res), round(as.numeric(proc.time() - timer)[3], 2)) ## return res } Source: gist 353c714 by daroczig\n","date":"October 29, 2015","externalUrl":null,"permalink":"/r-and-impala-its-better-to-kiss-than-using-java/","section":"Posts","summary":"One of the best things I like in working at CARD.com is that I am not only crunching R code in 24/7, but I also have the chance to interact with and improve the related data infrastructure with some interesting technologies.\nAfter joining the company in January, I soon realized that while Impala is a very powerful database for handling data that do not comfortably fit in MySQL, it’s still not as fast as one might expect when querying large amount of data from R. Sometimes I had to wait several minutes for a query to run! So I used this spare time to think about how to improve the workflow.\n","title":"R and Impala: it's better to KISS than using Java","type":"posts"},{"content":"Our next Data Science meetup:\nWinning Data Science Competitions\nby Jeong-Yoon Lee\nIn this talk, I am going to share my experiences in data science competitions. After an overview of the competition structure and best practices, I will present the 1st prize winner’s solution for KDD Cup 2015 with the behind-the-scenes story.\nSpeaker: Jeong-Yoon Lee is a Chief Data Scientist at Conversion Logic. Jeong is one of top competitors at Kaggle (ranked 14th out of 370K), and has won many data science competitions including KDD Cup 2012 and 2015. Prior to joining Conversion Logic, Jeong worked at Demand Media and Opera Solutions focusing on machine learning and data analytics. He received his PhD in computer science and MS in electrical engineering at USC.\nDate: October 27, 2015\nTimeline:\n6:30pm arrival, food/drinks and networking 7:00pm talk starts You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP here on Eventbrite.\nVenue: Hulu, 2500 Broadway, Santa Monica, CA 90404\nThanks Hulu for hosting!\nParking: There is some limited street parking around. There is a parking garage underneath, but tickets will not be validated.\n","date":"October 19, 2015","externalUrl":null,"permalink":"/winning-data-science-competitions/","section":"Posts","summary":"Our next Data Science meetup:\nWinning Data Science Competitions\nby Jeong-Yoon Lee\nIn this talk, I am going to share my experiences in data science competitions. After an overview of the competition structure and best practices, I will present the 1st prize winner’s solution for KDD Cup 2015 with the behind-the-scenes story.\n","title":"Winning Data Science Competitions","type":"posts"},{"content":"In a partnership with the City of Los Angeles government, LA City College (LACC) has formed a civic data analysis team for 2- and 4-year college students and anyone looking to enter the data science profession. The team will complete a beginning-level data analysis project on Friday afternoons during a 10-week period during the Fall 2015 semester, using data provided by the City of LA. We will present our results to City of LA staff in December.\nThe project is structured as a not-for-credit course through the college’s extension program. LACC faculty member Jocelyn Graf is leading the team and managing the project. LACC is well-known for its math program. The LACC Math Club consistently wins individual and group awards at the national AMATYC competition. In addition to strong math and computer science students, the team will draw from students in pre-health professions and political science because the data has a public health angle.\nThe partnership originated at the Tech LA Conference sponsored by LA Mayor Garcetti in May 2014, where the City launched its Open Data Initiative. There, Jocelyn approached the staff of the City’s Information Technology Agency (ITA) about finding a way to get community college students involved with analyzing the open data and received an enthusiastic response from Ted Ross, now the City’s CIO and General Manager of the ITA. However, rather than simply advise on which open data set to download, he offered something better: to work directly with City staff on a set of anonymized data to produce real policy proposals.\nIt took time to usher the project through the administrative structure of both the City and the College, but the team is finally launching this semester with its first project. Because it is housed in the extension, team members do not have to be enrolled in credit courses at LACC to participate. Jocelyn hopes that about half the team members will come from the community and nearby 4-year schools, including people interested in learning to apply statistics and programming to a real data set, as well those who would like to focus on data visualization or the policy side.\nInterested in participating as a team member? Visit our info session and intro workshop: Friday Oct. 2, 1:00-3:30 p.m. Franklin Hall Room 106, Los Angeles City College, 855 N. Vermont Ave., Los Angeles, CA 90029. Sign up here. After the first week, which is free, the complete Fall semester experience costs $249. Full and partial scholarships are available. Please visit the course page for more details.\nInterested in volunteering as a mentor? We would like working data scientists to provide lightning talks describing their work or explaining a specific analytical technique, followed by hands-on practice in the computer lab. Mentors are also needed to answer questions remotely for students using Python, R, or Excel and to advise on statistical analysis.\nThe project involves a non-disclosure agreement (NDA) for the data, which is not open to the public.\nThe data science team also has a Meetup page. Please join to follow our progress.\n","date":"September 29, 2015","externalUrl":null,"permalink":"/la-city-college-forms-data-science-team-with-city-of-la/","section":"Posts","summary":"In a partnership with the City of Los Angeles government, LA City College (LACC) has formed a civic data analysis team for 2- and 4-year college students and anyone looking to enter the data science profession. The team will complete a beginning-level data analysis project on Friday afternoons during a 10-week period during the Fall 2015 semester, using data provided by the City of LA. We will present our results to City of LA staff in December.\n","title":"LA City College Forms Data Science Team with City of LA","type":"posts"},{"content":"","date":"June 23, 2015","externalUrl":null,"permalink":"/tag/analytics/","section":"Tags","summary":"","title":"analytics","type":"tags"},{"content":"Click HERE to sign up on Eventbrite!\nThis is the second event of the new Data Warehouse, Business Intelligence and Analytics meetup.\nTalks:\n1. Eduardo Arino de la Rubia: MySQL to Vertica with Tungsten Replicator\nIn this very (very) short talk, I will discuss how you can, for free, stream your MySQL data to a Vertica Community Edition cluster, and do near real-time analytics on up to 1TB of data.\nBio: I am a computer programmer and technologist with over 30 years experience, working in many different fields ranging from expert systems for compliance to manufacturing robotics. I am a graduate of Middle Tennessee State University and live with my wife, son, and 3 dogs in lovely Santa Monica.\n2. Lior Abraham: Building Modern Analytics for Trillions of Data Points\nAbstract: There are many systems designed to address large volumes of data, but analytics, or ad-hoc queries, are often left in the dust. Query times on the order of hours or days. Data consumers are required to write code or complex SQL to answer even the most basic questions. At Facebook we built real-time systems that reduced to complex queries answered by hundreds of servers across terabytes of data in seconds or less. This talk covers how we applied the same principles to data, with one use case in-mind: analytics. How do we achieve scale at trillions of rows, maintain interactive speeds, keep broad flexibility and expressibility, and above all, make data accessible. Further, if we can achieve this, what is the impact on data science and data consumers.\nBio: Mr. Lior Abraham is a Co-Founder of Interana, Inc. Mr. Abraham was instrumental in scaling Facebook\u0026rsquo;s infrastructure from a few million users to over a billion, as well as bringing its most technically challenging products to market. This included building many of the back-end systems that powered Facebook, like the News Feed, and the real time comment stream for the first Obama Inauguration. He also built Facebook\u0026rsquo;s most popular internal tools for data analysis. He previously worked for Veritas/Symantec. He received his B.S. in Electrical Engineering and Computer Science from U.C. Berkeley.\nDate: July 1st, 2015 (Wednesday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite as space is limited due to the capacity of the room.\nVenue: Edmunds, 1620 26th Street, Santa Monica, CA 90404.Thanks Edmunds for hosting and providing food/drinks!\n","date":"June 23, 2015","externalUrl":null,"permalink":"/announcing-meetup-lior-abraham-of-interana/","section":"Posts","summary":"Click HERE to sign up on Eventbrite!\nThis is the second event of the new Data Warehouse, Business Intelligence and Analytics meetup.\nTalks:\n1. Eduardo Arino de la Rubia: MySQL to Vertica with Tungsten Replicator\nIn this very (very) short talk, I will discuss how you can, for free, stream your MySQL data to a Vertica Community Edition cluster, and do near real-time analytics on up to 1TB of data.\n","title":"Announcing Meetup! Lior Abraham of Inter|Ana","type":"posts"},{"content":"","date":"June 23, 2015","externalUrl":null,"permalink":"/tag/bi/","section":"Tags","summary":"","title":"bi","type":"tags"},{"content":"","date":"June 23, 2015","externalUrl":null,"permalink":"/tag/business-intelligence/","section":"Tags","summary":"","title":"business intelligence","type":"tags"},{"content":"","date":"June 23, 2015","externalUrl":null,"permalink":"/tag/dw/","section":"Tags","summary":"","title":"dw","type":"tags"},{"content":"","date":"June 23, 2015","externalUrl":null,"permalink":"/tag/scale/","section":"Tags","summary":"","title":"scale","type":"tags"},{"content":"","date":"June 18, 2015","externalUrl":null,"permalink":"/tag/carlos-guestrin/","section":"Tags","summary":"","title":"carlos guestrin","type":"tags"},{"content":"","date":"June 18, 2015","externalUrl":null,"permalink":"/tag/dato/","section":"Tags","summary":"","title":"dato","type":"tags"},{"content":"On May 14th at the Venice Arts Center, the Los Angeles Python and Data Science communities were fortunate to hear a talk from world-renowned expert and CEO of Dato, Carlos Guestrin. Often times when you have a meetup where a presenter is tightly identified with a company, the concern is that the presentation will fundamentally become a product pitch. We had nothing to be worried about in this case - Carlos presented for nearly an hour on a wide range of subjects from how the Machine Learning pipeline has changed over time to Deep Learning!\nWe were incredibly fortunate to have DataScience.LA community member Carl Mullins record the meetup, as well as heroically recover it in the face of data loss. (Thanks, Carl!)\nEnjoy watching Carlos’ presentation, and we’re looking forward to seeing you at our next meetup!\n","date":"June 18, 2015","externalUrl":null,"permalink":"/pydsla-video-carlos-guestrin-ceo-of-dato/","section":"Posts","summary":"On May 14th at the Venice Arts Center, the Los Angeles Python and Data Science communities were fortunate to hear a talk from world-renowned expert and CEO of Dato, Carlos Guestrin. Often times when you have a meetup where a presenter is tightly identified with a company, the concern is that the presentation will fundamentally become a product pitch. We had nothing to be worried about in this case - Carlos presented for nearly an hour on a wide range of subjects from how the Machine Learning pipeline has changed over time to Deep Learning!\n","title":"PyDSLA Video: Carlos Guestrin, CEO of DATO","type":"posts"},{"content":"I currently have the need for machine learning tools that can deal with observations of the order of 10 millions in the context of binary classification. That kind of data is a few GBs in size and it fits comfortably nowadays in the RAM of a decent single machine. It is a trivial task for linear models, as there are plenty of open source tools that can train a logistic regression with this amount of data on a single machine in a few seconds, even while using only 1 processor core (many of these tools are single-threaded). Linear models are also the gold standard of large-scale machine learning that can run on clusters, processing very large distributed datasets.\nHowever, many problems would profit from modeling non-linearities. The state of the art in predictive accuracy are machine learning algorithms such as random forests, boosting, neural networks (and more recently deep learning) and (non-linear) support vector machines. In this post we\u0026rsquo;ll take a look at random forests, perhaps the easiest to train of the above (as there are few \u0026ldquo;knobs\u0026rdquo; to set) and often one of the most accurate algorithms. In fact, they are regularly used as a starting point in Kaggle competitions. I\u0026rsquo;m planning to look at the other methods as well, so more posts will follow.\nRandom forests have several commonly known implementations in R packages, Python scikit-learn, Weka, H2O, Spark MLLib, Mahout, Revo ScaleR, among others. For the purposes of this post, I am interested in which tools can deal with 10 million observations and train a random forest in a reasonable time (i.e. a few hours at most). Therefore, I\u0026rsquo;m analyzing the scalability, speed and accuracy of various random forest implementations.\nIn terms of scalability, I\u0026rsquo;m studying if the algorithms are able to complete (in decent time) for the given data sizes with given memory (RAM) constraints. Note that the largest dataset in this study is less than 1GB, so scaling out to multiple machines should not be necessary even if the tool can run in a distributed mode, therefore it is not the focus in this study. Actually, some machine learning algorithms perform relatively poorly in the multi-node setting, where communication is over the network rather than via updating shared memory. While we are certainly living in a \u0026ldquo;big data\u0026rdquo; craze, more and more voices are recognizing the value of choosing simpler solutions and not just using distributed computing unnecessarily because of the hype. \u0026ndash; Speed (in the single node setting) is determined by the computational complexity of the algorithm, but also if the implementation can use multiple processor cores or not. Accuracy is measured by the area under the ROC curve (AUC).\nTraining datasets of sizes 10K, 100K, 1M, 10M are generated from the well-known airline dataset, using data from years 2005 and 2006. A test set of size 100K is generated from the same dataset using year 2007. The task is to predict whether a flight will be delayed by more than 15 minutes. The tests have been carried out on a Amazon EC2 c3.8xlarge instance (32 cores, 60GB RAM). For some of the models that ran out of memory with the larger data sizes a r3.8xlarge instance (32 cores, 250GB RAM) has been used occasionally.\nThe models have been trained with default values for the (hyper-)parameters. For each algo/tool and each size n we observe the following: training time, maximum memory usage during training, CPU usage on the cores, and AUC as a measure for predictive accuracy. Times to read the data, pre-process the data, score the test data are also observed but not reported (e.g. they were not the bottleneck).\nMore details on this setup and the entire study, along with all the code used to get the results, can be found at this Github repository. I\u0026rsquo;d like to emphasize this is not aimed to be a comprehensive benchmark. It started from my need at work to find a tool that can deal with datasets of similar size and structure. A more comprehensive benchmark would study different datasets of different structure (e.g. dense, sparse, etc.), how results are changing as a function of hyper parameter values, the scaling out to multiple nodes for the distributed systems, larger datasets, and the like.\nI also want to emphasize the importance of testing the tools on your data: your specific data structure, data size, etc. It is very common lately (especially in a startup environment) to go with the hype and use a tool primarily because of its social media prowess without trying out less flashy alternatives which may better fit your needs. I have seen too many people falling into the \u0026ldquo;big data\u0026rdquo; trap and using these \u0026ldquo;big data\u0026rdquo; tools on data sets that don\u0026rsquo;t really require such an approach (but this way adding extra costs in resources needed, efficiency or usability). Those who may be falling into this trap are often overheard saying, \u0026ldquo;We are building a Hadoop cluster. Our data is about 100 GB total. Our largest table is a few GBs.\u0026rdquo; Therefore, please don\u0026rsquo;t stop your inquiry after reading a few blog posts on the topic or even this benchmark. Instead, think about your tasks and try out several tools to find out what works for your problem.\nIn our experiments, random forests with 500 trees have been trained in each tool with default hyper-parameter values. The training times and AUC as a function of the dataset size are plotted in the figures below (with more details available on Github).\nThe R implementation (randomForest package) is slow and inefficient in memory use. It cannot cope by default with a large number of categories, therefore the data had to be one-hot encoded. The implementation uses 1 processor core, but with 2 lines of extra code it is easy to build the trees in parallel using all the cores and combine them at the end. However, it runs out of memory already for n = 1M. I have to emphasize this has nothing to do with R per se, as it is instead the particular (C and Fortran) RF implementation used by the randomForest package that is inefficient. (Note that I still stand by my argument that R is the best data science platform especially when it comes to data munging and visualization.)\nThe Python (scikit-learn) implementation is faster, more memory efficient and uses all the cores. Variables needed to be one-hot encoded (which is more involved than for R) and for n = 10M doing this exhausted all the memory. Even if using a larger machine with 250GB of memory (and 140GB free for RF after transforming all the data) the Python implementation runs out of memory and crashes for this larger size. The algo finished successfully though when run on the larger box with simple integer encoding (which for some datasets/cases might be actually a good approximation/choice).\nThe H2O implementation is fast, memory efficient and uses all cores. It deals with categorical variables automatically. It is also more accurate than R/Python, which may be because of dealing properly with the categorical variables, i.e. internally in the algo rather than working from a previously 1-hot encoded dataset (where the link between the dummies belonging to the same original variable is lost).\nSpark (MLlib) implementation is somewhat slower, provides the lowest accuracy and it crashes already at n = 1M due to inefficient memory handling. With 250G of RAM it finishes for n = 1M, but runs out of memory for n = 10M. However, as Spark can run on a cluster one can throw in even more RAM by using more nodes. Alternatively, on a single machine, it is possible to train random forests with a smaller number of trees (but then accuracy decreases). I also tried to provide the categorical variables encoded simply as integers and passing the categoricalFeaturesInfo parameter, but that made training much slower. A convenience issue, reading the data is more than one line of code and Spark does not provide a one-hot encoder for the categorical data (therefore I used R for that). Note again the low prediction accuracy vs the other methods. One can improve a bit by increasing the maximum depth of trees (but only to Spark\u0026rsquo;s limit of 30), but then training slows down further and AUC is still lower than with the other methods. Finding the reason for the lower AUC would need more investigation (the reason might be that predict for Spark decision trees returns 0/1 and not probability scores therefore the random forest prediction is based on voting not probability averaging, or different stopping criteria, or just an algorithm that uses some approximations that hurts accuracy). Update: See some improvements in the comments here, here and here.\nI also tried xgboost, a popular library for boosting which is capable to build random forests as well. It is fast, memory efficient and of high accuracy. Note the different shapes of the AUC and runtime vs dataset sizes for H2O and xgboost, however.\nIn addition to the above, several other random forest implementations have been tested (Weka, Revo ScaleR, Rborist R package, Mahout) but all of them proved slow and/or unable to scale to the larger sizes. While not the focus of this study, there are signs that running the distributed random forests implementations (e.g. H2O) on multiple nodes does not provide the speed benefit one would hope for (because of the high cost of shipping the histograms at each split over the network).\nBesides random forests, I also trained various implementations of logistic regression (i.e. linear models). Their AUC as a function of dataset size is plotted below. One can see that the linear models\u0026rsquo; accuracy increases only a little from 100K to 1M and it is virtually the same for 1M and 10M. This is because a simple linear structure can be extracted from a smaller dataset and having more data points will not change the classification boundary significantly. On the other hand, more complex models such as random forests can further improve with increasing data size by further adjusting the classification boundary. However, one needs to pay a price in increased computational time for these more complex models. For example, if using H2O it takes 4000 seconds to train a random forest on the largest size and only 5 seconds to train a linear model.\nAn interesting point to note is that the AUC for the random forest trained on 100K observations is better than the AUC on a linear model trained on 10M observations. Therefore the answer to the question \u0026ldquo;more data or better algorithms?\u0026rdquo; often posed nowadays is that it depends (rather than the often argued \u0026ldquo;more data usually beats better algorithms\u0026rdquo;).\nTo summarize, the commonly used R and Python random forest implementations have serious difficulties in dealing with training sets of tens of millions of observations. H2O or xgboost can deal with these datasets on a single machine (using memory and multiple cores efficiently). Most data scientists are using R or Python because of their excellent coverage of data munging, visualization and modeling (e.g. machine learning), their high-level APIs and the environments for interactive/exploratory data analysis they provide. (Note: this author argues that R is a better choice between the two). There is still a heated debate though about what to do when your data or computation is beyond what these two tools can handle. In fact, both R and Python can easily handle datasets of 10 million rows for aggregates, joins and other common data munging (i.e. \u0026ldquo;small/simple analytics\u0026rdquo;) operations taking at most a few seconds (e.g. see this benchmark), therefore still supporting a nice interactive workflow. For machine learning (i.e. \u0026ldquo;big/complex analytics\u0026rdquo;) on such datasets, with respect to algorithms such as random forests for example, one can use H2O or xgboost right from within R or Python almost seamlessly. Overall, I cannot help to observe that non-linear machine learning on such data sizes looks more like a high-performance computing task rather than a \u0026ldquo;big data\u0026rdquo; one.\nEssentially, data scientists should ask themselves if their analytics tasks really need \u0026ldquo;big data\u0026rdquo; tools before choosing tools that add extra complexity (sometimes along with efficiency or reliability issues) and that are still largely missing the multitude of statistical and data manipulation libraries and high-level APIs that make using R or Python so productive. Without taking the time to understand the tradeoffs and make an informed decision, many will spend a large part of their time fighting the tools instead of gaining value and insights from their data.\n","date":"May 19, 2015","externalUrl":null,"permalink":"/benchmarking-random-forest-implementations/","section":"Posts","summary":"I currently have the need for machine learning tools that can deal with observations of the order of 10 millions in the context of binary classification. That kind of data is a few GBs in size and it fits comfortably nowadays in the RAM of a decent single machine. It is a trivial task for linear models, as there are plenty of open source tools that can train a logistic regression with this amount of data on a single machine in a few seconds, even while using only 1 processor core (many of these tools are single-threaded). Linear models are also the gold standard of large-scale machine learning that can run on clusters, processing very large distributed datasets.\n","title":"Benchmarking Random Forest Implementations","type":"posts"},{"content":"","date":"May 19, 2015","externalUrl":null,"permalink":"/tag/random-forests/","section":"Tags","summary":"","title":"random forests","type":"tags"},{"content":"","date":"May 19, 2015","externalUrl":null,"permalink":"/tag/scikit-learn/","section":"Tags","summary":"","title":"scikit-learn","type":"tags"},{"content":"*** EVENTBRITE LINK HERE ***\nPyDSLA - Dato: A Platform for Practical, Large-Scale Machine Learning\nJoin PyDSLA for a talk by Carlos Guestrin about Presents Practical, Large-Scale Machine Learning in python.\nSummary:\nData Science is perhaps the hottest profession on the market today. Folks with backgrounds ranging from Statistics and Physics to Engineering and Computer Science are eager and excited to transition to this field. However, designing and deploying data analysis and machine learning apps at scale is a significant challenge to overcome: For some folks, machine learning algorithms and methods can be obscure, too mathy, and disconnected from practice. For others, writing and deploying scalable software requires significant effort, which distracts them from focusing on their deep analytic efforts.\nIn this talk, I will cover a key set of techniques for data engineering, modeling, and deployment, which can help getting started in large-scale machine learning problems. I will demo these techniques using GraphLab Create, and real-world, large-scale use cases.\nBio:\nCarlos is the CEO and co-founder of Dato (formerly GraphLab, Inc.) and the Amazon Professor of Machine Learning in Computer Science \u0026amp; Engineering at the University of Washington. A world-recognized leader in the field of Machine Learning, Carlos was named one of the 2008 “Brilliant 10″ by Popular Science Magazine, received the 2009 IJCAI Computers and Thought Award for his contributions to Artificial Intelligence, and a Presidential Early Career Award for Scientists and Engineers (PECASE).\nDate: May 14, 2015 (Thursday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:30pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite, as space is limited due to the capacity of the room. Make sure to RSVP now!\nVenue: Venice Arts, 1702 Lincoln Blvd, Venice, CA 90291\n","date":"May 5, 2015","externalUrl":null,"permalink":"/pydsla-dato-ceo-carlos-guestrin-presents-dato/","section":"Posts","summary":"*** EVENTBRITE LINK HERE ***\nPyDSLA - Dato: A Platform for Practical, Large-Scale Machine Learning\nJoin PyDSLA for a talk by Carlos Guestrin about Presents Practical, Large-Scale Machine Learning in python.\nSummary:\nData Science is perhaps the hottest profession on the market today. Folks with backgrounds ranging from Statistics and Physics to Engineering and Computer Science are eager and excited to transition to this field. However, designing and deploying data analysis and machine learning apps at scale is a significant challenge to overcome: For some folks, machine learning algorithms and methods can be obscure, too mathy, and disconnected from practice. For others, writing and deploying scalable software requires significant effort, which distracts them from focusing on their deep analytic efforts.\n","title":"PyDSLA: Dato CEO, Carlos Guestrin, Presents Dato","type":"posts"},{"content":"","date":"March 30, 2015","externalUrl":null,"permalink":"/tag/anonymity/","section":"Tags","summary":"","title":"anonymity","type":"tags"},{"content":"Where were you on the evening of Monday, March 23rd? Do you remember what you were up to? If you were one of the over 100 attendees to the latest installment of Szilard’s Data Science Meetup, you definitely remember! You were at Cross Campus Santa Monica, receiving a teaser on fundamentals of probabilistic programming by LA ML Meetup founder Rob Zinkov along with a detailed master class on modern machine learning architectures and approaches by Los Angeles’ own Whisper!\nWhisper is an anonymous social network that helps people share confessions, express themselves, and meet new people. Whisper positions itself as valuing a strict adherence to the principles of anonymity and privacy - principles which are often at odds with the very drivers behind common applications of Data Science. Whisper is facing many of the problems seen by others in data informed businesses which are driving the forefront of technology. These span from customization and recommendation to fraud prevention and spam, but unlike almost any other business they are addressing them with almost no information about their users. Imagine, having to recommend and curate content for users you know almost nothing about… That’s what the team at Whisper tackles every single day.\nWhisper graciously brought 3 team members on stage, each one giving a thoughtful and deep presentation of the state of their current technology, their tools and approaches, as well as interesting discoveries they had made attempting to solve these traditional data science problems in their decidedly non-traditional privacy minded world.\nThere is no video of this presentation, so unfortunately if you missed the live event, you definitely missed out on both the talk and the spirited discussion afterwards. Fortunately the team at Whisper has been gracious enough to share their slides with us.\nUlas Bardak, Maarten Bosma, Rohan Monga - Data Science @Whisper - LA Data Science Meetup - March 2015 · Speaker Deck We at DSLA are very excited to have hosted Whisper and all of you at this event, and we hope that we’ll see you at a meetup soon! Thanks, as always, to our wonderful venue Cross Campus, and thanks to our wonderful sponsors for this meetup, Factual.\n","date":"March 30, 2015","externalUrl":null,"permalink":"/data-science-whisper/","section":"Posts","summary":"Where were you on the evening of Monday, March 23rd? Do you remember what you were up to? If you were one of the over 100 attendees to the latest installment of Szilard’s Data Science Meetup, you definitely remember! You were at Cross Campus Santa Monica, receiving a teaser on fundamentals of probabilistic programming by LA ML Meetup founder Rob Zinkov along with a detailed master class on modern machine learning architectures and approaches by Los Angeles’ own Whisper!\n","title":"Data Science @ Whisper","type":"posts"},{"content":"","date":"March 30, 2015","externalUrl":null,"permalink":"/tag/whisper/","section":"Tags","summary":"","title":"whisper","type":"tags"},{"content":" What is DataFest? # DataFest is a team competition, where a team of up to 5 undergraduate students has less than 48 hours to provide the best insight gleaned from a large, complex data set \u0026mdash; and communicate that insight to an audience of their peers and the DataFest judges. DataFest 2015 will be April 24th through the 26th, beginning at 6pm on Friday the 24th.\nWho Can Compete? # Any undergraduate at UCLA, or other local undergraduates from colleges or universities with agreements with UCLA Statistics. Note that all members of the team must be undergraduates and teams can have no more than 5 people. To sign up, visit the official DataFest website.\nWhy Attend DataFest if You\u0026rsquo;re Not a Student? # I was fortunate enough to visit UCLA DataFest 2014 as an “industry visitor” and speak with students about their approaches and the data. The students are fantastically excited to have industry professionals to bounce ideas and approaches off of. It was not only suprisingly rewarding, but also quite enlightening to see the tools and technologies being used by the next generation of up-and-coming data scientists.\n","date":"March 4, 2015","externalUrl":null,"permalink":"/asa-datafest-2015-at-ucla/","section":"Posts","summary":"What is DataFest? # DataFest is a team competition, where a team of up to 5 undergraduate students has less than 48 hours to provide the best insight gleaned from a large, complex data set — and communicate that insight to an audience of their peers and the DataFest judges. DataFest 2015 will be April 24th through the 26th, beginning at 6pm on Friday the 24th.\n","title":"ASA DataFest 2015 at UCLA","type":"posts"},{"content":"","date":"March 4, 2015","externalUrl":null,"permalink":"/tag/data/","section":"Tags","summary":"","title":"data","type":"tags"},{"content":"This meetup will be hosted by Google and we\u0026rsquo;ll have Peter Lipman and Pete Meyer talk about the data science/statistics projects they have been working on at Google.\nHuman Evaluation Comparisons: Common Framework with Applications\nWe designed a study to determine whether being on device influenced a quality evaluator’s perception of ads quality. Here, we present a common framework, developed into an internal R package, that was used to analyze the study.\nBio: Peter Lipman received his Ph.D in Biostatistics from Harvard University in 2011. He joined Google in 2013 after working in the pharmaceutical industry as a clinical trial statistician in early-stage oncology research. Peter works in the LAX office on the Ads Human Evaluation team, helping internal clients measure the quality of ads in the Google network using human computation.\nTracking Changes in Evaluators\u0026rsquo; Weekly Work Schedules\nThe quality evaluators who provide ads quality evaluations determine their own schedules, with the downside that they may sometimes login and find no work available. We investigated using messaging to help them adjust their schedules to ensure work was available. We describe using circular earthmover distributions for identifying and measuring changes in weekly work schedules along with some negative binomial models for comparing the number of ratings done by groups of raters during specific blocks of time.\nBio: Pete Meyer received his Ph.D. in Statistics from the University of Chicago in 1993. He joined Google after 14 years as a biostatistician in the Dept. of Preventive Medicine at Rush University Medical Center in Chicago. He has worked on a variety of Google ads quality teams over the last 7 years and is currently on the Ads Human Evaluation team in the Google L.A. office.\nDate: March 10, 2015 (Tuesday)\nTimeline:\n6:15pm food/bev \u0026amp; networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite. You must have an RSVP and ID with names matching as Google security will be checking IDs at the gate.\nNo more free spots? To increase your chance to get in next time, consider signing up to the DataScience.LA mailing list (we announce all new meetups on that mailing list first).\nVenue: Google, 340 Main Street, Venice, CA 90291\n","date":"February 27, 2015","externalUrl":null,"permalink":"/data-sciencestatisticsr-google/","section":"Posts","summary":"This meetup will be hosted by Google and we’ll have Peter Lipman and Pete Meyer talk about the data science/statistics projects they have been working on at Google.\nHuman Evaluation Comparisons: Common Framework with Applications\nWe designed a study to determine whether being on device influenced a quality evaluator’s perception of ads quality. Here, we present a common framework, developed into an internal R package, that was used to analyze the study.\n","title":"Data Science/Statistics/R @Google","type":"posts"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tag/data-warehousing/","section":"Tags","summary":"","title":"data warehousing","type":"tags"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tag/datavis-2/","section":"Tags","summary":"","title":"datavis","type":"tags"},{"content":"The LA Data Warehouse, Business Analytics and Analytics meetup had a fantastic first event with a great audience (full house), an awesome host (thanks Edmunds: Punnoose for taking the lead and the Team onsite and helping) and a great mix of 3 talks overviewing the field.\nDespite the hype of data science and big data, a majority of the business value produced by data still happens in this more traditional DW/BI setting. On the other hand, DW/BI/Analytics is getting more fancy: columnar/MPP databases, modern HTML5 web-based BI/visualization software, more reliance on big data systems such as Hadoop/Spark, powerful GUI products for predictive analytics etc.\nOur first speaker, Szilard Pafka spoke about why he and his DataScience.LA partner Eduardo started this new meetup group and what can be expected ahead (format, topics to be covered). His slides are here:\nSzilard Pafka - LA DW/BI/Analytics Meetup Kick-off - Febr 2015 · Speaker Deck and video here:\nThis video is no longer available on YouTube (original link).\nSecond, Eduardo Arino de la Rubia discussed a project at his employer where they attempted to find the big data strategy that worked for their reasonably large, but very slow moving data. His findings were summarized on these slides:\nEduardo Arino de la Rubia - Big Data is not Hadoop - LA DW/BI/Analytics Meetup - Febr 2015 · Speaker Deck while the video of his talk is here:\nThis video is no longer available on YouTube (original link).\nFinally, Eric Klusman made an overview of the major platforms for business intelligence and analytics, slide here:\nEric Klusman - The BI software market - LA DW/BI/Analytics Meetup - Febr 2015 · Speaker Deck and video here:\nThis video is no longer available on YouTube (original link).\nIn summary it was an awesome start for a new meetup group and we hope to see you at our next event!\nYour Co-Organizers,\nSzilard \u0026amp; Eduardo\nP.S. - Here are some pictures from the event:\nwith some more pictures on our flickr account here.\n","date":"February 15, 2015","externalUrl":null,"permalink":"/dwbianalytics-meetup-summary-inaugural-event/","section":"Posts","summary":"The LA Data Warehouse, Business Analytics and Analytics meetup had a fantastic first event with a great audience (full house), an awesome host (thanks Edmunds: Punnoose for taking the lead and the Team onsite and helping) and a great mix of 3 talks overviewing the field.\n","title":"DW/BI/Analytics Meetup Summary: Inaugural Event","type":"posts"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tag/mpp-database/","section":"Tags","summary":"","title":"MPP database","type":"tags"},{"content":"","date":"February 15, 2015","externalUrl":null,"permalink":"/tag/visualization/","section":"Tags","summary":"","title":"visualization","type":"tags"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/htmlwidgets/","section":"Tags","summary":"","title":"htmlwidgets","type":"tags"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/knitr/","section":"Tags","summary":"","title":"knitR","type":"tags"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/shiny/","section":"Tags","summary":"","title":"shiny","type":"tags"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/yihui-xie/","section":"Tags","summary":"","title":"yihui xie","type":"tags"},{"content":"The Shiny framework, by RStudio, is growing in popularity as a tool that makes it easy for R developers to leverage the web as a deployment platform for sharing their analysis. However, since the very earliest release of Shiny, the development group has been bombarded with the same series questions over and over again, questions like “Does it work with d3.js?” and “Can I use JavaScript in Shiny?”\nShiny has never stopped you from doing either of these two things. On the other hand, it hasn’t made it particularly easy either. With this in mind, Yihui Xie (of knitR fame) and the RStudio team present the HTMLWidgets framework! HTMLWidgets brings the best of JavaScript visualization to R. You can, using modern RStudio builds, use HTMLWidgets right in the RStudio environment and interact with the plotting pane as if it was a modern browser. You can embed HTML Widgets in Shiny applications and RMarkdown documents. If you are an R developer with some JavaScript experience, you can even develop new widgets using the seamless R/JavaScript bridge provided by the HTMLWidgets package!\nFrom geospatial mapping to time series visualization, from d3.js interactivity to beautiful interactive tabular display of data, the HTMLWidgets framework provides a foundation for that next level of interactivity and fluency in your interfaces. In this video, Yihui covers a foundation and a rationale for the HTMLWidgets system, and gets you well on your way to contributing to this exciting new ecosystem. You can also find his slides here. Enjoy!\n","date":"February 13, 2015","externalUrl":null,"permalink":"/yihui-xie-presents-html-widgets/","section":"Posts","summary":"The Shiny framework, by RStudio, is growing in popularity as a tool that makes it easy for R developers to leverage the web as a deployment platform for sharing their analysis. However, since the very earliest release of Shiny, the development group has been bombarded with the same series questions over and over again, questions like “Does it work with d3.js?” and “Can I use JavaScript in Shiny?”\n","title":"Yihui Xie presents HTML Widgets","type":"posts"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/joe-cheng/","section":"Tags","summary":"","title":"joe cheng","type":"tags"},{"content":"The Shiny framework is a web application framework for the R programming language, designed by Joe Cheng, to help R programmers turn their analysis into interactive web applications. Fortunately, it does without requiring that the R programmer know HTML, CSS or JavaScript - greatly widening the possible audience of users. At the January DataScience.LA R Meetup, we were fortunate to have Joe Cheng, Software Engineer at RStudio and original architect of Shiny present an introduction to Shiny.\nHis presentation begins with a description of Shiny and why you should be excited about this tool. Joe quickly moves on to a number of great examples and some of the rationale behind the underlying “reactive” Shiny framework. Near the end of the talk, Joe even covers how to handle streaming real time updates with Shiny! This was a great talk, and whether you are just getting started in Shiny or have built a number of great apps already in production, there’s almost certainly something in this video for everyone. If you\u0026rsquo;re not into videos, check out the slides below:\nShiny - January 2015 LA R Users Group · Speaker Deck ","date":"February 13, 2015","externalUrl":null,"permalink":"/joe-cheng-presents-shiny/","section":"Posts","summary":"The Shiny framework is a web application framework for the R programming language, designed by Joe Cheng, to help R programmers turn their analysis into interactive web applications. Fortunately, it does without requiring that the R programmer know HTML, CSS or JavaScript - greatly widening the possible audience of users. At the January DataScience.LA R Meetup, we were fortunate to have Joe Cheng, Software Engineer at RStudio and original architect of Shiny present an introduction to Shiny.\n","title":"Joe Cheng presents Shiny","type":"posts"},{"content":"","date":"February 13, 2015","externalUrl":null,"permalink":"/tag/web/","section":"Tags","summary":"","title":"web","type":"tags"},{"content":"","date":"February 7, 2015","externalUrl":null,"permalink":"/tag/blaze/","section":"Tags","summary":"","title":"blaze","type":"tags"},{"content":"","date":"February 7, 2015","externalUrl":null,"permalink":"/tag/pandas/","section":"Tags","summary":"","title":"pandas","type":"tags"},{"content":"Join PyDSLA for a talk by Matthew Rocklin about the high level APIs being developed for Python for expressive data manipulation and analysis.\nSummary:\nPandas and NumPy serve us well. They give interactive high-level access to efficient low-level code. As datasets grow larger than memory this low-level code becomes obsolete but the high-level interface remains valuable. Blaze connects that high-level interface to a number of other computational systems like SQL, Mongo, or Spark, extending the scope of interactive analysis. In this talk we go over common use cases and Blaze as well as a bit of its internal design.\nBio:\nMatthew works for Continuum Analytics as a computational scientist. Before joining he was an academic in scientific computing (PhD University of Chicago) and an active open source contributor in the PyData ecosystem (SymPy, PyToolz, Theano.) Today he lives in lovely Oakland, CA and works to improve the usability of big-data technologies through Blaze and other projects.\nDate:\nFebruary 23, 2014 (Monday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:30pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite, as space is limited due to the capacity of the room. Make sure to RSVP now!\nVenue:\nVenice Arts, 1702 Lincoln Blvd, Venice, CA 90291\nThanks OpenMail for hosting!\n","date":"February 7, 2015","externalUrl":null,"permalink":"/pydsla-blaze-connecting-analysts-to-big-computation/","section":"Posts","summary":"Join PyDSLA for a talk by Matthew Rocklin about the high level APIs being developed for Python for expressive data manipulation and analysis.\nSummary:\nPandas and NumPy serve us well. They give interactive high-level access to efficient low-level code. As datasets grow larger than memory this low-level code becomes obsolete but the high-level interface remains valuable. Blaze connects that high-level interface to a number of other computational systems like SQL, Mongo, or Spark, extending the scope of interactive analysis. In this talk we go over common use cases and Blaze as well as a bit of its internal design.\n","title":"PyDSLA - Blaze: Connecting analysts to big computation","type":"posts"},{"content":"","date":"February 6, 2015","externalUrl":null,"permalink":"/authors/andrew-parker/","section":"Authors","summary":"","title":"Andrew Parker","type":"authors"},{"content":"","date":"February 6, 2015","externalUrl":null,"permalink":"/tag/hiring/","section":"Tags","summary":"","title":"hiring","type":"tags"},{"content":"","date":"February 6, 2015","externalUrl":null,"permalink":"/category/random/","section":"Categories","summary":"","title":"Random","type":"categories"},{"content":"This was originally posted on the author\u0026rsquo;s LinkedIn\nYou will find lots of advice on how to get a job as a data scientist. I want to add some signal to the noise.\nI happen to be hiring for a Data Scientist or two as of this writing, and was thinking of doing a blog post on offering advice for those looking. But then I thought of something better and more insightful: It is a cut-and-paste of an email that I sent to our Director of Talent Acquisition last week when he asked me, \u0026ldquo;Can you share with me what you\u0026rsquo;re looking for in regards to your Data Scientist candidates?\u0026rdquo;\nHere it is:\nDate: Mon, 26 Jan 2015 14:39:10 -0800\nDelivered-To: XXXX@ziprecruiter.com\nSubject: Re: Sorry for the mix up regarding Data Science candidates\nFrom: Andrew Parker \u0026lt;XXXX@ziprecruiter.com\u0026gt;\nTo: Ron Miguel \u0026lt;XXXX@ziprecruiter.com\u0026gt;\nCc: Nicole Prause \u0026lt;XXXX@ziprecruiter.com\u0026gt;\n\u0026ndash;001a113a9c2e9685c2050d95cea1\nContent-Type: text/plain; charset=UTF-8\nHi Ron,\nThanks for all your help on this. Nikky, please add your thoughts to this\nthread.\nIn general, for this Data Science position, the question is: Can this\nperson teach us something, and do they bring to the table some core\ncompetency that is missing from the team? Are they senior enough to teach\nothers when we hire more junior people in the near future?\n*If they are new PhDs:*\nThe thinking is that they\u0026rsquo;re probably smart, and really it\u0026rsquo;s a question of\nseeing if their interests and expectations are aligned with the actual\nprojects we have to work on, and to gauge their willingness to learn the\nproblem domain (SaaS business issues: LTV, Churn / Retention, etc.) and\ntools. Seeing extensive use of Python and R is most important. I want to\nget a sense that they were the \u0026ldquo;hacker\u0026rdquo; in their lab.\nThe nature of the data matters: specifically I like to see them deal with\nmessy, noisy, incomplete, unstructured, human inputted or driven data.\nExamples are NLP and text processing, linguistic modeling, survey and\nclinical data, economic and sociology data, web scraped (or API) data from\nTwitter, facebook, etc. Less ideal is data that is highly structured, and\nprecise or all theoretical.\n*If they are new Masters or BS graduates:*\nMany of these resumes are over-inflated and key-word stuffed, and I really\nhave to read between the lines to figure out what they\u0026rsquo;ve actually done. I\nfind these the most difficult to read.\n*If they have industry experience:*\nAnything applying analytical and data processing skills to problems\nrelevant to ZipRecruiter where they were the lead developer is what we\u0026rsquo;re\nlooking for. They have to be able to deal with raw and mostly unstructured\ndata. E.g. \u0026ldquo;Go look in this Apache access log file for what you need.\u0026rdquo; I am\nyet unable to accommodate candidates that are accustomed to an IT team\ndoing all the cleaning and curation of data.\n*On-site:*\nI\u0026rsquo;d like to have them present something and do a deep-dive on the data,\nanalysis, code, and insights. I think recent graduates and academics will\nhave a huge advantage since they are likely working with data and code they\ncan share publicly. So the on-site visit will be different from the usual\nsoftware engineering visit. Maybe for those that don\u0026rsquo;t have data handy that\nthey can talk about, Nikky and I will have to identify some data for them\nto do a sort of \u0026ldquo;take home project.\u0026rdquo; We haven\u0026rsquo;t gotten to that point yet\nthough.\n*Candidate work-flow:*\nWe\u0026rsquo;ll wait for your process guidelines, but in the mean time we\u0026rsquo;ll score\ncandidates and make more detailed internal notes.\nThanks!\n-Andrew\nAnd that\u0026rsquo;s it.\nKeep in mind that this is for a senior position and describes my team\u0026rsquo;s particular needs at this point in time. I sent this email well before I contemplated writing a blog post. I hope some of you find this helpful.\nIf you\u0026rsquo;re curious, we still have that open Data Scientist position here:\nhttps://www.ziprecruiter.com/jobs/ziprecruiter-bf882208/data-scientist-db7104a3\n","date":"February 6, 2015","externalUrl":null,"permalink":"/re-sorry-for-the-mix-up-regarding-data-science-candidates/","section":"Posts","summary":"This was originally posted on the author’s LinkedIn\nYou will find lots of advice on how to get a job as a data scientist. I want to add some signal to the noise.\nI happen to be hiring for a Data Scientist or two as of this writing, and was thinking of doing a blog post on offering advice for those looking. But then I thought of something better and more insightful: It is a cut-and-paste of an email that I sent to our Director of Talent Acquisition last week when he asked me, “Can you share with me what you’re looking for in regards to your Data Scientist candidates?”\n","title":"Re: Sorry for the mix up regarding Data Science candidates","type":"posts"},{"content":"","date":"January 26, 2015","externalUrl":null,"permalink":"/tag/advertising/","section":"Tags","summary":"","title":"advertising","type":"tags"},{"content":"","date":"January 26, 2015","externalUrl":null,"permalink":"/tag/datapop/","section":"Tags","summary":"","title":"datapop","type":"tags"},{"content":" Ben Zamanzadeh - Data Science @ Datapop - LA Data Science meetup - January 2015 · Speaker Deck On January 13th, DataScience.LA held its first Data Science meetup of 2015, hosting a speaker from a local semantic advertising startup, DataPop. Their CTO, Dr. Ben Zamanzadeh, gave an in-depth presentation (linked above) of his perspective on e-commerce and advertising, which focused on extracting semantic meaning and DataPop’s marketing knowledge graph. He began his talk by discussing the evolution of e-commerce and moved quickly to semantics, making some very strong statements regarding DataPop’s architectural choices. For example, even though they provide a semantic graph, they are based on Redshift and do not use graph databases - which was quite the surprise to us here at DataScience.LA! At one point, Dr. Zamanzadeh even commented,\n\u0026ldquo;Nobody that has an industrial-strength knowledge graph uses RDF.\u0026rdquo;\nHe then went on to describe how DataPop has built an ontology of sales strategies, and provided great insight on how semantics can help you differentiate these strategies if you are marketing on the uniqueness of your product or the attractiveness of the overall brand.\nDr. Zamanzadeh also gave a tour of things that weren\u0026rsquo;t as effective. Machine generated ads, he claims, won’t work with current technologies. They’ve tried it, and machine learning is currently incapable of outperforming human creativity in this space. He also discussed challenges in building their data set, the realization that this same dataset was biased, and how they addressed these issues.\nIt was an enlightening and incredibly well attended event, with well over 200 people in attendance. This meetup was also summarized by a few different posts, with the most notable one by DataPop’s own Alissa Volosin, posted on DataPop’s blog. Another attendee, Data Science community member Phyllis Zimbler Miller, tweeted her response, with a link to her writeup on linkedin.\nCheck out the pictures from this event on our flickr page. We got some really great pictures of the crowd, and it really drives the point home - this was a huge meetup with really fantastic turnout!\nOn another note, we would like to continue to encourage the #dsla community to tweet more during events! We want to spread the word about the great content provided at these meetups and the diversity of perspectives in our community. Next time you’re at a meetup, go ahead and tweet your thoughts with the hashtag #dsla. Happy Tweeting!\n","date":"January 26, 2015","externalUrl":null,"permalink":"/datascience-la-meetup-summary-data-science-datapop/","section":"Posts","summary":" Ben Zamanzadeh - Data Science @ Datapop - LA Data Science meetup - January 2015 · Speaker Deck On January 13th, DataScience.LA held its first Data Science meetup of 2015, hosting a speaker from a local semantic advertising startup, DataPop. Their CTO, Dr. Ben Zamanzadeh, gave an in-depth presentation (linked above) of his perspective on e-commerce and advertising, which focused on extracting semantic meaning and DataPop’s marketing knowledge graph. He began his talk by discussing the evolution of e-commerce and moved quickly to semantics, making some very strong statements regarding DataPop’s architectural choices. For example, even though they provide a semantic graph, they are based on Redshift and do not use graph databases - which was quite the surprise to us here at DataScience.LA! At one point, Dr. Zamanzadeh even commented,\n","title":"DataScience LA Meetup Summary: Data Science @ DataPop","type":"posts"},{"content":"","date":"January 26, 2015","externalUrl":null,"permalink":"/tag/semantics/","section":"Tags","summary":"","title":"semantics","type":"tags"},{"content":"We are pleased to announce the first meetup of the LA Data Warehouse, Business Intelligence and Analytics meetup group! On February 9th, we have 4 speakers scheduled to give an engaging overview along with a few of their insights into the field.\nDespite the hype of data science and big data, a majority of the business value produced by data still happens in the more traditional setting of business intelligence and analytics. On the other hand, DW/BI/Analytics tools are getting more fancy: columnar/MPP databases, modern HTML5 web-based BI/visualization software, and machine learning GUI products wrapping the newest algorithms like deep learning. There is a lot of neat stuff to explore and we’re excited to get started on February 9th!\nYou can RSVP here for this Meetup on Eventbrite (but you’ll want to keep on reading the info below).\nWe’d like to thank Edmunds for hosting this upcoming meetup and providing pizza \u0026amp; beverages. (Edmunds is a fantastic venue, as you’ll see!)\nAt the meetup we’ll raffle one free 2-day pass to the upcoming Strata + Hadoop World conference (February 18-20) - a $1800 value! (In case you are wondering if it’s a good conference, Szilard and Eduardo are attending again this year.)\nTo participate in the raffle, all you need is to drop your business card into the designated bowl during the meetup’s networking hour and we’ll raffle off the pass at around 7:00pm. (Disclosure: you consent that a copy of your card might be sent to O’Reilly)\nAlso, if you don’t win the raffle and are planning to go to Strata, you’re welcome to use our UGLA20 code to get a 20% discount!\nFinally and most importantly, please sign up for the meetup group here and also RSVP to the first meetup on Eventbrite here.\nTalks:\n1. Szilard Pafka: Intro, motivation, goals for this meetup group\nI will talk about why we have started a new meetup group and the various topics we\u0026rsquo;d like to cover in the next meetings. At the advent (and admittedly hype) of data science and big data, the traditional field of DW/BI/Analytics is experiencing an accelerated modernization. It\u0026rsquo;s these modern tools like MPP databases, html5 web-based BI solutions, powerful GUIs for predictive modeling etc. that we\u0026rsquo;d like to focus at our meetups.\nBio: Szilard has been working passionately with data for the past 20 years first in computational physics, then in finance and finally in the last decade as a data scientist. He is currently the Chief Scientist at Epoch and the founder/organizer of several popular data science meetups in LA (R, DataVis, Data Science/Machine Learning, Python Data Science and most recently DW/BI/Analytics).\n2. Eduardo Arino de la Rubia: All Big Data Stories are not Hadoop Stories\nIn this talk, I will discuss a project at my employer where we attempted to find the big data strategy that worked for our reasonably big, but very slow moving data. We will discuss the pitfalls of various solutions, and finish with our findings and go forward strategy. In a world dominated by marketing hype, this talk cuts right through it all and focuses on the outcomes.\nBio: I am a computer programmer and technologist with over 30 years experience, working in many different fields ranging from expert systems for compliance to manufacturing robotics. I am a graduate of Middle Tennessee State University and live with my wife, son, and 3 dogs in lovely Santa Monica.\n3. Adam Husein: RDBMS Usage from the Data Scientist\u0026rsquo;s Perspective\nAs a prolific user of databases with extensive experience across various database technologies, Adam will speak about how analysts and data scientists utilize data and the databases that store them.\nBio: Over the last 7 years, Adam has worked with or on over a half dozen database technologies across various companies in order to set up \u0026amp; deploy databases, create new data models, and provide analytics to internal customers.\n4. Eric Klusman: Overview of the Business Intelligence Landscape\nThis presentation will discuss the major platforms for business intelligence and analytics, including the roles played by open source software. Major vendors will be introduced and contrasted.\nBio: Eric Klusman is a data scientist with Kabam\u0026rsquo;s Los Angeles game studio. He has held a variety of positions in data engineering and data science with Los Angeles Internet companies and Wall Street securities firms.\n","date":"January 22, 2015","externalUrl":null,"permalink":"/announcing-first-la-data-warehouse-business-intelligence-and-analytics-meetup/","section":"Posts","summary":"We are pleased to announce the first meetup of the LA Data Warehouse, Business Intelligence and Analytics meetup group! On February 9th, we have 4 speakers scheduled to give an engaging overview along with a few of their insights into the field.\n","title":"Announcing First LA Data Warehouse, Business Intelligence and Analytics Meetup!","type":"posts"},{"content":"Shiny takes inputs from UI elements and sends them to the server, where the application can access them as R variables. While Shiny has security measures in place, as in any typical web application, it remains the developer\u0026rsquo;s responsibility to sanitize the inputs before using them. For example, Shiny has no way to protect you if you are using an input in a SQL query such as select ... from ... where field = 'input'. Someone manipulating the websocket communication can craft a specially-formatted input that can force the database to execute a query that it is not supposed to do, termed an SQL injection. This might give an attacker access to private data or the ability to do other nefarious things, and it is a common security issue.\nTo see this in action, take a look at this Shiny app demo. The input is read from a dropdown menu and it is used on the server side to construct a SQL query\nabbr \u0026lt;- input$inp_abbr dbGetQuery(con, paste0(\u0026#34;select * from states where abbr = \u0026#39;\u0026#34;,abbr,\u0026#34;\u0026#39;\u0026#34;)) Source: gist ef1d145 by szilard\nEven if the input is read from a list of predetermined values (such as a dropdown) it is quite easy for someone accessing the app to send arbitrary input. For example, using Chrome browser\u0026rsquo;s Developer Tools\u0026rsquo; Network panel one can see that for each UI update the following data is sent: {\u0026quot;method\u0026quot;:\u0026quot;update\u0026quot;,\u0026quot;data\u0026quot;:{\u0026quot;inp_abbr\u0026quot;:\u0026quot;CA\u0026quot;}}. You can now use the Developer Tools\u0026rsquo; Javascript Console to send ' or 1=1 -- instead of CA, and then the SQL query constructed by the code above becomes select * from states where abbr = '' or 1=1 --'. The app will then run this query and return all the values from that table, which is very different from the original intent of the app. With these kinds of simple tricks someone could read data from other tables as well, and in certain cases even alter or delete data.\nIt\u0026rsquo;s important to note that this is not a deficiency of Shiny per se; it is instead a problem in any web framework - and it is the developer\u0026rsquo;s responsibility to check the inputs and act accordingly. If you are developing Shiny apps in a corporate environment while accessing data from a SQL database, you should definitely pay attention to these possibilities. Luckily there are several ways to deal with this problem, a few of which I describe below.\nOne way is to check the inputs and if they don\u0026rsquo;t conform with certain patterns, do not run the query and give an error to the user. You could also use Shiny\u0026rsquo;s validate function to have a more user friendly message if you prefer. It is usually safer to specify which patterns are allowed (i.e. whitelist) such as \u0026ldquo;only alphanumeric characters, dot, space and hyphen\u0026rdquo; rather then specifying those characters not allowed (i.e. blacklist) such as \u0026ldquo;single quote, backslash etc\u0026rdquo;. I wrote a very basic R package that provides functions for validating numeric, character, logical and Date values for the various Shiny UI elements. You can find it on github and use it by wrapping your inputs like this, and note that you can use regular expressions for specifying the pattern.\nabbr \u0026lt;- validinp_character(input$inp_abbr) ## or: abbr \u0026lt;- validinp_character(input$inp_abbr, pattern=\u0026#34;^((CA)|(NY))$\u0026#34;) Source: gist 86c14fe by szilard\nIt is also always recommended to restrict the input as much as possible (e.g. to a set of known values as I described in the second option above).\nAnother approach to secure web applications is to sanitize the inputs, or in other words replace non-alphanumeric characters (like single quotes) with \u0026ldquo;escaped\u0026rdquo; ones before continuing with the query. For example, if you are using MySQL via the RMySQL package you can use the mysqlEscapeStrings function to achieve this. Unfortunately there are numerous edge cases that must be considered during implementation and this sanitizing approach is usually considered error prone. On the other hand, if you are using the excellent dplyr package you are provided basic safeguards by escaping automatically. This is primarily meant to prevent accidental misuse rather than an explicit attack, however. As Hadley says in the dplyr docs, \u0026ldquo;This is unlikely to prevent any serious attack, but should make it unlikely that you produce invalid SQL.\u0026rdquo;\nPerhaps the most secure approach would be to use prepared statements, also called parameterized queries. Unfortunately the RMySQL package does not currently support this feature and the support in the other R database connectors is also somewhat in its infancy.\nRegardless of how you address this issue, you should also have additional layers of security such as granting minimal privileges to the database user used by the application. For a Shiny app, this means you might provide read-only access to a few tables. In addition, you should also restrict the Shiny app to a limited set of users whenever possible - either by using Shiny Server Pro, ShinyApps.io or by proxying via a web server and using its authentication features. You should also encrypt the traffic (https) if it is not on a private network. Furthermore, other than running Shiny as an unprivileged user (which is the default), you could also run the app in an isolated environment and/or use the RAppArmor package for extra security.\nFinally, while in this blog post I have focused upon SQL injection, validating the inputs is crucial in several other situations. With R being a \u0026lsquo;hyper-dynamic\u0026rsquo; language, there are countless functions in base R and common packages that do eval-like things, such as when data becomes code and it is executed. While a blacklisting approach seems hopeless to me in this case, the sandboxR project has attempted to map R\u0026rsquo;s \u0026lsquo;unsafe\u0026rsquo; functions - and the long blacklist therein should horrify you with how many ways things can go wrong. Therefore, you must think carefully and thoroughly check your inputs if you are developing a Shiny app (or any web application, for that matter). Be safe!\n","date":"January 12, 2015","externalUrl":null,"permalink":"/secure-your-shiny-apps-against-sql-injection/","section":"Posts","summary":"Shiny takes inputs from UI elements and sends them to the server, where the application can access them as R variables. While Shiny has security measures in place, as in any typical web application, it remains the developer’s responsibility to sanitize the inputs before using them. For example, Shiny has no way to protect you if you are using an input in a SQL query such as select ... from ... where field = 'input'. Someone manipulating the websocket communication can craft a specially-formatted input that can force the database to execute a query that it is not supposed to do, termed an SQL injection. This might give an attacker access to private data or the ability to do other nefarious things, and it is a common security issue.\n","title":"Secure your Shiny apps (against SQL injection)","type":"posts"},{"content":"","date":"January 12, 2015","externalUrl":null,"permalink":"/tag/security/","section":"Tags","summary":"","title":"security","type":"tags"},{"content":"","date":"January 12, 2015","externalUrl":null,"permalink":"/tag/sql/","section":"Tags","summary":"","title":"SQL","type":"tags"},{"content":"(Posting the slides and video recording from this event is long overdue - our apologies for the delay.)\nAt the September LA Data Science/Machine Learning meetup we had IRIS.TV\u0026rsquo;s Chief Data Scientist Tom Sullivan and VP of Engineering Joel Spitalnik presenting.\nIRIS.TV is a video recommendation engine, and in their excellent talk Tom \u0026amp; Joel shared with us their approach to personalization and how they use machine learning to achieve their goals. They also shared the architecture of their systems, including how data is flowing between the various parts (i.e. the ETL pipeline). One thing that popped up as a question and I found interesting is that the usage of R and Python in their company is split 50-50. In fact, while Tom is using mainly R, Joel is using mainly Python. While this is perhaps unusual at a relatively small startup, our survey on software tools used for data science revealed that most data scientists in LA (at least those attending the meetups) are using R or Python, with a roughly equal split/small advantage for R. It just happens that for obvious reasons (such as network effects) that one tool dominates in most companies.\nThe talk touched on several other issues, and you can view the slides here:\nTom Sullivan \u0026amp; Joel Spitalnik - Data Science @ IRIS.TV - LA Data Science meetup - Sep 2014 · Speaker Deck I\u0026rsquo;d like to thank Cross Campus for hosting and Factual for sponsoring and providing pizza. (I would also like to personally thank Aaron Crow for not only being a great software engineer and organizer of several meetups, but also for making sure we have great pizza).\nFinally, below are a few pictures from this meetup (Thanks, Jeff Weakley!) and more meetup pics are available in this flickr album. If you are interested, there are even more pictures from our previous meetups available in our flickr account here.\n","date":"January 8, 2015","externalUrl":null,"permalink":"/data-science-la-meetup-summary-data-science-iris-tv/","section":"Posts","summary":"(Posting the slides and video recording from this event is long overdue - our apologies for the delay.)\nAt the September LA Data Science/Machine Learning meetup we had IRIS.TV’s Chief Data Scientist Tom Sullivan and VP of Engineering Joel Spitalnik presenting.\nIRIS.TV is a video recommendation engine, and in their excellent talk Tom \u0026 Joel shared with us their approach to personalization and how they use machine learning to achieve their goals. They also shared the architecture of their systems, including how data is flowing between the various parts (i.e. the ETL pipeline). One thing that popped up as a question and I found interesting is that the usage of R and Python in their company is split 50-50. In fact, while Tom is using mainly R, Joel is using mainly Python. While this is perhaps unusual at a relatively small startup, our survey on software tools used for data science revealed that most data scientists in LA (at least those attending the meetups) are using R or Python, with a roughly equal split/small advantage for R. It just happens that for obvious reasons (such as network effects) that one tool dominates in most companies.\n","title":"Data Science LA Meetup Summary: Data Science @ IRIS.TV","type":"posts"},{"content":"","date":"January 8, 2015","externalUrl":null,"permalink":"/tag/pictures/","section":"Tags","summary":"","title":"pictures","type":"tags"},{"content":"","date":"December 30, 2014","externalUrl":null,"permalink":"/tag/bokeh/","section":"Tags","summary":"","title":"bokeh","type":"tags"},{"content":"","date":"December 30, 2014","externalUrl":null,"permalink":"/tag/d3-js/","section":"Tags","summary":"","title":"d3.js","type":"tags"},{"content":"","date":"December 30, 2014","externalUrl":null,"permalink":"/category/datavis/","section":"Categories","summary":"","title":"DataVis","type":"categories"},{"content":"It pains me to admit it, but even though I had visited their site, created an account, and played around with their tools, I didn’t really get the value proposition behind Plot.ly. I already use ggplot, bokeh, and d3.js, I already use knitr and ipython notebooks, so why do I need a new way of posting my plots on a web page?\nOn my own and before the meetup, it made no sense. Why was this company getting such positive press? How on earth could Plot.ly have attracted a customer list including some of the Who’s Who of scientific and research institutions? After the meetup presentation, which covered topics ranging from rich visualization to collaboration architectures, I spoke to the Plot.ly team, and I brought this up… how I had missed the value proposition completely, how their website did a poor job of explaining why I should care. Their response was great…\n“We hear this all the time, and we’ve ended up traveling around telling people what we do instead of clearly showing them on the site. Traveling to everyone’s meetups isn’t very scalable, so we have a website redesign in the works.”\nIt was a very funny, very human moment which conveyed so much about the truth of running a Data Science / Visualization startup in 2014. Our field is complicated, and sometimes effectively telling a story is hard - even if you’re in the business of helping people tell stories.\nFor me, the Plot.ly meetup was one of the most impactful and interesting meetups of 2014 for DataScience.LA, and it’s a shame that so many of our members missed it. Out of nearly 120 RSVPs, only 30 members made it in person (with nearly 10 still on a waiting list). Sure, it was rainy and cold and on the brink of the holiday season, but I guarantee you that every single person that missed it would have considered the trip worthwhile.\nThose in attendance had the privilege of speaking one-on-one with a Plot.ly team that is genuinely breaking new ground in the world of inter-language visualization collaboration. I learned that Plot.ly isn’t just about beautiful, interactive, web based graphics. Plot.ly is about collaboration across data science teams using disparate programming languages. I can build a graph in R using ggplot2, push it to Plot.ly, and my collaborator can download it for use in Python using matplotlib. The folks at Plot.ly have built a generalized vocabulary to describe data visualization across these tools, functioning as universal translators between the major programming languages used in Data Science.\nEnjoy the video of Plot.ly’s wonderful, educational, and insightful presentation. If you live in the area, make it a New Year’s resolution to make it to one of our DSLA meetups in 2015… and if you RSVP’d and won’t be able to make it, please go ahead and cancel. One day you may be the one on a meetup waiting list eagerly awaiting your opportunity to attend!\n","date":"December 30, 2014","externalUrl":null,"permalink":"/datavis-with-plot-ly-plotlygraphs-meetup-summary/","section":"Posts","summary":"It pains me to admit it, but even though I had visited their site, created an account, and played around with their tools, I didn’t really get the value proposition behind Plot.ly. I already use ggplot, bokeh, and d3.js, I already use knitr and ipython notebooks, so why do I need a new way of posting my plots on a web page?\n","title":"DataVis with Plot.ly (@plotlygraphs) - Meetup Summary","type":"posts"},{"content":"","date":"December 30, 2014","externalUrl":null,"permalink":"/tag/ggplot2/","section":"Tags","summary":"","title":"ggplot2","type":"tags"},{"content":"","date":"December 12, 2014","externalUrl":null,"permalink":"/tag/data-visualization/","section":"Tags","summary":"","title":"Data Visualization","type":"tags"},{"content":"","date":"December 12, 2014","externalUrl":null,"permalink":"/tag/matplotlib/","section":"Tags","summary":"","title":"matplotlib","type":"tags"},{"content":"Plotly is a web-based platform for making graphs and analyzing data. Plotly\u0026rsquo;s APIs and web app let teams work together online to make 2D, 3D, and streaming graphs using R, Python, MATLAB, Julia, SAS, and and Excel, all together, all in one platform. The goal is to support technical and non-technical collaboration at all levels. Plotly translates ggplot2, matplotlib, and MATLAB plots into interactive, web-based, shareable plots.\nSpeaker\u0026rsquo;s bio**:**Jack Parmer (CEO of Plotly) studied physics and organic electronics at Stanford University, then worked at various bay area cleantech start-ups before starting Plotly. He\u0026rsquo;s interested in Plotly\u0026rsquo;s ability to help data-driven teams solve complex problems with more collaborative workflows. He lives in Oakland, California.\nDate**:**December 16, 2014\nTimeline**:**\n6:00pm food/bev \u0026amp; networking 7:00pm talk start promptly You must have a confirmed RSVP and arrive between 6:00-6:55pm in order to pass security. Please RSVP here on Eventbrite.\n","date":"December 12, 2014","externalUrl":null,"permalink":"/meetup-datavis-with-plotly-on-december-16th/","section":"Posts","summary":"Plotly is a web-based platform for making graphs and analyzing data. Plotly’s APIs and web app let teams work together online to make 2D, 3D, and streaming graphs using R, Python, MATLAB, Julia, SAS, and and Excel, all together, all in one platform. The goal is to support technical and non-technical collaboration at all levels. Plotly translates ggplot2, matplotlib, and MATLAB plots into interactive, web-based, shareable plots.\n","title":"Meetup: DataVis with Plotly on December 16th","type":"posts"},{"content":"","date":"December 12, 2014","externalUrl":null,"permalink":"/tag/plotly/","section":"Tags","summary":"","title":"plotly","type":"tags"},{"content":"","date":"December 9, 2014","externalUrl":null,"permalink":"/authors/jeff-weakley/","section":"Authors","summary":"","title":"Jeff Weakley","type":"authors"},{"content":"It was a whole bunch of intense fun photographing so many of you at our Los Angeles User Groups Holiday Party last week! (Only junkies and programmers would think “User” is a good name for a holiday party.) But we do throw a helluva party, and nearly 80 of you were photographed for head shots. You guys are awesome, and we all really enjoyed it!\nBelow are two links to your fabulous photos on Flickr, for those who were photographed for head shots. So you don’t have to look through so many of other people’s pics to find yours, I’ve implemented a ‘Binary Search Algorithm’, you know - cut ‘em in half. If you were shot more towards the beginning of the party, you’re probably in Part 1, and if you were later in the evening, most likely your photos are in Part 2.\nThere something uniquely intimate about photographing someone. Even though we barely talked I feel like I got to know many of you in a way that words often get in the way of. I really enjoyed setting up a mini-set like I did, it was a fun experiment for me - and it was nice to have so many attractive and interesting models to photograph!\nI’m very proud of what we did together\u0026hellip; but as great as you all are, the photos aren’t yet perfect. If you have a friend who knows Photoshop, let them know that a little cropping and color correction could help some of the photos become a little more awesome. (To download a photo from Flickr, click on the photo you like and it will open in its own window. You’ll see an icon for “downloading” in the right hand corner.)\nHave a great holiday season!\n-Jeff\n","date":"December 9, 2014","externalUrl":null,"permalink":"/photos-from-our-la-user-groups-first-annual-holiday-party/","section":"Posts","summary":"It was a whole bunch of intense fun photographing so many of you at our Los Angeles User Groups Holiday Party last week! (Only junkies and programmers would think “User” is a good name for a holiday party.) But we do throw a helluva party, and nearly 80 of you were photographed for head shots. You guys are awesome, and we all really enjoyed it!\n","title":"Photos from our LA User Groups First Annual Holiday Party","type":"posts"},{"content":"","date":"December 9, 2014","externalUrl":null,"permalink":"/tag/random-2/","section":"Tags","summary":"","title":"random","type":"tags"},{"content":"","date":"December 4, 2014","externalUrl":null,"permalink":"/tag/dplyr/","section":"Tags","summary":"","title":"dplyr","type":"tags"},{"content":"Yesterday, I published a post here on DataScience.LA with a simple/basic benchmark for dplyr. It was the main part of a short talk at our last Los Angeles R meetup. Below, I continue my thoughts with some reflections on dplyr\u0026rsquo;s API and philosophy, AKA the other part of my talk.\nI think of dplyr as an awesome API for data munging tabular data in R. It implements a \u0026ldquo;grammar of data manipulation\u0026rdquo; similar in impact to the grammar of graphics of ggplot2. It can be considered a mini-DSL. (If you\u0026rsquo;d like to get more insights about Hadley\u0026rsquo;s approach and emphasis on designing good APIs, check out his interview with DataScience.LA). The strength of the syntax comes from both the very well-thought building blocks (like select, filter, group by etc.) and the %\u0026gt;% (pipe) operator that increases enormously the readability of your code.\nWhat strikes me is how using dplyr changed my views and understanding about SQL. In dplyr the building blocks are simple but powerful primitives that do one thing and do it well, similar to the Unix philosophy. (The analogy happens to go even further with the use of pipes to combine the elements.) SQL is more convoluted, but in essence it provides the same data transformations for tabular data. I\u0026rsquo;ve come to see SQL as not quite so dumb after all, although it\u0026rsquo;s still not that nice from an API point of view. (Also worth mentioning that in order to scale out to larger datasets stored in databases, dplyr generates SQL under the hood.)\nAnother point I\u0026rsquo;d like to make is that despite its awesomeness, dplyr does not solve all your data manipulation problems. It does provide, however, a very efficient solution to say 80% of them, thus increasing your productivity by leaps and bounds. In fact, somewhat surprisingly, by \u0026ldquo;restricting\u0026rdquo; you to think in these simple primitives, it helps you solve problems faster - this might sound counterintuitive at first, but trust me and give it a try.\nBesides the awesome API, dplyr is also fast! Romain François re-wrote the critical parts using Rcpp/C++ and now dplyr is way faster than base R, plyr or sqldf and it is approaching data.table\u0026rsquo;s speed. Note that all of these tools are single threaded for now, yet manipulating large datasets with millions of rows takes subseconds or seconds at most. Further speed increases - possibly relevant for even larger datasets - could be achieved with parallelization, something that we\u0026rsquo;ve heard the developers are thinking about.\nOne final feature to mention here is the ability to work with several \u0026ldquo;data sources\u0026rdquo; (data.frames, data.tables, several database backends) that can be manipulated using the same API. For the databases (e.g. SQLite, MySQL, PostgreSQL etc.) dplyr generates SQL that is executed in the database, as previously mentioned.\nFor the past decade or so since I have been using R, my workflow has been (1) write some SQL to extract the data from the database (while also reducing the data size by filtering, aggregating, etc.), (2) get that data into R via the corresponding DBI connector and (3) perform data munging, data visualization and modeling in R. (Note that the SQL was always embedded in the R code and called from there, so this did not fragment the workflow.) In recent years, all three steps fit nicely in a Rmarkdown file processed by knitr at a push of a button in RStudio.\nWhat I\u0026rsquo;ve been experimenting with lately instead is to grab all the data into R - contrary to the myth, the R database connectors are blazing fast and can read for example 10GB of data in a few minutes. From there I can use dplyr for data munging and the rest of my workflow remains the same. (It remains to be seen how this works out with large datasets.)\nAll these changes have been made possible lately by several factors: (1) large amounts of RAM (~100GB) are now cheap, (2) for many well-established companies datasets - contrary to the big data hype - are not exploding (in fact, affordable RAM has increased faster than most datasets), (3) R itself has become way more memory efficient (Did you know that since R 3.1 if you create a new column in a data.frame, R is not going to make copies of all the other columns? This makes some operations 100x faster and with less memory overhead) and (4) packages like dplyr or data.table provide blazing fast operations (and are much more memory efficient).\nWith all this it is no doubt that dplyr is the most awesome in-memory data manipulation tool on the planet. Or Universe. Go, Hadleyverse!\n(If you did not catch it yesterday, check out my post with the benchmark.)\n","date":"December 4, 2014","externalUrl":null,"permalink":"/dplyr-some-more-reflections/","section":"Posts","summary":"Yesterday, I published a post here on DataScience.LA with a simple/basic benchmark for dplyr. It was the main part of a short talk at our last Los Angeles R meetup. Below, I continue my thoughts with some reflections on dplyr’s API and philosophy, AKA the other part of my talk.\n","title":"dplyr - some more reflections","type":"posts"},{"content":"","date":"December 4, 2014","externalUrl":null,"permalink":"/tag/hadleyverse/","section":"Tags","summary":"","title":"Hadleyverse","type":"tags"},{"content":"","date":"December 3, 2014","externalUrl":null,"permalink":"/tag/data-table/","section":"Tags","summary":"","title":"data.table","type":"tags"},{"content":"Earlier this month I gave a short talk about dplyr at the Los Angeles R meetup. It was one of the 5 talks about the useR! 2014 conference, and you can see the summaries and slides from the other 4 talks here.\nOf course, by now we hope everyone using R has heard of dplyr. So why the h@#* would we have another talk about dplyr? And why another blog post? First, dplyr is that awesome! Second, I was suspecting that despite this awesomeness, many people are still not using this tool. Indeed, a quick show of hands at last month\u0026rsquo;s meetup revealed that while over 90% of those present have heard about dplyr, only about 20% are using it. Finally, most talks and blog posts (see e.g. many good ones on r-bloggers) describe how to use it or reflect on how awesome the API is (which cannot be reiterated enough!), but only a few posts speak about performance. It so happened that a week before the meetup I was exploring dplyr and came up with a very basic/simple benchmark that I thought worthwhile to share in both a talk and in this post.\nIn fact, in the first half of my talk I presented a short overview of dplyr and reflected a bit on its features and philosophy. (Note: I\u0026rsquo;ll save this part for a subsequent blog post here on DataScience.LA.) As for the benchmark, my main goal was to get an idea about the performance of dplyr\u0026rsquo;s building blocks (filter, aggregate, join etc.) for largish datasets (and with as little work to do as possible).\nFor a few values of size parameters n and m (n = 10 and 100 million, m = 100, 1000 and 1 million, respectively) I generated data.frames with n rows and 2 columns: x integers between 1 and m, and y random numbers (the size of the datasets in RAM is ~100MB and ~1GB, respectively). A second data.frame of size m is generated for benchmarking joins.\nThe basic tabular operations (filter, aggregate, join etc.) were applied using base R, dplyr (with data.frame and data.table backends, respectively) and standard data.table (with and without key).\nI acknowledge this is just a simple/basic/limited/incomplete benchmark, one could do more with various data types (e.g. character), several grouping variables (x1,x2,\u0026hellip;), more values for size parameters (n,m), different distributions of values in the data.frames etc. (or with real-world datasets). In fact Matt Dowle has published a more complete benchmark for group by operations here, and Brodie Gaslam has also done previously some simple benchmarking of group by, mutate and filter (but not join!) here.\nThe full R code and results are available for your own exploration on github. Latest CRAN versions of R, dplyr and data.table at the time of the study have been used (R 3.1.1, dplyr 0.3.0.2 and data.table 1.9.4). The main results (relative run-times, smaller is better) are summarized below:\nbase dplyr-df dplyr-dt dplyr-dt-k dt dt-k Filter 2 1 1 1 1 1 Sort 30-60 20-30 1.5-3 [1] 1.5-3 [1] New column 1 1 (6) 4 (6) 4 (4) 1 (4) 1 Aggregation 8-100 4-30 4-6 1.5 1.5-5 1 Join \u0026gt;100 4-15 4-6 1.5-2.5 - 1 (Notes: the larger numbers usually appear for larger m, i.e. for more (and smaller) groups; the fastest runtimes are usually a few seconds, pretty good for 10-100 million row datasets).\nSome comments:\n(1) base R is slow for aggregates (base::tapply) and hopelessly slow for (large) joins (base::merge).\n(2) Having a key for data.table or dplyr with data.table source (i.e. having data pre-sorted in place) obviously helps with aggregation and joins (and analyzing sorting in this case is a bit pointless).\n(3) dplyr with data.table backend/source is almost as fast as plain data.table (because in this case dplyr acts as a wrapper and mainly calls data.table functions behind the scenes) - so, you can kinda have both: dplyr API (my personal preference) and speed\n(4) Sorting in dplyr is slower on data.frames, but Hadley says it can be made faster.\n(5) The main reason why dplyr (with data.frame) is slower than data.table for aggregates and joins is explained here (data.table uses radix sort and binary search vs dplyr using hashing for joins).\n(6) While on my first run of the benchmark data.table was surprisingly 4x slower in defining a new column relative to either base R or dplyr (see numbers in parenthesis in the table above), the data.table developers have made some changes and now it\u0026rsquo;s all square (in the development version of data.table on github for now, and in the next CRAN release in the future).\n(7) One unrelated thing I learned with this little project is that R 3.1 (released earlier this year) has made a change that made some data.frame manipulations 100-1000x faster (see here for how slow it was, and adding a new column to a data.frame in base R now is on par with data.table as you can see in the table above).\n(8) The developers of both packages (Hadley \u0026amp; Romain for dplyr and Matt \u0026amp; Arun for data.table) have done outstanding jobs and are constantly improving with features and speed (and they are very responsive to questions/suggestions).\nSince most data scientists are using R or Python as their primary tool (see the results of our survey from the Los Angeles Data Science meetup), I also created a similar benchmark for pandas (the main data munging tool in the Python world) vs R\u0026rsquo;s data.table. It looks like pandas (0.15.1) is slower than data.table for both aggregates and joins (contrary to measurements and claims from almost 3 years ago and which cannot be verified in absence of source code publicly available). For example for n = 10 million and m = 1 million, runtimes (in seconds) are below (and full code and results are in this IPython notebook):\npandas data.table Aggregate 1.5 1 Aggregate (keys/pre-sorted) 0.4 0.2 Join 5.9 - Join (keys/pre-sorted) 2.1 0.5 Creating keys (sort) 3.7 0.7 (Note: the first line is similar to Matt Dowle\u0026rsquo;s benchmark for group by without keys published here).\nIn conclusion, dplyr is pretty fast (way faster than base R or plyr) but data.table is somewhat faster especially for very large datasets and a large number of groups. For datasets under a million rows operations on dplyr (or data.table) are subseconds and the speed difference does not really matter. For larger datasets one can choose dplyr with data.table as a backend, for example. For even larger datasets, or for those preferring the data.table syntax, data.table might be the choice. Either way, R is the best tool for data munging tabular data with millions of rows.\nStay tuned for some reflections on dplyr\u0026rsquo;s API and philosophy later this week, with a new post published here soon!\n","date":"December 3, 2014","externalUrl":null,"permalink":"/dplyr-and-a-very-basic-benchmark/","section":"Posts","summary":"Earlier this month I gave a short talk about dplyr at the Los Angeles R meetup. It was one of the 5 talks about the useR! 2014 conference, and you can see the summaries and slides from the other 4 talks here.\n","title":"dplyr and a very basic benchmark","type":"posts"},{"content":"Data Science @ Datapop: The Marketing Knowledge Graph\npresented by Dr. Ben Zamanzadeh, CTO at DataPop\nToday’s eCommerce has evolved into a complex multi-dimensional optimization problem. DataPop’s semantic technology and Marketing Knowledge Graph is used to generate Omni-Channel advertising campaigns. This presentation will review our technology stack, machine learning models, discuss what worked what did not work for us and do a live demo of Marketing Knowledge Graph.\nDr. Ben Zamanzadeh is passionate about innovation, machine learning and solving challenging R\u0026amp;D problems. Previously he served as VPE at LeadPoint and led Yahoo’s Domain Search and R\u0026amp;D teams. Ben has experience in search, ad tech, eCommerce, financial, mobile, data science, machine learning and semantic computing. Ben holds a Ph.D. in Computer Science from the UCLA where he serves on the CS Advisory Board.\nDate: January 13, 2015\nTimeline:\n6:30pm food/drinks and networking 7:30pm talk starts promptly You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP here on Eventbrite.\nVenue: Cross Campus in Santa Monica will kindly host this meetup, and Factual will kindly sponsor this meetup, providing food+drinks.\n","date":"November 25, 2014","externalUrl":null,"permalink":"/data-science-meetup-january-13-data-science-datapop/","section":"Posts","summary":"Data Science @ Datapop: The Marketing Knowledge Graph\npresented by Dr. Ben Zamanzadeh, CTO at DataPop\nToday’s eCommerce has evolved into a complex multi-dimensional optimization problem. DataPop’s semantic technology and Marketing Knowledge Graph is used to generate Omni-Channel advertising campaigns. This presentation will review our technology stack, machine learning models, discuss what worked what did not work for us and do a live demo of Marketing Knowledge Graph.\n","title":"Data Science Meetup - January 13: Data Science @ Datapop","type":"posts"},{"content":"","date":"November 25, 2014","externalUrl":null,"permalink":"/tag/marketing/","section":"Tags","summary":"","title":"marketing","type":"tags"},{"content":"","date":"November 25, 2014","externalUrl":null,"permalink":"/tag/rd/","section":"Tags","summary":"","title":"R\u0026D","type":"tags"},{"content":"At this meetup we\u0026rsquo;re fortunate to have RStudio\u0026rsquo;s lead developer Joe Cheng flying in from Seattle to talk about Shiny. Shiny makes it possible to develop interactive web applications by writing only a handful of R code. It lowers the barriers to constructing interactive dashboards that you can use to better explain your results to your boss and/or co-workers, or it can help you productionalize your analysis so that others in your company can make use of it regularly without the need to continually bother you.\nBefore the meetup we\u0026rsquo;ll be hosting a more advanced Shiny workshop - and note that you must sign up separately for this session.\nTimeline:\n5:00 pm - 6:30 pm Advanced Shiny workshop (you must RSVP separately - see link below)\n6:30 pm - 7:15 pm Pizza and networking\n7:15 pm Meetup talk on Shiny\nThe meetup talk will cover:\nShiny: What is it? What\u0026rsquo;s new? - by Joe Cheng, RStudio\nShiny is a package for R that\u0026rsquo;s designed to easily turn your analyses into interactive web applications\u0026ndash;no web development knowledge required. Joe Cheng will demonstrate how easy it is to get started with Shiny, and then walk through some examples that show some of the interesting things you can do with it, including some recent developments.\nBio: Joe is a software engineer who has worked at a number of startups including Allaire, Upromise, and Onfolio, and also at Microsoft on Windows Live Writer. At RStudio he built a powerful, intuitive, browser-based IDE for the R programming language. Also created the Shiny web framework for R, and the RPubs document publishing service.\nDate: January 21, 2015 (Wednesday)\nYou must have a confirmed RSVP and please arrive by 7:00 pm the latest. Please RSVP here on Eventbrite for the meetup talk and you must RSVP separately here for the Advanced workshop at 5pm (spots are very limited).\nVenue: Amobee (formerly Adconion) in Santa Monica will kindly host this meetup along with providing pizza and drinks. (A big Thank You to both Mikhail and Amobee for their support!)\n","date":"November 20, 2014","externalUrl":null,"permalink":"/la-r-meetup-january-21-rstudios-shiny-with-joe-cheng/","section":"Posts","summary":"At this meetup we’re fortunate to have RStudio’s lead developer Joe Cheng flying in from Seattle to talk about Shiny. Shiny makes it possible to develop interactive web applications by writing only a handful of R code. It lowers the barriers to constructing interactive dashboards that you can use to better explain your results to your boss and/or co-workers, or it can help you productionalize your analysis so that others in your company can make use of it regularly without the need to continually bother you.\n","title":"LA R meetup - January 21: RStudio's Shiny with Joe Cheng","type":"posts"},{"content":"","date":"November 20, 2014","externalUrl":null,"permalink":"/tag/workshop/","section":"Tags","summary":"","title":"workshop","type":"tags"},{"content":"Last week the LA R meetup featured another round of 5 speakers each highlighting a few things they had found interesting at the useR! 2014 conference. This was the second such event after the first one in September - probably because useR! was awesome enough that it inspired 10 volunteers to stand up on stage and talk more about it.\nThis event had quite a few speakers of note. Over the last 5+ years since I\u0026rsquo;ve been organizing the R meetup in LA (which retrospectively was also the very first data science meetup in Los Angeles - Machine Learning, Hadoop and many others followed years later), I\u0026rsquo;ve had the chance to get to know quite a few members of the LA data community, and it happens that a majority of the speakers for this meetup were top-notch professionals who have also given a few of our data talks previously. Therefore, expectations ran high\u0026hellip;\nWhile I (Szilard Pafka) was the first speaker and I originally planned to talk about my perspective as an organizer at useR! followed by a high-level overview of the conference, but a few days before the meetup I decided to talk about using dplyr with largish datasets instead (a very simple benchmark I\u0026rsquo;ve been running and have been excited about). Since I\u0026rsquo;d like to expand this idea for our blog, I\u0026rsquo;ll save the summary of this part for a subsequent post in the next few days - so stay tuned!\nOur second speaker, Ajay Gopal, the data everything guru at the Santa Monica startup CARD.com talked about a few aspects of using R in production. Ajay was also one of our excellent panelists at the previous LA R meetup about R in production organized during one of the evenings during the conference this past July.\nOne of Ajay\u0026rsquo;s main challenges at CARD.com is to develop and operate a software infrastructure ecosystem heavily built on/around R. He noted happily that R has come a long way and there are now several options for taking a piece of R code and deploying it in 15 minutes into an application that can run on the Web and be used by many or consumed by other software components. He mentions several solutions such as Jeroen Ooms\u0026rsquo; OpenCPU, RStudio\u0026rsquo;s shiny, Revolution Analytics\u0026rsquo; DeployR, Gergely Daroczi\u0026rsquo;s rapporter.net as well as Domino Data Labs. You can see Ajay\u0026rsquo;s slides here:\nAjay Gopal - enteRprise - LA R Meetup - Nov 2014 · Speaker Deck Our next speaker was Eric Kim, Director of Analytics at The Search Agency, who talked about using Shiny to build custom interactive reports in order to optimize SEM revenues for advertisers. When Eric was tasked with building the analytics infrastructure at his company, he decided to turn to R and Shiny instead of the traditional \u0026ldquo;Business Intelligence\u0026rdquo; (BI) products. He showed us some of this progress, and we must say we are all cheering for both him and Shiny! (Note: there were no slides available from Eric\u0026rsquo;s talk because of his presentation format)\nOur forth speaker was Daniel Gutierrez of Amulet Analytics who is also a managing editor and author at insideBIGDATA. He gave his \u0026lsquo;Best Of\u0026rsquo; the conference in several categories which you can view here:\nDaniel Gutierrez - Best of useR! 2014 - LA R meetup - Nov 2014 · Speaker Deck Finally, our awesome DataScience.LA volunteer/co-founder Eduardo Ariño de la Rubia of Ingram Content Group gave a talk about R package dependency management in software deployments and RStudio\u0026rsquo;s packrat. Fortunately for us users (but somewhat unfortunately for his talk, hehe) RStudio made packrat so easy to use in the latest RStudio release that all you need now is to check a \u0026ldquo;Use packrat\u0026rdquo; box when you create a new RStudio project. (Eduardo referred to that box \u0026ldquo;This is my whole dumb talk\u0026rdquo; in the slides). Nevertheless, he gives us some insights about how packrat works when used from the command line. See his slides here:\nEduardo Ariño de la Rubia - Packrat - LA R Meetup - Nov 2014 · Speaker Deck With all these great talks, this was certainly an amazing evening! We\u0026rsquo;d like to give a big Thank You to Amobee (formerly Adconion) for hosting this meetup and for providing pizza. (Also, a special thanks to my friend Mikhail for making all the necessary arrangements!)\nBelow, we leave you with a few pictures from the event:\n","date":"November 18, 2014","externalUrl":null,"permalink":"/la-r-meetup-summary-highlights-from-user-2014-part-2/","section":"Posts","summary":"Last week the LA R meetup featured another round of 5 speakers each highlighting a few things they had found interesting at the useR! 2014 conference. This was the second such event after the first one in September - probably because useR! was awesome enough that it inspired 10 volunteers to stand up on stage and talk more about it.\n","title":"LA R Meetup Summary: Highlights from useR! 2014 - Part 2","type":"posts"},{"content":"","date":"November 18, 2014","externalUrl":null,"permalink":"/tag/packrat/","section":"Tags","summary":"","title":"packrat","type":"tags"},{"content":"","date":"November 18, 2014","externalUrl":null,"permalink":"/category/user-conference/","section":"Categories","summary":"","title":"useR conference","type":"categories"},{"content":"","date":"November 18, 2014","externalUrl":null,"permalink":"/tag/userr-2014/","section":"Tags","summary":"","title":"userR! 2014","type":"tags"},{"content":"","date":"November 10, 2014","externalUrl":null,"permalink":"/tag/ipython-notebook/","section":"Tags","summary":"","title":"IPython notebook","type":"tags"},{"content":"","date":"November 10, 2014","externalUrl":null,"permalink":"/tag/parallel-processing/","section":"Tags","summary":"","title":"parallel processing","type":"tags"},{"content":"Last week, the Python Data Science LA meetup debuted with a fabulous event. It was already much anticipated, with more than 250 people showing interest (RSVP+waiting list), and a lucky 100 converged upon the hip Venice Arts venue to hear our speakers. Our sponsor OpenMail has also done a fantastic job in getting us this venue, bringing food+drinks and taking care of all the details. Thanks OpenMail Team!\nIn this first event, we (your organizers, Szilard and Eduardo) wanted to showcase Python tools used during the various parts of the data analysis process (data munging, data visualization, modeling) and also highlight an environment that facilitates an interactive (and productive) workflow. At the expense of having a very long night, we managed to discuss quite a few topics: pandas for data munging, various visualization libraries, scikit-learn for modeling/machine learning, the IPython notebook as an environment for interactive data analysis - and in addition, we also had a talk on parallel processing. It was an exciting evening indeed!\nAll these topics were delivered by 5 awesome speakers (thanks again to our speakers for volunteering!) While the talks were perfectly accessible, the amount of information presented was quiet overwhelming, so we have prepared this post for the benefit of both those attending the event as well as those who couldn\u0026rsquo;t make it. Below we bring you the slides, code (IPython notebooks) and the video recording of each talk (thanks to our volunteer videographer Jeff Weakley for the work with the videos!)\nWe also have to thank all those attending for providing a vibrant atmosphere during the networking hour before the talks, as well as their attentiveness during the presentations (and thanks for that patience - it was a really long meetup). Finally, I would like to point out that about half of the attendees (including ourselves the organizers) are also using R, as R and Python are the 2 most widely used (and best :)) tools for data science. For more on this, please see our post on our survey conducted at a previous Data Science meetup to see what\u0026rsquo;s commonly used for data munging/datavis/modeling. It\u0026rsquo;s important to note that this survey was one of the main drivers that led us to start this Python Data Science meetup group, based upon the ideas driving those meetups currently serving the R community in LA.\nOur first speaker John Fries (CTO of OpenMail, formerly software engineer at Google) took first the stage to talk about pandas. We all know that we data scientists spend 80% of our time with data munging and the joke says the remaining 20% is spent complaining about the need to do data munging. Considering this truism, having a library that provides a high level and expressive API for manipulating tabular data is essential for our productivity. In order to be efficient (i.e. fast and with low memory footprint) it has to store its columns contiguously in memory and provide bulk operations (unlike e.g. a matrix built of Python lists). Pandas achieves this by building on top of Numpy and provides operations e.g. for filtering, aggregation or joining - not surprisingly similar in scope (and syntax) to SQL. It is an essential tool for data science with Python.\nSee John\u0026rsquo;s slides here:\nJohn Fries - Pandas - PyDSLA meetup - Nov 2014 · Speaker Deck and the video recording of his talk here:\n▶ Data Munging with Pandas - John Fries, CTO, OpenMail John Lin, Data Scientist at TrueCar and former experimental economist presented on IPython notebooks. Collaboration and communication issues are central to the work of a data scientist, from being able to provide reproducibility in findings to effortless communication, iPython notebooks address many of these problems. Showcasing their capabilities as a programming and data science environment, John discussed the interaction method and some great tips and tricks for power users. IPython notebooks are particularly powerful as a collaboration tool for a team of data scientists. The ability to send someone on your local network a link to your notebook or to serialize notebooks and make them available has certainly ignited and accelerated innovation the PyData ecosystem.\nSlides:\nJohn Lin - IPython Notebook - PyDSLA meetup - Nov 2014 · Speaker Deck You can also view his IPython notebook here.\n▶ Using iPython Notebook for Data Analysis - John Lin, Data Scientist, TrueCar Our next speaker, Tamara Knutsen (Front End engineer at OpenMail) provided a survey of the available visualization options in Python. Visualization is central to the work of data scientists, particularly during exploratory analysis. Being able to take complex multi-dimensional data, in which human intuition would fail, and provide visual artifacts which can be reasoned about is one of the most powerful sense-making tools available to a data scientist. In the past, the Python visualization ecosystem had been unfavorably compared with options in other Data Science environments. Tamara\u0026rsquo;s presentation, containing a series of compelling visualizations built in an iPython notebook, also showcased some of the more modern tools for visualization and exploratory data analysis. Starting with the venerable Matplotlib to build some standard visualizations, her presentation quickly moved on to some fantastic advanced topics such as graph applications and interactive visualizations. From Seaborn to Bokeh, from heatmaps to violin charts, this presentation is a fantastic resource for anyone who wants to get up to speed quickly on data visualization leveraging the PyData ecosystem.\nYou can view her IPython notebook (including awesome plots) here.\n▶ Interactive Data Exploration and Visualization in IPython - Tamara Knutsen, OpenMail Rudy Gilmore, a Data Scientist at TrueCar, then presented the incredibly technical topic of parallelism in an easy to understand and accessible fashion. As the limits of physics have been reached and individual CPUs are no longer getting faster, processor manufacturers have adopted the strategy of multiple on-die cores and multiple processors in servers. While this still provides a platform for faster processing and more powerful algorithms, it requires programmers take a step back from sequential SISD programming models. Rudy gave examples of different classes of algorithms, including which ones were better candidates for parallel applications, and discussed Python modules for (threading and multiprocessing) while discussing approaches that would best fit each of these cases.\nSlides:\nRudy Gilmore - Parallel processing - PyDSLA meetup - Nov 2014 · Speaker Deck ▶ Multiprocessing in Python - Rudy Gilmore, Data Scientist, TrueCar Eduardo Ariño de la Rubia had the privilege of being the final speaker at our inaugural meetup. The organizers decided we couldn\u0026rsquo;t have a Python Data Science meetup without including discussion of the fantastic machine learning ecosystem available to python users in scikit-learn. Unfortunately giving a 10-minute presentation on machine learning and scikit-learn is quite a challenge, so it focused on providing a high level view of the different machine learning API\u0026rsquo;s exposed by scikit-learn. Starting with unsupervised techniques such as dimensionality reduction and clustering, then showcasing the classification and regression algorithms, Ed attempted to provide a survey of what can be accomplished within the scikit-learn ecosystem.\nSlides:\nEduardo Arino de la Rubia - scikit-learn - PyDSLA meetup - Nov 2014 · Speaker Deck ▶ Machine Learning with scikit-learn - Eduardo Ariño de la Rubia, Ingram Content Group Since we are a brand new, young meetup, we want to make sure the Los Angeles Python community knows we are actively looking for speakers! If you are interested in giving either a 30-60 minute talk or a 5-10 minute lightning talk on a Python Data Science topic, drop us a line!\nIn summary it was an awesome start for a new meetup and we hope to see you at our next event!\nYour Co-Organizers,\nSzilard \u0026amp; Eduardo\n(this post has been written indeed by the two of us ;))\np.s. - Below are some pictures of our inaugural audience. Enjoy!\nwith some more pictures here.\n","date":"November 10, 2014","externalUrl":null,"permalink":"/python-data-science-meetup-summary-data-science-with-python/","section":"Posts","summary":"Last week, the Python Data Science LA meetup debuted with a fabulous event. It was already much anticipated, with more than 250 people showing interest (RSVP+waiting list), and a lucky 100 converged upon the hip Venice Arts venue to hear our speakers. Our sponsor OpenMail has also done a fantastic job in getting us this venue, bringing food+drinks and taking care of all the details. Thanks OpenMail Team!\n","title":"Python Data Science Meetup Summary: Data Science with Python","type":"posts"},{"content":"Part 2 of the dplyr tutorial is now available! Hadley will be covering Grouped Mutate/Filter, Joins, Do, and Databases. (You can find Part 1 of the tutorial here.)\nFor our viewers who have just discovered the wonders of R, keep in mind that this tutorial was filmed in early July of 2014. As R packages naturally update over time (well, depending on the programmers!), please make sure to check Hadley\u0026rsquo;s Githubfor the latest changes to the dplyr package.\nIf you have questions or comments about the tutorial, dplyr, or R, feel free to ask them here on the DSLA blog and we\u0026rsquo;ll do our best to address questions via comments. Don\u0026rsquo;t forget about Twitter and StackOverflow which are handy references as usual. Thanks for watching, and good luck!\n","date":"November 3, 2014","externalUrl":null,"permalink":"/hadley-wickhams-dplyr-tutorial-at-user-2014-part-2/","section":"Posts","summary":"Part 2 of the dplyr tutorial is now available! Hadley will be covering Grouped Mutate/Filter, Joins, Do, and Databases. (You can find Part 1 of the tutorial here.)\nFor our viewers who have just discovered the wonders of R, keep in mind that this tutorial was filmed in early July of 2014. As R packages naturally update over time (well, depending on the programmers!), please make sure to check Hadley’s Githubfor the latest changes to the dplyr package.\n","title":"Hadley Wickham’s dplyr tutorial at useR! 2014, Part 2","type":"posts"},{"content":"","date":"November 3, 2014","externalUrl":null,"permalink":"/category/sessions/","section":"Categories","summary":"","title":"sessions","type":"categories"},{"content":"","date":"November 3, 2014","externalUrl":null,"permalink":"/authors/tim-phan/","section":"Authors","summary":"","title":"Tim Phan","type":"authors"},{"content":"","date":"November 3, 2014","externalUrl":null,"permalink":"/tag/tutorials/","section":"Tags","summary":"","title":"tutorials","type":"tags"},{"content":"","date":"October 29, 2014","externalUrl":null,"permalink":"/tag/data-vis/","section":"Tags","summary":"","title":"data vis","type":"tags"},{"content":"At this meetup, five more LA R meetup members will present short talks of what they have learned at useR! 2014. This is the 2nd round of talks about useR! since we had twice as many volunteers to speak as we could accommodate in one meetup (indeed, useR! was that awesome). The first round of talks at our September meetup is summarized here (with slides).\nTalks:\nSzilard Pafka, Epoch and DataScience.LA\nBenchmarking dplyr and data.table (with biggish data) - this is my own work but inspired by useR!\nAjay Gopal, CARD.com\nenteRprise - perspectives from useR2014! on R in production\nEric Kim, The Search Agency\nHow The Search Agency is using Shiny to build custom interactive reports to optimize SEM revenues for advertisers\nDaniel Gutierrez, insideBIGDATA\nA personal “Best of” list for those useR! 2014 presentations I attended\n5. Eduardo Ariño de la Rubia, Ingram Content Group\nPackrat: Making your life more predictable with dependency management\nDate: November 11, 2014 (Tuesday)\nTimeline:\n6:00pm pizza arrives and networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP here on Eventbrite.\nVenue: Amobee (formerly Adconion) in Santa Monica will kindly host this meetup and provide pizza and drinks. (A big Thank You to both Mikhail and Amobee!)\n","date":"October 29, 2014","externalUrl":null,"permalink":"/la-r-meetup-november-11-highlights-from-the-user-2014-conference-part-2/","section":"Posts","summary":"At this meetup, five more LA R meetup members will present short talks of what they have learned at useR! 2014. This is the 2nd round of talks about useR! since we had twice as many volunteers to speak as we could accommodate in one meetup (indeed, useR! was that awesome). The first round of talks at our September meetup is summarized here (with slides).\n","title":"LA R Meetup - November 11: Highlights from the useR! 2014 conference - Part 2","type":"posts"},{"content":"","date":"October 29, 2014","externalUrl":null,"permalink":"/tag/reproducibility/","section":"Tags","summary":"","title":"reproducibility","type":"tags"},{"content":"I ♥ data. But I also ♥ privacy. Even though I have been running a few data meetups for the past few years, I have managed to keep myself out of public eye - even in my professional life. I must admit, I\u0026rsquo;m giving a bit away in an recent interview with Data Science Weekly. They had excellent questions to pose, and I’d like to take a moment here to summarize two main points from the interview: my thoughts on when and how much time it took me to became a data scientist and how I feel about the hype in the field today. If you\u0026rsquo;d like, you can read the full interview here.\nI think I became what is called today a data scientist many years ago (before the data science term was used to describe this), but it took me a good number of years, at least 10, to be comfortable with the term. I often like to cite Peter Norvig, Director of Research at Google: “Teach Yourself Programming in Ten Years”. Like most others in the data business years ago, I came from a data-intensive hard-science field (Physics) and I picked up the tools of trade from various books in various disciplines and by doing work with data. It’s certainly way easier now, as there are many more books, online courses etc., and you don’t need to grab this knowledge from various disciplines on your own. The software tools are also more mature - but it still takes some time to become a seasoned data scientist.\nThe field has certainly changed, but many things stay the same. I like to temper the extraordinary hype I often see around data science with the long history of the field prior to the term data science. Many of the machine learning algorithms we are using today were originally developed in the 1990s or 2000s; many of today\u0026rsquo;s software tools have their foundations from more than a decade ago. Of course, building on this strong foundation are innovations from a quickened pace which have resulted in new, primarily “add-on” tools which have massively improved my productivity over the last years (note: read the full interview to find out more). Despite this pace and improved productivity, it\u0026rsquo;s important that we remember that we have been engaging in many of these kinds of data analysis and building advanced analytics applications long before the boom of the past five years. During these changes in the field, let\u0026rsquo;s also keep honest about what data science can and cannot do, as there are still many challenges and limitations we face. And even in cases when it can produce results, there are several pitfalls (e.g. invalid statistical procedures) that require thinking and expertise in order to avoid.\nFinally, I\u0026rsquo;d like to thank Hannah Brooks of Data Science Weekly for reaching out to me for this interview.\n","date":"October 27, 2014","externalUrl":null,"permalink":"/going-a-bit-more-public/","section":"Posts","summary":"I ♥ data. But I also ♥ privacy. Even though I have been running a few data meetups for the past few years, I have managed to keep myself out of public eye - even in my professional life. I must admit, I’m giving a bit away in an recent interview with Data Science Weekly. They had excellent questions to pose, and I’d like to take a moment here to summarize two main points from the interview: my thoughts on when and how much time it took me to became a data scientist and how I feel about the hype in the field today. If you’d like, you can read the full interview here.\n","title":"Going (a bit more) public","type":"posts"},{"content":"","date":"October 27, 2014","externalUrl":null,"permalink":"/tag/interviews/","section":"Tags","summary":"","title":"interviews","type":"tags"},{"content":"","date":"October 20, 2014","externalUrl":null,"permalink":"/authors/david-diez/","section":"Authors","summary":"","title":"David Diez","type":"authors"},{"content":"David Diez works on the OpenIntro team, which develops open-source textbooks, statistical software labs, and other course resources for intro-level statistics. This video above is the talk he gave at useR! 2014 entitled “Textbooks fail where Software Succeeds” in which he outlines the challenges unique to open-source textbooks, the state of OpenIntro, and how people can get involved. Below, he discusses these challenges further and offers his thoughts around their solutions.\nTextbook prices have risen faster than tuition and fees at colleges, and they’ve even risen faster than health care costs. This is a clear signal to us here at OpenIntro that something is fundamentally wrong with the way the textbook industry currently operates.\nOpenIntro and every open-source textbook developer face an entrenched and profitable publishing industry. We want to compete head-to-head with the traditional publishers in a free market, which we believe will generate better educational products at better prices for students. However, the intro-level textbook system has developed in a way that we think prevents this free market from existing for three reasons:\nStudents, who are the market’s price-sensitive consumers, do not freely choose which textbook to purchase. That’s a job of each course instructor or department.\nThe teachers and departments who decide which book students should buy aren’t impacted by textbook price: they typically receive free textbook copies called “desk copies” for teachers and teaching assistants from the publisher.\nVirtually no one other than college students buy new introductory-level textbooks, so there are few price-sensitive consumers with decision powers which influence the market.\nThese three realities form the foundation of why intro-level textbooks are so expensive today. If textbook prices are going to ever change in a free-market fashion, we believe that we will need to fundamentally change at least one of these three pillars.\nTo displace the first pillar, it would require teachers to give up the ability to assign a particular textbook for a course to students and instead rely on students selecting suitable course materials, perhaps by reading reviews and comparing prices. Instructors could also rely on open-source exercise repositories to avoid reliance on a single textbook for homework problems. However, these possible solutions are not without their challenges. For example, it would be difficult to ensure each student purchases a textbook that covers all of the required topics, and some textbooks also use slightly different terms and notation. Such problems are far from insurmountable, but the known solutions add some burden onto instructors’ workloads, including adjunct faculty who frequently teach many courses simultaneously.\nThe second pillar also has no simple, universal solution. The primary goal is ensuring that departments and instructors consider price as carefully as students consider price. There are a remarkable number of instructors who pay close attention to the downsides of textbook costs, e.g. low-income students frequently skip buying course textbooks, and these instructors will seriously examine low-cost or open-source textbook options. However, this is not a universal solution \u0026ndash; at least not yet. Here are a couple ways to try to nudge more instructors and departments to move towards a price-sensitive mindset:\nOne option is to increase price awareness among teachers and departments by highlighting the damage done to learning outcomes when students avoid textbooks due to their high price. This is the strategy OpenIntro has generally followed, e.g. through blog posts and talks at conferences.\nA second option is regulatory, either on the state or national level. For example, regulations could restrict publishers from gifting “desk copies” of textbooks and other materials to state college departments. (I’ll talk about how OpenIntro currently handles gifting of textbooks later on.) Stopping this textbook gifting would ensure, for example, that whenever a publisher comes out with a new edition, departments would be hit with an unexpected cost, just like the students currently experience. If this happened in a course that had one instructor and 4 teaching assistants, the cost of buying 5 copies of the new edition could reach $1000 or more. This could further motivate real consideration of lower-cost options.\nThe third pillar considers the possibility that there may be a third option: create a large, new market outside of the university system for introductory-level textbooks that is price-sensitive and that will be tempting enough for publishers to pursue. This new market would need to be a comparable size to that of college students to be effective. However, this solution has two problems: there is no such market (e.g. instruction of industry employees is a much smaller market), and many consumers in existing markets are already well-served by free online resources. That is, to break this pillar would require publishers to see the value in pursuing a smaller market where they will most likely have trouble competing, even when doing so will likely harm their current profit margin. (For some perspective, consider that a substantial fraction of the market for advanced statistics textbooks is from practitioners and researchers. That is, such books typically compete in a price-sensitive market, which often keeps prices of advanced statistics textbooks below the prices of introductory-level statistics textbooks, even though the market for each advanced book is much smaller.)\nHow is OpenIntro currently contributing to a solution for students? # We estimate that over 5,000 students (probably closer to 8,000 students) used OpenIntro’s open-source textbooks in 2014 courses, which amounts to $500,000 to $800,000 in student savings. For perspective: this is still less than 1% of the intro-level statistics textbook market, so we have a long ways to go.\nOur open-source textbooks are downloaded about 10,000 times each month, and over 25,000 times each month during peak usage. OpenIntro is also sustainable, as our staff is 100% volunteer (and regularly expanding), and our costs only amount to about $2500 annually.\nHow OpenIntro textbook gifting works, and how you can contribute # Our textbooks are free online and physical copies are priced under $10 on Amazon (meaning they are sold royalty-free.) In 2013, we decided to start giving instructors up to 2 free OpenIntro textbooks. Since we have suggested possible regulation around such textbook gifts, that sounds hypocritical, right? Well, there are two big differences between what we do at OpenIntro and what traditional publishers do.\nOur textbook prices are typically 5-10% of the textbook price of our competitors. That is not a typo. Our prices differ by an order of magnitude, and we also give electronic versions away to everyone for free.\nMuch more importantly, the “free” copies that publishers provide to teachers are indirectly paid for from the profits of textbook sales to students, which we think should change. In OpenIntro’s model, we have no profits \u0026ndash; from students or anyone else \u0026ndash; so we pay out-of-pocket for teacher copies since we know getting our books in the hands of teachers will ultimately help students save money.\nOpenIntro’s model ensures there is no financial conflict of interest, so we think that sending each teacher a $10 textbook or two is quite reasonable.\nWe also now welcome Amazon gift cards at givebooks@openintro.org to offset our out-of-pocket expenses. These funds will be used exclusively to buy textbooks for teachers, or if we get an overwhelming response, to also buy textbooks for high schools and low-income students. If you’d like to help, consider sending an Amazon gift of $10. (Please note: gift cards are not tax-deductible.)\nAcknowledgements # I appreciate the opportunity to write a blogpost for DataScience.LA, especially on such a controversial topic. A big thank you to Mine Çetinkaya-Rundel and Andrew Bray of OpenIntro, who provided great advice and substantial feedback on this post.\nEditor\u0026rsquo;s note: The opinions expressed in this guest post do not necessarily reflect the opinions of the DataScience.LA Editorial Board, and we thank David for contributing this post and helping to stimulate community discussion around this topic.\n","date":"October 20, 2014","externalUrl":null,"permalink":"/openintro-org-on-why-introductory-textbooks-are-so-expensive/","section":"Posts","summary":"David Diez works on the OpenIntro team, which develops open-source textbooks, statistical software labs, and other course resources for intro-level statistics. This video above is the talk he gave at useR! 2014 entitled “Textbooks fail where Software Succeeds” in which he outlines the challenges unique to open-source textbooks, the state of OpenIntro, and how people can get involved. Below, he discusses these challenges further and offers his thoughts around their solutions.\n","title":"OpenIntro.org on why introductory textbooks are so expensive","type":"posts"},{"content":"","date":"October 20, 2014","externalUrl":null,"permalink":"/tag/undergraduates/","section":"Tags","summary":"","title":"undergraduates","type":"tags"},{"content":"","date":"October 15, 2014","externalUrl":null,"permalink":"/category/interviews/","section":"Categories","summary":"","title":"interviews","type":"categories"},{"content":"JJ Allaire has courageously bet, time and time again, on the cutting edge of technologies. In doing so, he has not only benefited from the fertile rains but has often times himself been the rainmaker. He has, to a great degree, democratized development on the web by bringing the power to create interactive web applications to a wider audience than those select few who had access to both a shell account and Unix knowledge. ColdFusion was a platform by the web and for the web, and its adoption revolutionized programming ‘for the web’ as its own worthy pursuit. This story also resolutely validates the idea that by creating tools for developers at the beginning of a time of revolutionary change in technology, one could not only validate the tools but also help stage the unfolding narrative as one with many participants.\nOver the past few years, JJ has invested his time in developing the RStudio IDE, including the supporting frameworks and tools. Even before his involvement in the software industry, JJ fostered his love for how data could influence the world around him as a Political science major. After seeing how “the definitive” environment for interactive data analysis had not yet been built, he set about building it - and for that we here at DataScience.LA thank him.\nSuffice it to say, I personally think quite highly of JJ. It is with great pleasure that we are releasing this conversation, the final video of our useR! 2014 interview series this week - especially while we’re gearing up for Strata. I hope you enjoy!\n▶ JJ Alaire – Interview by DataScience.LA at useR 2014 ","date":"October 15, 2014","externalUrl":null,"permalink":"/jj-allaire-the-user-2014-interview/","section":"Posts","summary":"JJ Allaire has courageously bet, time and time again, on the cutting edge of technologies. In doing so, he has not only benefited from the fertile rains but has often times himself been the rainmaker. He has, to a great degree, democratized development on the web by bringing the power to create interactive web applications to a wider audience than those select few who had access to both a shell account and Unix knowledge. ColdFusion was a platform by the web and for the web, and its adoption revolutionized programming ‘for the web’ as its own worthy pursuit. This story also resolutely validates the idea that by creating tools for developers at the beginning of a time of revolutionary change in technology, one could not only validate the tools but also help stage the unfolding narrative as one with many participants.\n","title":"JJ Allaire, the useR! 2014 interview","type":"posts"},{"content":"","date":"October 15, 2014","externalUrl":null,"permalink":"/tag/rstudio/","section":"Tags","summary":"","title":"RStudio","type":"tags"},{"content":"Hadley Wickham (perhaps you’ve heard of his work) presented a 2 hour workshop on dplyr at this year’s useR! conference at UCLA. This tutorial was definitely a highlight of the week-long conference for me, and working on this tutorial video has also made me very appreciative of how versatile the dplyr package can be. It clearly is the chef’s knife of data science tools.\nHadley’s presentation was just under 2 hours long, and the edited footage where we omitted breaks gives us 90 minutes of wisdom and inspiration. I’ve split this tutorial into 2 relatively even parts for your learning convenience. If this is your first-ever attempt at learning dplyr, I definitely suggest concentrating on the basics presented here in Part 1 before moving on to next week’s video. Two great pieces of advice to follow during this tutorial come from some of the R greats:\nOne of Martin Maechler\u0026lsquo;s rules of good R programming practice is to never copy and paste. Try to always type the commands; go line by line through the code and do your best to understand why it is what it is.\nIn his introduction, Hadley Wickham provides a gem that I want to highlight here.\nWhenever you’re learning a new tool, for a long time you’re going to suck\u0026hellip; But the good news is that is typical, that’s something that happens to everyone, and it’s only temporary.\nPart 1 (this video) covers the following topics:\nA introduction, a bit of theory, and a description of the data Single table verbs (filter/select/arrange/mutate/summarise) and grouped summaries Data pipelines I designed this video to be as user-friendly as possible, in hopes of inspiring newcomers to R and rStudio alike. Hadley’s talk was obviously geared towards an intermediate/advanced audience, so I’ve added my own annotations (in light blue) as quick tips for beginners. As you’ll see, Hadley’s workshop often took short breaks for “homework”. I highly urge you pause the video during each problem set and attempt to figure it out on your own before proceeding to the answers. There are also several occasions where Hadley goes off-script from the “dplyr-tutorial.pdf” and tweaks his own solution to the problem sets with answers from the crowd. Don’t worry if the answers on the PDF don’t match the video - remember that there are many different methods of programming in R, and part of the learning process is to find your own style. Most importantly, when you get stuck don’t forget to consult our amazing #rstats community available via Twitter, StackOverflow, Reddit, and other various places across the internet.\nNote: I did not have access to Hadley’s console while editing this video, so the console overlays you’ll see are my best attempts to recreate the code he is using. For this reasons, any hypothetical errors are certainly mine and not Hadley’s.\nIn order to give you time to digest Part 1 before embarking upon Part 2 of this tutorial, we will be publishing Part 2 next week. This video will cover grouped mutate/filter \u0026amp; window functions, joins via two table verbs, and the “Do” function and related databases. Feel free to provide feedback on this tutorial in the comments below, or via my Twitter at @timothy_phan.\nHadley\u0026rsquo;s scripts from this tutorial can be accessed here. Press \u0026ldquo;Download as .zip\u0026rdquo; in the top right corner to download the entire directory. Happy learning, and remember: figuring out how to teach yourself new concepts is essential to improving as a data scientist.\nGood luck, and stay tuned for Part 2 next week!\n","date":"October 13, 2014","externalUrl":null,"permalink":"/hadley-wickhams-dplyr-tutorial-at-user-2014-part-1/","section":"Posts","summary":"Hadley Wickham (perhaps you’ve heard of his work) presented a 2 hour workshop on dplyr at this year’s useR! conference at UCLA. This tutorial was definitely a highlight of the week-long conference for me, and working on this tutorial video has also made me very appreciative of how versatile the dplyr package can be. It clearly is the chef’s knife of data science tools.\n","title":"Hadley Wickham's dplyr tutorial at useR! 2014, Part 1 ","type":"posts"},{"content":"Hadley Wickham is hard at work, releasing packages which leverage the expressive power of R to keep easy things intuitive, and to make hard things possible. Having already developed a number of packages which address other steps in the data processing and analysis pipeline, he chose to address the process by which R developers ingest, manipulate, and transform data. This package, dplyr, is a narrowly focused library, designed to provide 90% of the functionality required in most data munging tasks. It may not solve every possible problem in data munging, but dplyr’s built in primitives will reduce your need to drop to “base R” for most common tasks. It does so by leveraging the magritr package’s “pipe” operator, and is inspired by dataflow programming constructs. It is fast, it is powerful, and it has been the single largest productivity boost that many of us here at DataScience.LA have adopted in 2014. In this video, Hadley provides a very quick introduction to the dplyr package, describing his philosophy behind the package’s design while providing a number of illustrative examples as to the flexibility, expressiveness, and power of this tool.\nSince this is such an awesome and productivity-enhancing tool, we will also be releasing a series of useR! 2014 tutorial videos, a guided hands-on session on dplyr, over the next two weeks so stay tuned. I hope you enjoy!\n","date":"October 13, 2014","externalUrl":null,"permalink":"/hadley-wickham-presents-dplyr-at-user-2014/","section":"Posts","summary":"Hadley Wickham is hard at work, releasing packages which leverage the expressive power of R to keep easy things intuitive, and to make hard things possible. Having already developed a number of packages which address other steps in the data processing and analysis pipeline, he chose to address the process by which R developers ingest, manipulate, and transform data. This package, dplyr, is a narrowly focused library, designed to provide 90% of the functionality required in most data munging tasks. It may not solve every possible problem in data munging, but dplyr’s built in primitives will reduce your need to drop to “base R” for most common tasks. It does so by leveraging the magritr package’s “pipe” operator, and is inspired by dataflow programming constructs. It is fast, it is powerful, and it has been the single largest productivity boost that many of us here at DataScience.LA have adopted in 2014. In this video, Hadley provides a very quick introduction to the dplyr package, describing his philosophy behind the package’s design while providing a number of illustrative examples as to the flexibility, expressiveness, and power of this tool.\n","title":"Hadley Wickham presents dplyr at useR! 2014","type":"posts"},{"content":"","date":"October 8, 2014","externalUrl":null,"permalink":"/tag/multiprocessing/","section":"Tags","summary":"","title":"multiprocessing","type":"tags"},{"content":"I\u0026rsquo;m pleased to announce the first event of our new Python Data Science LA meetup, started by two of us here at DataScience.LA (our very own Szilard and Eduardo).\nWe\u0026rsquo;ll have 5 speakers at this event, covering various aspects of the Python Data Science ecosystem (NumPy, SciPy, pandas, matplotlib, scikit-learn, NetworkX, NLTK, IPython notebook, etc.)\nWe\u0026rsquo;d like to thank OpenMail (especially John Fries) for sponsoring this meetup, i.e. hosting us at Venice Arts and providing food and beverages while we have an opportunity to network.\nHere\u0026rsquo;s more information on the talks scheduled for this meetup:\n1. Data Munging with Pandas - John Fries, CTO, OpenMail\nData munging skills are the keys to the data analysis kingdom. In this talk we\u0026rsquo;ll go over fundamental data munging techniques using the Pandas library and present some data munging brain teasers.\nSpeaker bio: MIT Bachelor\u0026rsquo;s in Mathematics, Software engineer at Google for 6 years where his major projects included Picasaweb, Google News, \u0026amp; YouTube.\n2. Using the iPython Notebook for data analysis - John Lin, Data Scientist, TrueCar\nThis talk will illustrate how to use the iPython Notebook, while comparing and contrasting this tool with “regular” Python software development. Also included will be the advantages of using iPython Notebook for data analysis and pitfalls to avoid.\nSpeaker bio: John Lin received his economics training at the University of Michigan and Caltech, and has worked as a programmer. He now combines both loves as a Data Scientist at TrueCar, working to improve the car shopping experience and making it awesome.\n3. Interactive data exploration and visualization in IPython - Tamara Knutsen, Front End Engineer, OpenMail\nVisualization allows for insightful exploration of data. I will present the different visualization and plotting Python libraries for interactive data exploration in IPython notebook, covering Matplotlib, Seaborn, mpld3, Bokeh, VisPy and more.\nSpeaker bio: Former computational neuroscientist at Caltech, now programming front end and analytics at OpenMail.\n4. Multiprocessing in Python - Rudy Gilmore, Data Scientist, TrueCar\nThis talk is an introduction to running your data analysis code in parallel in Python. I will discuss some of the options for running code on multiple processors, while showing a few simple examples.\nSpeaker bio: Former research physicist, now working in data science.\n5. Machine learning with scikit-learn - Eduardo Ariño de la Rubia, Ingram Content Group\nHear an introduction to the machine learning possibilities in Python using the fantastic scikit-learn ecosystem. I will discuss the basics of individual models available, including examples, while incorporating higher-level constructs such as pipelines.\nSpeaker bio: Long-time programmer who is especially enamored with bio-inspired soft computing and machine learning methods.\nDate: November 3, 2014 (Monday)\nTimeline:\n6:30pm food/bev \u0026amp; networking 7:30pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP for this meetup here on Eventbrite, as space is limited due to the capacity of the room. Make sure to RSVP now!\nVenue: Venice Arts, 1702 Lincoln Blvd, Venice, CA 90291\n","date":"October 8, 2014","externalUrl":null,"permalink":"/python-data-science-meetup-data-science-with-python/","section":"Posts","summary":"I’m pleased to announce the first event of our new Python Data Science LA meetup, started by two of us here at DataScience.LA (our very own Szilard and Eduardo).\nWe’ll have 5 speakers at this event, covering various aspects of the Python Data Science ecosystem (NumPy, SciPy, pandas, matplotlib, scikit-learn, NetworkX, NLTK, IPython notebook, etc.)\n","title":"Python Data Science Meetup: Data Science with Python","type":"posts"},{"content":"Joe Cheng is a software engineer. Unfortunately the term gets thrown around pretty lightly, so I’d like to make sure that I convey the appropriate amount of gravitas when I refer to him as such. Joe Cheng is a *great* Software Engineer. He has worked for some of the largest technology companies on the planet, building cutting edge tools which empowered the web browser as a full blown platform instead of a historical curiosity. Joe is one of those software engineers who saw a future with the web as a platform for modern software, then worked hard to make it a reality. When the time came to build the next generation of data science IDEs, an IDE targeting the web as a platform, RStudio called Joe.\nJoe and I share a traditional computer science background, which made this one of my favorite conversations of useR! 2014. We discussed our view of R, not as professional statisticians or data scientists, but as programmers. R is a fantastic and purpose-built language and environment, but it is unlike most programming environments that either Joe or I have ever encountered. If you’ve ever been curious to learn what shocks traditional computer programmers when they first learn R, you can watch that part of our discussion here.\nJoe is currently focused on RStudio’s web application framework, Shiny. Data science as communication is a hot topic, and so much of modern communication can benefit from these web-based, truly interactive presentations. Inspired by the cutting-edge Meteor framework, Shiny leverages reactive principles to weave together rich applications with minimal code. It’s not a general purpose framework, you wouldn’t use it to build a storefront, but it doesn’t claim to be such. We talk about the development of the Shiny framework, the choices that were made, and how Shiny is accessible to novice and expert R users alike by design here.\nAs a bit of an aside, there’s one amazing tidbit from this video which should not be missed. Joe Cheng nearly had his love of computer programming bored out of him by the worst summer camp *ever*! I hope you enjoy the interview as much as I did!\n","date":"October 6, 2014","externalUrl":null,"permalink":"/a-conversation-with-joe-cheng-at-user-2014/","section":"Posts","summary":"Joe Cheng is a software engineer. Unfortunately the term gets thrown around pretty lightly, so I’d like to make sure that I convey the appropriate amount of gravitas when I refer to him as such. Joe Cheng is a *great* Software Engineer. He has worked for some of the largest technology companies on the planet, building cutting edge tools which empowered the web browser as a full blown platform instead of a historical curiosity. Joe is one of those software engineers who saw a future with the web as a platform for modern software, then worked hard to make it a reality. When the time came to build the next generation of data science IDEs, an IDE targeting the web as a platform, RStudio called Joe.\n","title":"A Conversation with Joe Cheng at useR! 2014","type":"posts"},{"content":"","date":"September 26, 2014","externalUrl":null,"permalink":"/tag/deployment/","section":"Tags","summary":"","title":"deployment","type":"tags"},{"content":"","date":"September 26, 2014","externalUrl":null,"permalink":"/tag/jj-allaire/","section":"Tags","summary":"","title":"jj allaire","type":"tags"},{"content":"There comes a time in a software toolchain’s lifecycle where the focus shifts from developer productivity and preference to the ability to deploy solutions in production which coexist gracefully in the enterprise. Configuration management and dependency management are two of the first points of friction in this transition, as the free-wheeling methods preferred by developers are at odds with the structured methods and isolation desired by administrators, who serve as gatekeepers to enterprise deployment. R is currently at this very point in the transition, in the midst of transformation from a language and environment used primarily in academic research to its growing place in the enterprise. At useR! 2014 there were a series of presentations on how dependency management can be addressed within the R ecosystem, ranging from “provider” side options (e.g. freezing CRAN and private repositories) to the client side options, where a project maintains its own dependencies as part of its codebase.\nIn the video above, JJ Allaire and Kevin Ushey give a detailed presentation of Packrat at useR! 2014. Packrat is a client-side solution which allows a project to manage dependencies within its own directory tree. This is a tried and true approach, currently utilized in communities from Ruby to Perl. Packrat allows the developer to isolate projects from each other, even though they may share the same operating system image and physical hardware. This allows two separate projects to use completely different versions of the same library and coexist gracefully, an impossibility without such solutions. Packrat also allows for greater reproducibility, as underlying changes in system libraries will no longer impact the results of a run. Finally, packrat provides deployment portability by bundling all of an R application’s dependencies within the project, thereby freeing the DevOps group from having to build Puppet/Ansible/Chef code which installs R packages as a dependency for deploying a project.\nI have adopted this package for all new R projects and have personally experienced greater peace of mind in knowing that in some small part, this approach makes the integration of my projects easier on the enterprise and helps make R a “well behaved” member of the enterprise toolchain. I hope you find similar benefits by using Packrat in your own work!\n▶ Rstudio\u0026#39;s \u0026#34;Packrat\u0026#34; talk by J.J Allaire and Kevin Ushey ","date":"September 26, 2014","externalUrl":null,"permalink":"/packrat-presentation-at-user-2014/","section":"Posts","summary":"There comes a time in a software toolchain’s lifecycle where the focus shifts from developer productivity and preference to the ability to deploy solutions in production which coexist gracefully in the enterprise. Configuration management and dependency management are two of the first points of friction in this transition, as the free-wheeling methods preferred by developers are at odds with the structured methods and isolation desired by administrators, who serve as gatekeepers to enterprise deployment. R is currently at this very point in the transition, in the midst of transformation from a language and environment used primarily in academic research to its growing place in the enterprise. At useR! 2014 there were a series of presentations on how dependency management can be addressed within the R ecosystem, ranging from “provider” side options (e.g. freezing CRAN and private repositories) to the client side options, where a project maintains its own dependencies as part of its codebase.\n","title":"Packrat presentation at useR! 2014","type":"posts"},{"content":"","date":"September 24, 2014","externalUrl":null,"permalink":"/tag/data-munging/","section":"Tags","summary":"","title":"data munging","type":"tags"},{"content":"This is a re-publication of a blog post from a blog I created not long before we got the idea to start DataScience.LA, and in many aspects it is the genesis of DSLA (the blog\u0026rsquo;s name, the people getting together - thanks Eduardo, Amelia and Leigh). It was also the spark that lead to the creation of the Python Data Science LA meetup.\nAt a previous LA Data Science/Machine Learning meetup we did an informal survey (using a quick show of hands and a very approximate count) of the software tools used for data analysis by those in the audience. With about 200 people in attendance and about 60% saying they are data scientists or perform a similar job, I believe the results below are more representative and less biased than our previous attempts via meetup polls (where relatively few responded) or even more formal surveys others have attempted (e.g. my friends at KDnuggets or Rexer Analytics). I know, there is a likely bias in our results as well, towards open source tools, maybe our meetups are just too cool for all those SAS users out there (I kid, I kid.)\nConsidering the various parts of the process of analyzing data,\nwe surveyed the audience for tools used in:\ndata munging (“explore”, “clean” and “transform” above) – both exploratory data analysis (EDA) and operational ETL, visualization – both exploratory and presentational, machine learning/modeling. We started with tools that I (Szilard) thought must be the most popular, but we also asked what other tools people are using, so we didn’t miss any hidden gems.\nWhen we asked about data munging we found that about 60% are using R in some part of their data science process, 50% were using Python, 40% SQL, 30% Hadoop (mostly Hive), 20% Unix shell. Only 10% acknowledged using Excel. Other tools used by some of our attending data scientists included Perl, Matlab, SAS – Pig, Impala, Shark within Hadoop. Clojure, Scalding, Elasticsearch made an appearance with just one user each.\nFor visualization, about 40% were using R, 30% using Python, 10% Tableau, 10% building custom in Javascript (although we admit we did not ask which libraries). Just a few people are using Excel or Matlab, and there was a Splunk user out there too.\nWhen it came to machine learning, only 30% of the audience raised their hands saying they used R, with about 30% using Python. There were quite a few tools mentioned with 2-3 hands apiece, such as Vowpal Wabbit, SAS, SPSS, Matlab, Mahout. In the minority were users of Spark MLlib, Graphlab, Shogun, and Weka.\nI feel that this quick and dirty poll gives us a starting picture of a typical data scientist\u0026rsquo;s toolbox across a few of our major tasks. With a more formal survey we can get into further detail within our rough categories, such as breaking down by usage for EDA vs ETL or separating into groups those who use various R/Python packages. As you could imagine, we’re thinking of doing this in the near future - stay tuned.\nI’d like to thank Eduardo Arino de la Rubia for helping with the survey. Feel free to post your personal tools of choice in the comments below (especially if it’s not mentioned above)!\n","date":"September 24, 2014","externalUrl":null,"permalink":"/data-science-toolbox-survey-results-surprise-r-and-python-win/","section":"Posts","summary":"This is a re-publication of a blog post from a blog I created not long before we got the idea to start DataScience.LA, and in many aspects it is the genesis of DSLA (the blog’s name, the people getting together - thanks Eduardo, Amelia and Leigh). It was also the spark that lead to the creation of the Python Data Science LA meetup.\n","title":"Data Science Toolbox Survey Results... Surprise! R and Python win","type":"posts"},{"content":"","date":"September 24, 2014","externalUrl":null,"permalink":"/tag/excel/","section":"Tags","summary":"","title":"Excel","type":"tags"},{"content":"","date":"September 24, 2014","externalUrl":null,"permalink":"/tag/tableau/","section":"Tags","summary":"","title":"Tableau","type":"tags"},{"content":"","date":"September 24, 2014","externalUrl":null,"permalink":"/tag/unix/","section":"Tags","summary":"","title":"Unix","type":"tags"},{"content":"Big Data Evangelist\n","date":"September 23, 2014","externalUrl":null,"permalink":"/authors/subash-dsouza/","section":"Authors","summary":"Big Data Evangelist\n","title":"Subash D'Souza","type":"authors"},{"content":"I have been following Big Data, Hadoop in particular, for the past four years and a lot has changed since then. At the time I was looking for the next big wave of technologies, especially on the back end of development as that was my forte at the time. After our initial introduction, the more I learned the more I felt Hadoop was the right tool to get the job done. Focusing upon the development aspects of job vs actual maintenance of the servers/cluster has always been a no-brainer for me. Now that Hadoop has been adopted by a wide variety of industries and has grown quite popular, it has become much more apparent that there are indeed areas where Hadoop is lacking. Security and real-time analytics are two of these concerns which both affect real-time processing and reporting, but fortunately there is a lot of current work happening to address these limitations. Consequently, there is a group of big data vendors who are investing a large number of resources as a remedy, bringing us newer tools such as Yarn, Tez, Storm, etc. Also, acquisitions of XA Secure by Hortonworks and Gazzang by Cloudera are contributing to the solving these issues around security.\nWithin Hadoop, Mapreduce has always been a focal point since it is involved in the majority of data processing. There is a fundamental idea, however, that there could be something even better than Mapreduce, especially in terms of better performance and optimizations. There has even been recent talk of Mapreduce going away. Perhaps this is true, but maybe not. From my perspective, people will always take a look at tools that will claim to deliver the golden unicorn but the question remains of whether these new tools will stay in the spotlight over time.\nBy no means am I saying Spark is indeed this unicorn we’ve all been waiting for to solve these problems with Hadoop - but it certainly has a lot of promise. Spark supports in-memory data sharing across DAGs (Directed Acyclic Graphs), so that different jobs can work with the same data at very high speed. It even allows for cyclic data flows. In addition to Spark itself, it also offers tools such as Spark SQL, Spark Streaming, MLib (for machine learning) and GraphX. For these and many other reasons, I believe Spark is the part of every developer’s/admin’s toolset which will make real time processing in Hadoop more possible, an exciting prospect indeed.\nAs a Big data evangelist, part of my job is to understand how the big data market shifts with time. Considering that most major vendors are on board with Spark (e.g. Cloudera, MapR, Datastax, Pivotal \u0026amp; Hortonworks) there is less uncertainty as to whether Spark will remain popular or whether it is a fad that will pass with time. In my view, I truly believe that Spark is on the same upward trajectory that Hadoop was following 4-5 years ago. There is a great blog post explaining Spark’s advantages and drawbacks, which I feel you should read for just this reason, in case you haven’t read it yet.\nConsidering this case for Spark, are you interested in learning more? Come take a look at the Spark Panel meetup on September 30th. We have also recently added a getting started with Spark \u0026amp; Scala session on November 18th. Hope to see you there!\n","date":"September 23, 2014","externalUrl":null,"permalink":"/the-case-for-apache-spark/","section":"Posts","summary":"I have been following Big Data, Hadoop in particular, for the past four years and a lot has changed since then. At the time I was looking for the next big wave of technologies, especially on the back end of development as that was my forte at the time. After our initial introduction, the more I learned the more I felt Hadoop was the right tool to get the job done. Focusing upon the development aspects of job vs actual maintenance of the servers/cluster has always been a no-brainer for me. Now that Hadoop has been adopted by a wide variety of industries and has grown quite popular, it has become much more apparent that there are indeed areas where Hadoop is lacking. Security and real-time analytics are two of these concerns which both affect real-time processing and reporting, but fortunately there is a lot of current work happening to address these limitations. Consequently, there is a group of big data vendors who are investing a large number of resources as a remedy, bringing us newer tools such as Yarn, Tez, Storm, etc. Also, acquisitions of XA Secure by Hortonworks and Gazzang by Cloudera are contributing to the solving these issues around security.\n","title":"The case for Apache Spark","type":"posts"},{"content":"","date":"September 22, 2014","externalUrl":null,"permalink":"/tag/dirk/","section":"Tags","summary":"","title":"dirk","type":"tags"},{"content":"First things first, Dirk Eddelbuettel was recently named ordinary. This seems contradictory, since Dirk is a known HPC expert, an organizer of the R in Finance conference, the creator of Rcpp, and a Debian contributor. These are only a few of the many accolades bestowed upon Dirk without even a hint of puffery. And yet, Dirk Eddelbuettel is considered ordinary. What makes Dirk ordinary? It should be mentioned that to the Vienna-based nonprofit that provides R’s leadership, the R Foundation for Statistical Computing, ‘ordinary’ means something quite different to the lay person. Ordinary members provide guidance and direction for the R Project for Statistical Computing. It’s hard to imagine someone more qualified than Dirk for this task.\nA longtime R user, Dirk’s professional life is in finance as a self-described quant (one of the ‘Rocket Scientists of Wall Street’). Dirk has worked for some of the largest financial organizations in the world, and has open sourced packages which allow for tasks ranging from vanilla options pricing to discounted cash flow analysis. Even though the world of finance is a “massive net importer” of open source tools, he has succeeded in providing finance packages for the open source community. In addition to these contributions, he has also provided the larger R community with the infinitely useful Rcpp package.\nRcpp is the most widely used language extension for R, and this package provides a novel and seamless method of integrating high performance C++ code with R without requiring the usual extension incantations. Dirk’s book, Seamless R and C++ Integration with Rcpp (Springer, 2013) makes it surprisingly easy to pick up the Rcpp basics in an afternoon. For example, I recently found myself facing a CPU-bound data manipulation task and was able to convert the code from R to Inline C++ using Rcpp - and my code received a 26x speedup for a few hours of work invested.\nIt’s easy to see why, with all of these contributions, the R Foundation would find Dirk ordinary. I was incredibly fortunate at useR! 2014 to sit down with Dirk and have a long conversation about many of these ordinary topics, the video of which is included here. Enjoy!\n▶ Dirk Eddelbuettel - Interview by DataScience.LA at useR 2014 ","date":"September 22, 2014","externalUrl":null,"permalink":"/dirk-eddelbuettel-the-user-2014-interview/","section":"Posts","summary":"First things first, Dirk Eddelbuettel was recently named ordinary. This seems contradictory, since Dirk is a known HPC expert, an organizer of the R in Finance conference, the creator of Rcpp, and a Debian contributor. These are only a few of the many accolades bestowed upon Dirk without even a hint of puffery. And yet, Dirk Eddelbuettel is considered ordinary. What makes Dirk ordinary? It should be mentioned that to the Vienna-based nonprofit that provides R’s leadership, the R Foundation for Statistical Computing, ‘ordinary’ means something quite different to the lay person. Ordinary members provide guidance and direction for the R Project for Statistical Computing. It’s hard to imagine someone more qualified than Dirk for this task.\n","title":"Dirk Eddelbuettel, the useR! 2014 Interview","type":"posts"},{"content":"","date":"September 22, 2014","externalUrl":null,"permalink":"/tag/finance/","section":"Tags","summary":"","title":"finance","type":"tags"},{"content":"","date":"September 22, 2014","externalUrl":null,"permalink":"/tag/hpc/","section":"Tags","summary":"","title":"hpc","type":"tags"},{"content":"","date":"September 22, 2014","externalUrl":null,"permalink":"/tag/rcpp/","section":"Tags","summary":"","title":"Rcpp","type":"tags"},{"content":"The industry as a whole has been a very large net importer of open source technologies.\n","date":"September 22, 2014","externalUrl":null,"permalink":"/dirk-eddelbuettel-on-open-source-in-finance/","section":"Posts","summary":"The industry as a whole has been a very large net importer of open source technologies.\n","title":"Dirk Eddelbuettel on open source in finance","type":"posts"},{"content":"","date":"September 18, 2014","externalUrl":null,"permalink":"/tag/dataviz/","section":"Tags","summary":"","title":"dataviz","type":"tags"},{"content":"","date":"September 18, 2014","externalUrl":null,"permalink":"/tag/highcharts/","section":"Tags","summary":"","title":"highcharts","type":"tags"},{"content":"At useR! 2014 Ramnath Vaidyanathan gave a tutorial and a presentation on one of his current project, rCharts. This R package allows you to create, customize and share interactive visualizations straight from R. rCharts leverages several existing javascript visualization libraries such as polychart, MorrisJS, NVD3, xCharts, HighCharts, leaflet, and Rickshaw, providing R users a fantastic array of new features and visualization styles.\nrCharts also provides a convenient method to share these charts, either in standalone mode, by using a Shiny application, or even in a blog post using knitR! Continuing with the theme of the conference of R as an interface, Ramnath has succeeded in creating a powerful system to smoothly move between the familiar lattice plotting interface and the best available visualization libraries.\nIn this talk, Ramnath provides a high level overview of rCharts, as well as providing a deep dive into the kinds of visualizations possible with this tool - including the interactive visualizations now at your fingertips. From barcharts to mapping, from static to interactive, Ramnath covers many common use cases with this package, and we encourage you to follow along with the video below using his provided slides from the original talk. Enjoy!\n▶ Ramnath Vaidyanathan- \u0026#34;Interactive Visualizations from R\u0026#34; talk at useR 2014 ","date":"September 18, 2014","externalUrl":null,"permalink":"/interactive-visualizations-from-r-using-rcharts/","section":"Posts","summary":"At useR! 2014 Ramnath Vaidyanathan gave a tutorial and a presentation on one of his current project, rCharts. This R package allows you to create, customize and share interactive visualizations straight from R. rCharts leverages several existing javascript visualization libraries such as polychart, MorrisJS, NVD3, xCharts, HighCharts, leaflet, and Rickshaw, providing R users a fantastic array of new features and visualization styles.\n","title":"Interactive Visualizations from R using rCharts","type":"posts"},{"content":"","date":"September 18, 2014","externalUrl":null,"permalink":"/tag/javascript/","section":"Tags","summary":"","title":"javascript","type":"tags"},{"content":"","date":"September 18, 2014","externalUrl":null,"permalink":"/tag/rcharts/","section":"Tags","summary":"","title":"rcharts","type":"tags"},{"content":"","date":"September 17, 2014","externalUrl":null,"permalink":"/authors/amelia/","section":"Authors","summary":"","title":"Amelia McNamara","type":"authors"},{"content":"","date":"September 17, 2014","externalUrl":null,"permalink":"/tag/grad-school/","section":"Tags","summary":"","title":"grad school","type":"tags"},{"content":"(This is my third post on statistics graduate school. You can read the first, on coursework, the difference between a Masters and a PhD, and proving your worth here, and the second, on advancing to candidacy, writing the dissertation, and having fun in grad school, here.)\nMany of the questions I get when I tell people I\u0026rsquo;m working towards a Ph.D. are related how I pay for it. So, here\u0026rsquo;s the dirt. Most doctoral programs in the STEM fields (science, technology, engineering and math) will not admit a student to their program unless they are also prepared to support the student financially - either until they have completed their dissertation or for a certain number of years. When I was admitted to UCLA, my acceptance came with four years of guaranteed funding, for example. That guarantee meant that I would have some way to pay the bills for at least four years, but as it has turned out I\u0026rsquo;ve only used about two of my guaranteed years and filled in the financial gaps other ways.\nMy first year, I was supported by a first-year fellowship from the UCLA statistics department, which paid my (out-of-state!) tuition and provided me with a stipend to live on, with minimal-strings-attached. By that I mean I wasn\u0026rsquo;t required to complete additional work (e.g. either research or as a teaching assistant) however my funding was contingent on my GPA and a few other academic factors. Because the only requirements were academic, that freed me to focus completely on my studies in my first year. But of course, academic funding of this type is rarely year-round\u0026ndash; the first-year fellowship only paid my stipend during the academic year, leaving the summer uncovered. Luckily, I was chosen as one of UCLA\u0026rsquo;s Chancellor\u0026rsquo;s Prize recipients, which provided a stipend to cover living expenses during my first two summers at UCLA.\nDuring the summer after my first year, I started working part-time through a National Science Foundation grant at UCLA, called Mobilize (you can read more about that project here). By the time my second academic year rolled around, the grant\u0026rsquo;s Principal Investigator had asked me to join the project more officially as a Graduate Student Researcher (GSR). GSRs are often considered more prestigious than teaching appointments, but I have never really understood why. This GSR appointment was a so-called \u0026ldquo;50% appointment,\u0026rdquo; which is the standard appointment for a graduate student. It means that you work 50% time (20 hours a week) in exchange for having your tuition paid and receiving a salary/stipend. The only exception to this 50% or 20 hours per week rule is during the summer, where graduate students are allowed to work 100% or 40 hours per week for the university - although some students get around this rule by having additional jobs off-campus during the school year.\nAlthough having a 50% appointment seems like a good idea, having time in your schedule for a variety of experiences during graduate school can be important to gain skills. For example, last year I chose to reduce my time as a Graduate Student Researcher with the Mobilize project to a 25% appointment to allow me to gain experience as a teaching assistant. (Technically, I am a teaching \u0026ldquo;fellow\u0026rdquo; because I have advanced to candidacy.) By reducing my time with Mobilize, I was able to also have a 25% appointment as a TA with the department, which means that I was a TA for one class each quarter. All of my classes were upper division classes aimed at undergraduate statistics majors, and I typically taught two sections of the course as a TA. The professor for the class would see the entire group three times a week, and then the class would be broken into two to engage in a discussion section led by me. In addition to my hours prepping and in class, I also held office hours and helped professors grade exams which is great experience to gain during graduate school.\nAs an aside, most TAs at UCLA do not grade homework for the classes they teach. Instead, there is an additional student (typically a graduate student, but sometimes an advanced undergraduate) who is hired as a grader. Unlike PhD students, Masters students are not given funding guarantees when they enter the department, so they must either pay their tuition outright or try to find a job as a TA or grader. Students with guarantees (usually PhD-level students) are given first preference for these positions, so Masters students generally end up as graders.\nThe past two summers, I have also been working with an off-campus group called the Communications Design Group as an intern. And this coming year, I will be supported by two university fellowships: the Collegium of University Teaching Fellows fellowship, which will allow me to design and teach my own course, and the Dissertation Year Fellowship, which (much like the first-year fellowship) will allow me to focus specifically on finishing my dissertation this year.\nMy wide variety of funding sources is relatively typical for a graduate student at many universities, and especially the statistics department at UCLA - but this piecemeal approach often sounds strange to people at other schools or in other fields. Sometimes an offer of funding will come directly from your advisor, who will essentially pay you to work in their lab or on your dissertation project year-round. Some departments require everyone to teach most every quarter, and don\u0026rsquo;t offer a lot of alternatives for funding. No matter what, money is yet another place where Ph.D. programs usually differ from Masters programs. All respectable Ph.D. programs should be providing an offer of funding when they admit students, but Masters programs are typically out-of-pocket on the part of the student. Yet another reason to pursue the PhD, in my book!\n","date":"September 17, 2014","externalUrl":null,"permalink":"/paying-for-statistics-grad-school-aka-the/","section":"Posts","summary":"(This is my third post on statistics graduate school. You can read the first, on coursework, the difference between a Masters and a PhD, and proving your worth here, and the second, on advancing to candidacy, writing the dissertation, and having fun in grad school, here.)\n","title":"Paying for statistics grad school (AKA the $$)","type":"posts"},{"content":"","date":"September 15, 2014","externalUrl":null,"permalink":"/tag/cran/","section":"Tags","summary":"","title":"CRAN","type":"tags"},{"content":"At the useR! 2014 conference, without a doubt one of the overriding themes was R’s history, legacy, and future as an interface into the “best of the best” algorithms which were available. Romain Francois’ package, Rcpp11, is very much the spirit and the letter of this ethos and telos of R as an interface. This package is a reimplementation of Dirk Eddelbuettel\u0026rsquo;s Rcpp package with an emphasis on C++ 11’s features. Whether it’s an R user hitting performance limitations, or someone trying to integrate a C/C++ library into their analysis, Romain’s tools are there to help.\nAn R programmer for over a decade, Romain started as an “R first” developer, only later moving to more traditional software engineering languages. In this interview, Romain and I discuss a wide range of topics, from R’s history as an interface for algorithms, through Romain’s experiences in HPC and his collaboration with Hadley Wickham on dplyr. Romain also shares a little bit about his other passion, stand-up comedy.\nVIDEO BREAKDOWN # Please click here to see the video directly on YouTube. Or if you prefer, subscribe to our podcast to get the audio!\nAn Overview of Rcpp11 # At useR! 2014, Romain presented on Rcpp11, (from his abstract):\nRcpp11 is a complete rewrite of the Rcpp library, it takes advantage of C++11 to make the user experience of combining R and C++ even better and more future proof. Rcpp11 was also an opportunity to review the code base of Rcpp, identify mistakes and fix them. Rcpp11 is a smaller, cleaner implementation of the Rcpp api, written with C++11 in mind.\nHe presents an introduction to Rcpp here, in the video below.\n▶ Romain Francois - Interview by DataScience.LA at useR 2014 Collaboration with Hadley Wickham # I have already mentioned using dplyr as my greatest productivity enhancement of 2014. While most people would associate Hadley with dplyr, the pacakge has been the result of a fruitful collaboration between the two of them. Here, Romain discusses what it’s like collaborating with a community leader on such a useful package.\n▶ Romain Francois - Interview by DataScience.LA at useR 2014 Advice to New Developers # Romain provides some targeted and valuable advice to new users and developers starting their journey with R. While discussing the many twists and turns of a programmers journey, he highlights one singular action to both bolster your confidence and increase your understanding of the larger R ecosystem.\n▶ Romain Francois - Interview by DataScience.LA at useR 2014 Stand-up Comedy # Romain was kind enough to provide us with a bit of insight into his parellel existence. When he’s isn’t touring around the world giving tutorials and talks about R/HPC, he is also a stand-up comic. Tune in below to hear his insights on how comedy is suited for him and what it allows him to do:\n▶ Romain Francois - Interview by DataScience.LA at useR 2014 Conclusion # Romain’s technological contributions to our field are many, and his depth of knowledge in how to integrate R with external packages and languages is among the very best. A few weeks ago, Romain found himself speaking to the Budapest R Meetup run by friend of DataScience.LA, Gergely Daróczy. At this meetup, Romain gave an hour-long presentation (in English) with a great deal of detailed information about Rcpp11. After you’ve checked out our interview above, please make sure to catch his presentation, available here.\n","date":"September 15, 2014","externalUrl":null,"permalink":"/interview-with-romain-francois-at-user-2014/","section":"Posts","summary":"At the useR! 2014 conference, without a doubt one of the overriding themes was R’s history, legacy, and future as an interface into the “best of the best” algorithms which were available. Romain Francois’ package, Rcpp11, is very much the spirit and the letter of this ethos and telos of R as an interface. This package is a reimplementation of Dirk Eddelbuettel’s Rcpp package with an emphasis on C++ 11’s features. Whether it’s an R user hitting performance limitations, or someone trying to integrate a C/C++ library into their analysis, Romain’s tools are there to help.\n","title":"Interview with Romain Francois at useR! 2014","type":"posts"},{"content":"","date":"September 15, 2014","externalUrl":null,"permalink":"/tag/rcpp11/","section":"Tags","summary":"","title":"Rcpp11","type":"tags"},{"content":"","date":"September 15, 2014","externalUrl":null,"permalink":"/tag/romain-francois/","section":"Tags","summary":"","title":"romain francois","type":"tags"},{"content":"","date":"September 15, 2014","externalUrl":null,"permalink":"/category/quotes/","section":"Categories","summary":"","title":"Quotes","type":"categories"},{"content":"If I could meet young me I would punch him in the face.\n","date":"September 15, 2014","externalUrl":null,"permalink":"/romain-francois/","section":"Posts","summary":"If I could meet young me I would punch him in the face.\n","title":"Romain Francois","type":"posts"},{"content":"","date":"September 11, 2014","externalUrl":null,"permalink":"/tag/maechler/","section":"Tags","summary":"","title":"maechler","type":"tags"},{"content":"","date":"September 11, 2014","externalUrl":null,"permalink":"/tag/martin/","section":"Tags","summary":"","title":"martin","type":"tags"},{"content":"Martin Maechler is a member of R-Core. This distinction puts him in the very apex of understanding of the R programming language and environment. He is a professor of statistics with wide and varied interests and a leading expert in statistical computing environments. He is also a prolific programmer, evidenced by his maintenance of the officially recommended cluster package for unsupervised clustering and analytics, the lead developer of Emacs Speaks Statistics, and co-maintainer of the Matrix package… plus too many other contributions to list them all here.\nAt useR! 2014 at UCLA, Martin gave an updated version of his useR! 2004 talk in Vienna, focusing on good programming “rules.” In his invited talk (video below), “Good Practices in R programming”, Martin showcases a synthesis of his many years of experience and expertise, from gotchas which may trip you up unexpectedly to fundamentals of good engineering and communication practices. DataScience.LA is fortunate to have been able to record his presentation and provide it to the community at large - watch to find out why!\nMartin focuses on a number of critical facets of programming, both in R and in general. He focuses on the nature of namespaces, and the ‘function’ as the fundamental building block of good idiomatic R code. He then takes a tour through analytics and reproducible research, discussing the modern tools available and how they affect project and code layout and organization. He also provides highlights of what every statistical/mathematical programmer should know about computer arithmetic.\nWhether you are an novice R user or expert R programmer, an analyst or a programmer, there is something for you in this talk… and heeding this advice will help make you a better programmer, statistician, scientist, and member of the R community!\n▶ Martin Mächler\u0026#39;s Keynote Speech from useR 2014 \u0026#34;Good Practices in R Programming\u0026#34; ","date":"September 11, 2014","externalUrl":null,"permalink":"/martin-maechler-invited-talk-at-user-2014-good-practices-in-r-programming/","section":"Posts","summary":"Martin Maechler is a member of R-Core. This distinction puts him in the very apex of understanding of the R programming language and environment. He is a professor of statistics with wide and varied interests and a leading expert in statistical computing environments. He is also a prolific programmer, evidenced by his maintenance of the officially recommended cluster package for unsupervised clustering and analytics, the lead developer of Emacs Speaks Statistics, and co-maintainer of the Matrix package… plus too many other contributions to list them all here.\n","title":"Martin Maechler Invited Talk at useR! 2014 - Good Practices in R Programming","type":"posts"},{"content":"","date":"September 11, 2014","externalUrl":null,"permalink":"/tag/r-core/","section":"Tags","summary":"","title":"r-core","type":"tags"},{"content":"Your R \u0026lsquo;programs\u0026rsquo; should become works of art.\n","date":"September 11, 2014","externalUrl":null,"permalink":"/martin-maechler-on-programming/","section":"Posts","summary":"Your R ‘programs’ should become works of art.\n","title":"Martin Maechler on Programming","type":"posts"},{"content":"","date":"September 10, 2014","externalUrl":null,"permalink":"/tag/business/","section":"Tags","summary":"","title":"business","type":"tags"},{"content":"Last week the LA R meetup featured 5 speakers each highlighting a few things they had found interesting at the useR! 2014 conference (which I had the pleasure of helping to organize this year in Los Angeles). This meetup proved to be interesting even to those of us who attended the conference (which was about half of our audience), simply because there were so many cool things to hear and enough parallel sessions at the conference that no one could possibly capture all of the action.\nEven better than the talks were the slides, which were packed with more goodies to share. Unless you have the superpowers to absorb all that info on the fly and happened to attend the meetup, you might find it useful to skim again through these slides, which we\u0026rsquo;ve posted below. (In fact, due to the sheer awesomeness of the conference, I have 5 more volunteer speakers who would like to share their thoughts in a subsequent LA R meetup!)\nOur first speaker Tim Triche, a Fellow at the USC Norris Comprehensive Cancer Center, started with the 3 approaches to interfaces discussed by John Chambers in his keynote: Rcpp, RLLVM and h2o. He then started to compare R and Python from these and a few other perspectives (e.g. data.frames, visualization, etc.), which included strong opinions and funny critiques of the trailing tool in each particular domain. The slides are embedded right here below, however if you are not reading this post directly on DataScience.LA and you cannot see them, they are also linked here.\nTim Triche - Highlights of useR conference - LA R meetup - Sep 2014 · Speaker Deck Our next speaker Shannon Callan, Founder of Digital Trend Analytics, discussed three useR talks on finance topics which includes useful tidbits on portfolio management and Bayesian change-point analysis - the slides to which are embedded below.\nShannon Callan - Highlights of useR conference - LA R meetup - Sep 2014 · Speaker Deck Next, Juan Natera, Senior Software Engineer at Campus Explorer briefly discussed what I think it is the most important R-related takeaway from 2014: dplyr. If you happen to have been using it for months then join the club; if you heard about it at useR and started using it then you\u0026rsquo;re on your way. But if are not using it - or have not even heard of it, please go try dplyr today\u0026hellip; and you\u0026rsquo;re guaranteed to be more productive tomorrow. Slides from Juan\u0026rsquo;s talk are linked here if you don\u0026rsquo;t see them embedded below.\nJuan Natera - Highlights of useR conference - LA R meetup - Sep 2014 · Speaker Deck Then, Wayne Smith of California State University, Northridge touched on some new R books, R packages of note and a few posters he liked (link here).\nWayne Smith - Highlights of useR conference - LA R meetup - Sep 2014 · Speaker Deck Finally, Arno Vigen, CFO of AireSpring, summarized a few of the business-oriented talks from useR (R Connector to Salesforce.com (links to the useR talk\u0026rsquo;s slides) and R AnalyticFlow (links to PDF of the talk\u0026rsquo;s original slides).\nTo get a feel of the meetup, here are a few pictures courtesy Carl Mullins. Thanks, Carl!\nFinally, big thanks to Amobee (formerly Adconion) for hosting us and for pizza (and special thanks to my friend Mikhail for making all the necessary arrangements).\nStay tuned for another round of talks about the useR conference at the next LA R meetup in November, with an announcement soon!\n","date":"September 10, 2014","externalUrl":null,"permalink":"/la-r-meetup-summary-highlights-from-the-user-2014-conference/","section":"Posts","summary":"Last week the LA R meetup featured 5 speakers each highlighting a few things they had found interesting at the useR! 2014 conference (which I had the pleasure of helping to organize this year in Los Angeles). This meetup proved to be interesting even to those of us who attended the conference (which was about half of our audience), simply because there were so many cool things to hear and enough parallel sessions at the conference that no one could possibly capture all of the action.\n","title":"LA R Meetup Summary: Highlights from the useR! 2014 Conference","type":"posts"},{"content":"","date":"September 8, 2014","externalUrl":null,"permalink":"/tag/biostatistics/","section":"Tags","summary":"","title":"biostatistics","type":"tags"},{"content":"","date":"September 8, 2014","externalUrl":null,"permalink":"/tag/consulting/","section":"Tags","summary":"","title":"consulting","type":"tags"},{"content":"","date":"September 8, 2014","externalUrl":null,"permalink":"/tag/emacs/","section":"Tags","summary":"","title":"emacs","type":"tags"},{"content":"","date":"September 8, 2014","externalUrl":null,"permalink":"/tag/generalized-nonlinear-models/","section":"Tags","summary":"","title":"generalized nonlinear models","type":"tags"},{"content":"Heather Turner is a biostatistician and Associate Fellow at the University of Warwick, as well as a consultant. She is a former editor of the R Journal, was a local organizer for the useR! 2011 conference, and holds a wealth of information about the field of data science, the state of the technology, and the wider community.\nAs an academic with roots outside of the traditional Computer Science pathway, Heather holds fascinating insights on what makes the R programming language interesting and different. She became familiar with R during her PhD studies as an user, and a few years later she developed the Generalized Non-Linear Models package which is still in use today. Our conversation, in the video above, begins with this transition from R user to R developer.\nThere is certainly more in this video than I could mention here - don\u0026rsquo;t miss her new work using Shiny in statistical education and how to best encourage younger generations to enter the field. This is genuinely a must-watch video, full of great information combined with level-headed advice. I hope you find it as enlightening as I have!\nVIDEO BREAKDOWN # Please click here to see the video directly on YouTube. Or if you prefer, subscribe to our podcast to get the audio!\nThe Biostatistics Workflow # Heather’s workflow would seem non-standard to R developers in biostatistics, for a variety of reasons. Her background in developing the CRAN Generalized Non-Linear Models package and her partnership with stricter, software-oriented collaborators set her upon on a slightly different track, for example. Her description of how she accomplishes her work is fascinating - including the fact that she prefers Emacs over RStudio!\n▶ Dr. Heather Turner - Interview by DataScience.LA at useR 2014 Women, Minorities and R # A pleasant surprise at my first R conference (useR! 2014) was the seemingly proportional representation of women and minorities who use R for their work. Computer programming has historically struggled to truly include people of color and women, making it refreshing to see that R and the larger data science community indeed come from a variety of backgrounds. In this part of the video, I ask Heather how we can keep this trend moving towards the positive, along with her thoughts on the Women in R panel hosted at useR! 2014.\n▶ Dr. Heather Turner - Interview by DataScience.LA at useR 2014 The Process of Package Development # One of the great features of R is CRAN, a repository of hundreds of packages providing pre-tested, pre-built, ready to run solutions for issues ranging from model building to data cleansing, interfacing to databases, and everything in between. Because Heather mentioned package development a number of times during the interview, I decided to dive a little deeper to better elucidate the process of package development in R.\n▶ Dr. Heather Turner - Interview by DataScience.LA at useR 2014 Consulting # On top of her academic work, Heather is also quite active as a consultant. For those interested in what the consulting opportunities available to an R superstar can look like, she provides high-level insight into her current work and where these kinds of opportunities can be found.\n▶ Dr. Heather Turner - Interview by DataScience.LA at useR 2014 ","date":"September 8, 2014","externalUrl":null,"permalink":"/heather-turner-the-user-2014-interview/","section":"Posts","summary":"Heather Turner is a biostatistician and Associate Fellow at the University of Warwick, as well as a consultant. She is a former editor of the R Journal, was a local organizer for the useR! 2011 conference, and holds a wealth of information about the field of data science, the state of the technology, and the wider community.\n","title":"Heather Turner, the useR! 2014 Interview","type":"posts"},{"content":"Last week, I got the opportunity to share the highlights from the eyeo festival at the DataVis LA meetup. The eyeo festival is a yearly event held in Minneapolis that draws together artists, technologists, designers, and many more people who work with art and computers. As I said in the meetup description, a lot of the things that caught my eye this year could be described better by the term \u0026ldquo;art\u0026rdquo; than straight \u0026ldquo;data visualization,\u0026rdquo; although there\u0026rsquo;s usually both at the festival. I also really liked that this year\u0026rsquo;s festival emphasized the history of computer art and data visualization. By bringing in Frieder Nake, Lillian Schwartz, and Roman Verostko, the organizers grounded us in the history that started in the 1960s.\nFor my highlights, I called out:\nFrieder Nake \u0026ndash; Would You Do It?\nNicholas Felton \u0026ndash; Too Big to Fail\nAdam Harvey \u0026ndash; The Electromagnetic Spectrum of Counter/Surveillance\nLillian Schwartz \u0026ndash; Computer Films: 1970-present\nCod.Act (André et Michel Décosterd) \u0026ndash; Pendulum Choir\nI wasn\u0026rsquo;t expecting to have time to talk about anything else at the meetup, but I got through all that relatively quickly, so I was able to finish up with a few more highlights I had stashed at the back of my slide deck:\nMike Bostock \u0026ndash; Visualizing Algorithms\nKate Crawford \u0026ndash; Big Data Anxieties\nKim Rees \u0026ndash; The Future of Data\nRoman Verostko \u0026ndash; Algorithmic Leverage\nSantiago Ortiz \u0026ndash; Six Months\nRobert Hodgin \u0026ndash; Default Title\nIf you want to see more about the highlights I called out, see my slides (which are full of all kinds of goodies \u0026ndash; photos, tweets, speaker notes, and links): We were lucky to have Jeff Weakley in attendance at the meetup, and he recorded the talk and put it on YouTube (same video as embedded above). He also took a bunch of gorgeous photos of the crowd, see the few below, or the entire set on Flickr. For these and many more photos (including some great headshots of people who attended the meetup) check out the Flickr set. (If you\u0026rsquo;re jonesing for a gorgeous photo of yourself, Jeff comes to the data science LA meetups pretty frequently. While I can\u0026rsquo;t promise he\u0026rsquo;ll always be in attendance, there\u0026rsquo;s a pretty high probability.)\n","date":"September 5, 2014","externalUrl":null,"permalink":"/datavis-la-meetup-summary-eyeo-festival-highlights/","section":"Posts","summary":"Last week, I got the opportunity to share the highlights from the eyeo festival at the DataVis LA meetup. The eyeo festival is a yearly event held in Minneapolis that draws together artists, technologists, designers, and many more people who work with art and computers. As I said in the meetup description, a lot of the things that caught my eye this year could be described better by the term “art” than straight “data visualization,” although there’s usually both at the festival. I also really liked that this year’s festival emphasized the history of computer art and data visualization. By bringing in Frieder Nake, Lillian Schwartz, and Roman Verostko, the organizers grounded us in the history that started in the 1960s.\n","title":"DataVis LA Meetup Summary: Eyeo Festival Highlights","type":"posts"},{"content":"","date":"September 5, 2014","externalUrl":null,"permalink":"/tag/eyeo/","section":"Tags","summary":"","title":"eyeo","type":"tags"},{"content":"","date":"September 5, 2014","externalUrl":null,"permalink":"/tag/summary/","section":"Tags","summary":"","title":"summary","type":"tags"},{"content":"","date":"September 4, 2014","externalUrl":null,"permalink":"/tag/ggvis/","section":"Tags","summary":"","title":"ggvis","type":"tags"},{"content":"","date":"September 4, 2014","externalUrl":null,"permalink":"/tag/interfaces/","section":"Tags","summary":"","title":"interfaces","type":"tags"},{"content":"At useR! 2014, one of the most anticipated presentations was Joe Cheng’s Shiny: R made interactive. It was but one of the of a fantastic series of talks by RStudio representatives; check out Winston Chang’s ggvis: Interactive graphics in R for another great talk. Shiny begins with your R code and ends with the customer’s view: an interactive, browser-accessible application. This enables you to place a great deal of power into your users’ hands to customize their view and provides a variety of new and innovative ways to interact with the data, such as by tuning parameters and focusing on sub-populations. Essentially, Shiny functions by allowing you to “wire up” up a reactive application, greatly speeding up the development of a data and analysis based web app.\nShiny comes out of the box with a number of standard industry UI widgets such as date inputs, file inputs, sliders, buttons - the standard widget toolbox. Shiny also leverages a simple panel-based layout manager which, in concert with the widgets, allows for a fluid development experience.\nVideo Highlights # If you’re interested in bypassing Joe Cheng’s fantastic explanation and you want to cut right to his first demo, he presents a web application for k-means clustering on the iris data set:\n▶ Joe Cheng\u0026#39;s rShiny at useR! 2014 Joe’s next demo involves a slick mapping component. Here you begin to see how Shiny apps have the potential to leverage intriguing data sources and provide first rate visualizations beginning only with ZIP codes and a limited amount of R code:\n▶ Joe Cheng\u0026#39;s rShiny at useR! 2014 He then shows an amazing example of what an R developer with no previous web development experience can build using Shiny.\n▶ Joe Cheng\u0026#39;s rShiny at useR! 2014 For his final demo, Joe embeds Shiny interactive documents using R markdown and combines them with Yihui Xie’s knitR. This is demoed in the video below with an application which parses the logs for RStudio’s CRAN mirror and visualizes package download trends.\n▶ Joe Cheng\u0026#39;s rShiny at useR! 2014 Shiny is a fun and easy way to widen the audience interacting with your data analyses. It leverages fantastic modern programming paradigms to provide a system with sane defaults and powerful components to interface the web with R. Enjoy!\n","date":"September 4, 2014","externalUrl":null,"permalink":"/shiny-r-made-interactive-user-2014/","section":"Posts","summary":"At useR! 2014, one of the most anticipated presentations was Joe Cheng’s Shiny: R made interactive. It was but one of the of a fantastic series of talks by RStudio representatives; check out Winston Chang’s ggvis: Interactive graphics in R for another great talk. Shiny begins with your R code and ends with the customer’s view: an interactive, browser-accessible application. This enables you to place a great deal of power into your users’ hands to customize their view and provides a variety of new and innovative ways to interact with the data, such as by tuning parameters and focusing on sub-populations. Essentially, Shiny functions by allowing you to “wire up” up a reactive application, greatly speeding up the development of a data and analysis based web app.\n","title":"Shiny: R made interactive @ useR! 2014","type":"posts"},{"content":"Today I am happy to announce that we are expanding our partnership on several fronts with my friend Subash D’Souza, a leader of the Big Data community in LA.\nDuring the past few years I have been organizing several data science meetups in Los Angeles (LA R, DataVis LA, Data Science/Machine Learning) and Subash has provided the same kind of leadership on the big data front (LA Hadoop, HBase, LA Big Data). While I think of big data as primarily technology and of data science as a blend of statistics and technology (and not only for large data sets), there is clearly a significant overlap between these two fields. Considering this overlap, we have already been collaborating for some time regarding these various meetups, e.g. organizing joint events or speaking at one another’s meetups.\nAt the Big Data Camp LA (which Subash orchestrated this past June), I also had the pleasure to be involved with running the Data Science Track (selecting the speakers and moderating the sessions). This was a free event which lasted all day and if you missed it, you can find the slides from the Data Science Track talks here (there were 2 other tracks, Hadoop and NoSQL).\nSo, today we are pleased to announce that Subash is joining the DataScience.LA Team and will contribute to DataScience.LA in various ways on a variety of Big Data topics.\nWe are also announcing that two of the Big Data Meetups organized by Subash that have significant common ground with data science (the LA Big Data and LA Spark user groups) will be considered our partner meetups. For example, we’re looking forward to advertising their events and vice versa.\nFinally, DataScience.LA is looking forward to getting involved with the Data Science Track for the Big Data Camp LA next year.\n","date":"September 3, 2014","externalUrl":null,"permalink":"/announcing-partnership-with-la-big-data-community/","section":"Posts","summary":"Today I am happy to announce that we are expanding our partnership on several fronts with my friend Subash D’Souza, a leader of the Big Data community in LA.\nDuring the past few years I have been organizing several data science meetups in Los Angeles (LA R, DataVis LA, Data Science/Machine Learning) and Subash has provided the same kind of leadership on the big data front (LA Hadoop, HBase, LA Big Data). While I think of big data as primarily technology and of data science as a blend of statistics and technology (and not only for large data sets), there is clearly a significant overlap between these two fields. Considering this overlap, we have already been collaborating for some time regarding these various meetups, e.g. organizing joint events or speaking at one another’s meetups.\n","title":"Announcing Partnership with LA Big Data Community","type":"posts"},{"content":"","date":"September 3, 2014","externalUrl":null,"permalink":"/tag/big-data-camp/","section":"Tags","summary":"","title":"big data camp","type":"tags"},{"content":"","date":"September 3, 2014","externalUrl":null,"permalink":"/tag/partners/","section":"Tags","summary":"","title":"partners","type":"tags"},{"content":"","date":"September 2, 2014","externalUrl":null,"permalink":"/tag/data-cleaning/","section":"Tags","summary":"","title":"data cleaning","type":"tags"},{"content":"","date":"September 2, 2014","externalUrl":null,"permalink":"/tag/data-wrangling/","section":"Tags","summary":"","title":"data wrangling","type":"tags"},{"content":"","date":"September 2, 2014","externalUrl":null,"permalink":"/authors/leigh/","section":"Authors","summary":"","title":"Leigh Arino de la Rubia","type":"authors"},{"content":"As someone who has spent a significant amount of time ‘data wrangling’ outside the field of data science, I found this recent New York Times article both highly interesting and nothing new - I’ve seen the same issue discussed in a variety of fields with similar automation solutions presented each time. Granted, there are fields where this janitorial drudgery is absolutely necessary, such as molecular biology. As a scientist working on researching mRNA expression you certainly have to keep RNase-free conditions, meaning you have to ‘clean up’ your bench in a more detailed way than you’d ever imagine before you can run your first PCR cycle. This includes specific protocols for washing your dishes several times with special water before you even start because if you don’t, you’re not likely to get any bands in your electrophoresis gel and you’ll lose several days (or weeks!) of work.\nThere are other fields where you can get by with less prep if you so choose, but it makes the later process less enjoyable or more risky, like cooking. A chef needs to prep her ingredients, however whether all is mise en place before one turns on the stove is a decision left to the individual. Between these extremes there are many places to look for insight, and there are two types of wrangling that parallel the problems facing data scientists today: the struggles of the qualitative researcher and trials of the knitter. I offer them both here as food for thought for the data science community, from the novice starting to explore data science to the seasoned expert who uses their R toolbox daily.\nI’ve taught educational research to a variety of students, from psychology undergraduates looking for a semester project to doctoral-level graduate students using RStudio in my research methods course. Many times these willing students would walk into my office with a preliminary idea for a quantitative project and they’d leave excited instead about a qualitative (or mixed-methods) endeavor. Why? Because it teaches them how to plan, gather, and interact with a large volume of data in a way free of their research preconceptions rather than helping them pseudo-engage in backfilling a quantitative research question with simple answers. These students wanted to learn how to conduct their own research in preparation for a thesis/dissertation - and I was prepared to teach them on-the-job skills. But isn’t qualitative research just hearsay and conjecture? Isn’t it a lesser form of research? (For reference, it’s just as methodical as quantitative and certainly has its place in the sun.)\nThere is often only one researcher involved in this kind of data collection \u0026amp; analysis, so how much can we really learn from the endeavor of qualitative research? I argue that we can learn more about ourselves as human researchers, how we interact with data, and how we bias our results from qualitative research - then we can use this information to improve our quantitative research processes and get even more out of later data than we thought possible.\nThere are software products to address these data ‘problems’ in qualitative research, from transcription to analysis, and there have been for many years. Unsavory tasks ahead? Automate the process of moving from audiotape to transcript so you can get to the ‘real stuff’ quicker. Not sure whether there are great connections to support your theory-building? Try some analysis software to see what pops out. Each time I’d caution my students against using these ‘time-savers’, however, because of what you lose in between the lines. You may gain time and lower your frustration level, but you lose your thoroughness of interaction with a set of data in a way which increases your insight and intuition while predicting and solving other issues down the road. You miss the building of excitement that can happen while you’re waiting to get to the patterns and find the ‘answers’. You get 70% (at most) of what the data has to say with this approach. When you invest this time rather than automate it away, you get that other 30% of information back which can make all the difference when you’re presenting the results to others and trying to stand out in a sea of researchers. Struggling through the mire is very useful in qualitative research, and teaches the researcher the time-tested ways of thoroughly knowing your data, correcting for human bias through procedures, and finding emergent patterns in a large volume of information. You can’t buy better experiences than those in the long-term.\nThe second bellwether comes from a different field entirely, and is a hobby for most who engage. I’m a longtime knitter and crocheter and I’ve taught a few hundred people to knit or crochet over the years. Each time I teach the yarn arts, I ask the student to bring a skein of yarn and a crochet hook to our session. Our first task isn’t to chain or learn a beginning stitch, though, it’s to make a ball of yarn from the skein. It’s part of the planning process, I say. We have to get to know our yarn first, to predict problems down the line and find out if this specific yarn is the best choice for what we’re making. We need to feel the yarn in our fingers, familiarize ourselves with the weight, drape, and shape of the strand. How does it split? Is that a feature or a problem? Are there any unexpected knots or frays hiding inside that skein? We should know this before we start our time-intensive alchemy from a mess of yarn to a finished scarf. Sure, we can make a scarf without this metadata… but it most likely won’t be as smooth of a working-up process or result in as polished of a finished product. Yet, the balling of yarn is seen by many as the janitor-work of the crafting world. There are many ways of automating the process with a ball winder or yarn caker - all of which omit the touching of the yarn by human fingers to gauge its usefulness and character while informing the rest of the process. I find this choice a mistake, and again to be short-sighted.\nI hope these examples help you think more deeply about your own data cleaning issues and/or opportunities, and consider what you may lose or gain from automating these processes in your work. Happy Wrangling, whether it happens with thorny csv files or soft bamboo yarns!\n","date":"September 2, 2014","externalUrl":null,"permalink":"/the-importance-of-janitorial-work-in-research/","section":"Posts","summary":"As someone who has spent a significant amount of time ‘data wrangling’ outside the field of data science, I found this recent New York Times article both highly interesting and nothing new - I’ve seen the same issue discussed in a variety of fields with similar automation solutions presented each time. Granted, there are fields where this janitorial drudgery is absolutely necessary, such as molecular biology. As a scientist working on researching mRNA expression you certainly have to keep RNase-free conditions, meaning you have to ‘clean up’ your bench in a more detailed way than you’d ever imagine before you can run your first PCR cycle. This includes specific protocols for washing your dishes several times with special water before you even start because if you don’t, you’re not likely to get any bands in your electrophoresis gel and you’ll lose several days (or weeks!) of work.\n","title":"The Importance of 'Janitorial Work' in Research","type":"posts"},{"content":"","date":"September 2, 2014","externalUrl":null,"permalink":"/tag/raffles/","section":"Tags","summary":"","title":"raffles","type":"tags"},{"content":"We have a few pieces of swag that we are raffling off to the community at our upcoming meetup events. Our stash consists of 25 data-oriented books, primarily from O’Reilly plus 25 T-shirts from useR! 2014 conference and Revolution Analytics (see picture above). Did we get your attention?\nIn order to participate in the raffles, you will need to sign up for our premium mailing list. In addition to the raffles, those on the premium mailing list will receive priority announcements for meetups and important community news as it happens, plus much, much more.\nBefore each meetup (LA R, DataVis or Machine Learning/Data Science), we’ll select the names of that event’s winners and will start calling each name, one-by-one. When your name is called (and you are present at the meetup), you will have your choice of one raffle item (book or T-shirt) until the day’s items have all been claimed. We will have four winners at each meetup, i.e. 2 books and 2 shirts will be given away each time. Are you one of the lucky names who will be called? Read more below to see our algorithm for determining the raffle winners.\nThe drawing will consist of sampling names from the premium mailing list, however the relative weights (i.e. your chance of winning) will not be equal, but instead a linear decreasing function of your signup time rank so that the first person who signed up will have 3x the chance compared to the last person on the list, using chronological order. As an example, if at the time of a drawing the list has 100 names, #1 gets weight 3.00, #2 gets weight 2.98, and so on continuously decreasing, #100 gets weight 1.00.\nTo increase the fun and your chances of winning, you can double your relative weight by providing a link to your LinkedIn profile when you sign up for the premium list (or later, by updating your preferences if you have already signed up).\nWith R and dplyr we can do all this with a few lines of code:\nlibrary(dplyr) read.csv(\u0026#34;members_export.csv\u0026#34;) %\u0026gt;% mutate(pcrnk = percent_rank(CONFIRM_TIME), wdbl = ifelse(LinkedIn!=\u0026#34;\u0026#34;,2,1), w = (3-2*pcrnk)*wdbl) %\u0026gt;% sample_n(30, weight = w) %\u0026gt;% mutate(name = paste(First.Name,Last.Name)) %\u0026gt;% select(name) Source: gist 7850049 by szilard\nYou can see, to maximize your chances of winning, sign up to the premium mailing list as soon as possible and provide your LinkedIn profile. There are over 50 fantastic items waiting for our lucky winners! Sign up here now and increase your chances today!\n","date":"September 2, 2014","externalUrl":null,"permalink":"/win-books-and-t-shirts-in-our-datascience-la-meetup-raffles-powered-by-r-and-dplyr/","section":"Posts","summary":"We have a few pieces of swag that we are raffling off to the community at our upcoming meetup events. Our stash consists of 25 data-oriented books, primarily from O’Reilly plus 25 T-shirts from useR! 2014 conference and Revolution Analytics (see picture above). Did we get your attention?\n","title":"Win Books and T-shirts in Our DataScience.LA Meetup Raffles, Powered by R and dplyr!","type":"posts"},{"content":"","date":"September 1, 2014","externalUrl":null,"permalink":"/tag/animate/","section":"Tags","summary":"","title":"animate","type":"tags"},{"content":"In science consensus is irrelevant. What is relevant is reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.\n\u0026ndash; Michael Crichton\nYihui Xie intended to make it easier to do his homework, but instead found himself tackling one of the greatest problems in modern science: the reproducibility of results.\nWhat is Reproducibility?\nReproducibility is one of the main principles of the scientific method, and refers to the ability of a test or experiment to be accurately reproduced, or replicated, by someone else working independently.\nThrough his work on the knitR package, he has assembled a toolchain which allows the user to produce beautiful, ready-to-distribute documents containing a whole, self-supporting, and reproducible analysis. By leveraging powerful constructs such as adaptive, nearly transparent caching, Yihui has removed the barriers which have prevented many practitioners from fully addressing reproducibility in their work. Reproducible documents are now flexible, powerful, and fast. No single R package has impacted my personal workflow in the past decade as deeply as knitR has, moving the task of updating analyses with more current data from a chore to a pleasure.\nPreviously, presenting a data story in a flowing document required completing an array of invisible processes plus the usual copy/paste engineering in order to create an incomplete picture of the analysis and code used to produce it. Now, a single living knitR project can now be created in-line and shared in its entirety, as either a PDF, a git repository, or even a living Shiny document, giving my audience a nearly total understanding of the process by which I arrived at my solution.\nIn this interview, Yihui discusses how he came to the R programming language and how he set about building knitR. He also mentions the great momentum and energy of the R community in China, and what he’s currently focused on at RStudio.\nVideo Breakdown # Please click here to see the video directly on youtube. Or if you prefer, subscribe to our podcast to get the audio!\nInspiration for Building knitR # We began this conversation by asking about the knitR package and it’s origins. How does one wake up and decide to build the single best implementation of Knuth’s literate programming? The same way you eat an elephant, one bite at a time.\n▶ Yihui Xie - Interview by DataScience.LA at useR 2014 R Community in China # Yihui isn’t only a world class software developer and visionary, he’s also helped organize the R community in China since 2006. The Chinese R conference, started in 2008, has also grown in stature and attendance as the community has grown. The 7th R China conference, held on May 24th and 25th, 2014 and hosted at Yihui’s alma mater, Renmin University in Beijing, had over twice as many attendees as the global useR! 2014 conference in Los Angeles, for example.\n▶ Yihui Xie - Interview by DataScience.LA at useR 2014 Advice to new programmers # Yihui provides some fantastic advice for new programmers on how to get involved in R. His advice transcends just R development however, and should be seen as great advice for anyone wanting to get involved in programming and technology. He advises a non-traditional path to mastery, focusing on the tools and techniques that make R visually and experientially appealing.\n▶ Yihui Xie - Interview by DataScience.LA at useR 2014 ","date":"September 1, 2014","externalUrl":null,"permalink":"/yihui-xie-the-user-2014-interview/","section":"Posts","summary":"In science consensus is irrelevant. What is relevant is reproducible results. The greatest scientists in history are great precisely because they broke with the consensus.\n– Michael Crichton\nYihui Xie intended to make it easier to do his homework, but instead found himself tackling one of the greatest problems in modern science: the reproducibility of results.\n","title":"Yihui Xie, the useR! 2014 Interview","type":"posts"},{"content":"I wanted to do my homework more conveniently\n","date":"September 1, 2014","externalUrl":null,"permalink":"/yihui-xie-on-the-birth-of-knitr/","section":"Posts","summary":"I wanted to do my homework more conveniently\n","title":"Yihui Xie on the birth of knitR","type":"posts"},{"content":"I am excited to announce that we are starting a new meetup group - Python Data Science LA - to serve the needs of the LA data science community using Python!\nAt an earlier Data Science/Machine Learning meetup we conducted an informal survey on tools for Data Science which showed that a large majority of data scientists use R or Python, with R taking a slight lead. (I know, those attending the meetups bias the results towards open source tools.)\nLos Angeles has already a well established R meetup, which was in fact the first Data Science meetup in LA (and founded/run by yours truly since 2009). For the Python community, there are a few meetup options in LA (with Socal Python the most popular), however their focus lies with general software topics rather than data science, while in some other cities, there are already separate “Python Data Science” meetups. Since the aim of DataScience.LA is to serve the entire Los Angeles Data Science community, we are pleased to launch a Python Data Science meetup in LA!\nWe are planning the first Python Data Science LA event in November with 4-5 short talks about the Python Data Science ecosystem (e.g. NumPy, SciPy, pandas, matplotlib, scikit-learn, NetworkX etc.). If you’d like to speak at this meetup, please contact us and don’t forget to provide a topic, considering that we’re looking for 4-5 speakers.\nWe are also looking for a venue to host the Python Data Science LA meetups, keeping in mind that the ideal host will be a company using Python (and preferably for data analysis), is able to host us for free, plans to provide food and drinks, and can host at least 50, or preferably 100 people. Please contact us if your company is interested in hosting a meetup.\nFinally, if you’re interested in attending the meetups, please sign up for the Python Data Science LA meetup. You can also sign up for our premium mailing list that will provide priority RSVPs for our meetups and makes you eligible for our raffles, and much, much more.\n","date":"August 29, 2014","externalUrl":null,"permalink":"/announcing-the-new-pydata-la-meetup-group/","section":"Posts","summary":"I am excited to announce that we are starting a new meetup group - Python Data Science LA - to serve the needs of the LA data science community using Python!\nAt an earlier Data Science/Machine Learning meetup we conducted an informal survey on tools for Data Science which showed that a large majority of data scientists use R or Python, with R taking a slight lead. (I know, those attending the meetups bias the results towards open source tools.)\n","title":"Announcing the New Python Data Science LA Meetup Group","type":"posts"},{"content":"Matt Dowle holds a wealth of knowledge and is worth listening to for a variety of reasons. He has worked for some of the world’s largest financial organizations, holding positions at the focal point between money and data. He has been programming in R for well over a decade. He has lived the open source mantra, by not only consuming open source software but also releasing his own community contributions to CRAN. On top of those accomplishments, he’s also quite funny and spins a heck of a good yarn.\nAt useR! 2014 Matt Dowle gave a presentation on his data.table package, including the history and context behind its development. For the uninitiated, data.table provides extensions of the R data.frame object which allow fast aggregation of large data (e.g. 100GB in RAM), fast ordered joins, fast add/modify/delete of columns by group using no copies at all, list columns and a fast file reading.\nHowever, to say that Matt Dowle’s presentation was only about data.table is disingenuous. Matt’s presentation is a fantastic autobiographical snippet of how one practitioner moved from commercial tools for data science, where he was at the mercy of commercial entities for support and a blessing, towards open source tools where he has been in control of his destiny. Or, as he puts it:\n\u0026ldquo;\u0026hellip; if I don\u0026rsquo;t know how to fix it, I can hire somebody else to fix it for me.”\nIn this talk he gives examples of how he started out as a new user with preconceptions about how the world should work, then quickly learned that the tools simply didn’t work the way he wanted. With open source software, however, he could bend the tools to his will. He also discusses his choice to distribute his R package under an open source license, feeling a strong responsibility to do so as both a user and developer.\nMatt provides a number of great code examples which illuminate the fundamental ideas that structured the data.table package, in addition to benchmark examples of how operations which previously took hours to run instead complete in seconds with this package. Finally, he covers one of the great bugaboos for new (and experienced) R programmers: the dismal speed of importing a CSV file. He also discusses the incantations we are encouraged to learn in order to combat the issue, and his alternate implementation (fread, or ‘fast/friendly/file or finagle read’) which outpaces even the most elaborately crafted read.csv parameters. His slides are also available for download here. Enjoy!\n▶ Matt Dowle\u0026#39;s \u0026#34;data.table\u0026#34; talk at useR 2014 ","date":"August 28, 2014","externalUrl":null,"permalink":"/matt-dowles-data-table-talk-from-user-2014/","section":"Posts","summary":"Matt Dowle holds a wealth of knowledge and is worth listening to for a variety of reasons. He has worked for some of the world’s largest financial organizations, holding positions at the focal point between money and data. He has been programming in R for well over a decade. He has lived the open source mantra, by not only consuming open source software but also releasing his own community contributions to CRAN. On top of those accomplishments, he’s also quite funny and spins a heck of a good yarn.\n","title":"Matt Dowle's data.table Talk from useR! 2014","type":"posts"},{"content":"","date":"August 28, 2014","externalUrl":null,"permalink":"/tag/read-csv/","section":"Tags","summary":"","title":"read.csv","type":"tags"},{"content":"(Note: This is my second installment on grad school, you can read the first part on coursework, the difference between a Master\u0026rsquo;s and a Ph.D., and proving your worth, here.)\nAdvancement to candidacy # After finishing coursework, the next step in our PhD program is to \u0026ldquo;advance to candidacy.\u0026rdquo; This is the formal process of proposing your dissertation research. Our department has a diverse set of research interests, both theoretical and applied, so dissertations can take a variety of forms. This is yet another difference between Master\u0026rsquo;s and PhD projects\u0026ndash; typically, Master\u0026rsquo;s projects are more directed by faculty members and PhD projects are chosen primarily by the students themselves. Either way, the process begins with a conversation between the student and his/her advisor.\nBefore an in-person proposal of the project, the candidate prepares a written report outlining their research plan (called a \u0026ldquo;prospectus\u0026rdquo;) that is sent out to a committee of faculty members chosen by the student. This committee is led by a departmental professor, or “chair” who helps guide your project plus three other professors who will evaluate the quality of your project and provide guidance during the process. Of these three professors, one must be outside the statistics department. (One of the bigger challenges in forming any dissertation committee is that for faculty members to serve, they must be full, tenured professors in the department and it\u0026rsquo;s important to note that not all professors teaching for a department have attained this status.)\nIn my case, my doctoral committee is relatively large. My advisor is Mark Hansen, and my two departmental members are Mark Handcock and Song Chun Zhu. Then, because my work is quite computational, I\u0026rsquo;ve added two computer scientists: Todd Millstein and Alan Kay. Because I\u0026rsquo;m focused on statistics education I also included Rob Gould.\nOnce you\u0026rsquo;ve officially created your committee and written the prospectus, you have a formal presentation of your dissertation proposal. This usually means creating a Beamer presentation, dressing up in a suit or other formal wear, and sweating your way through an anxious presentation to be followed by a set of questions from your committee. The committee then deliberates in private, discussing your presentation and perusing your academic record. Following this deliberation they inform you of whether you have passed (AKA \u0026ldquo;advanced to candidacy\u0026rdquo;) or not. Note that in practice, everyone passes the proposal since your advisor wouldn\u0026rsquo;t allow you to move forward in presenting your proposal unless you were considered ready to pass.\nStudents typically advance to candidacy at the end of their second year of study or perhaps at the beginning of their third. I\u0026rsquo;ve completed a late proposal (after my third year) but since I\u0026rsquo;m now on track to finish in 2015, I do feel as though I\u0026rsquo;ve caught back up. In many PhD programs, once you\u0026rsquo;ve advanced to candidacy the department awards you a Master\u0026rsquo;s degree - and if you’d like you could decide to leave with that degree alone, keeping in mind that you originally had your sights on the PhD. However, as of last year the UCLA statistics department has stopped this practice, perhaps because of the increasing popularity of the Master’s degree as its own program and the desire to prevent PhD students from ‘dropping out’ along the way with a Master’s. Instead, students who achieve candidacy recieve a certificate declaring the student as a \u0026ldquo;C. Phil,\u0026rdquo; or Candidate of Philosophy.\nMy Candidate of Philosophy certificate I found this C. Phil process to be surprisingly motivating, since in order to get a Master\u0026rsquo;s degree I would still have to write a Master\u0026rsquo;s thesis. Facing a massive, complex writing project in either case, it seems like it wouldn\u0026rsquo;t be much more work to get the PhD.\nWriting and defending # Once you have advanced to candidacy you can begin to work on your project with your committee\u0026rsquo;s blessing. At this point (where I currently am) you \u0026lsquo;just\u0026rsquo; need to finish your dissertation. This requires lots of thinking, writing, consulting with committee members, and thinking again. It\u0026rsquo;s truly the hard work of the PhD, since there are few external deadlines given to you and it\u0026rsquo;s now up to you alone to both finish the research and write up your findings.\nEventually, you will defend this line of research in the formal \u0026ldquo;dissertation defense,\u0026rdquo; when your committee will evaluate your project and decide whether your work is acceptable or whether you will need to complete more work to graduate. Again, this is hopefully a formality since your advisor will let you know if you are ready to defend or not. While you may emerge from the defense with many required revisions, you should never completely \u0026ldquo;fail.\u0026rdquo; (At least that\u0026rsquo;s what I tell myself!)\nThe fun stuff # Of course, all work and no play makes graduate school a hellish ordeal! So, one of the best things about the UCLA statistics department is that we always try to have fun. Many times when visitors have come to the department they\u0026rsquo;ve remarked about how social and \u0026rsquo;normal\u0026rsquo; we all are\u0026ndash; not at all like the stereotypical Stats Nerd. It\u0026rsquo;s hard to say whether that stereotype actually exists, but I can certainly say with confidence that we have built a great community.\nFor example, at least once a year a grad student designs a new departmental t-shirt, which we all wear with pride.\nStatistics department t-shirts in the wild! We also have a yearly tradition of competing against other science graduate students in a keg race. Typically, the Physics department is able to finish their keg first, but this year the joint statistics and math team was victorious!\nStatistics and math victorious in the 2014 keg race Beyond these more formal events, we are often found hanging out together. Until this year, the graduate student offices were arranged to open into the same lounge area, where people would gather informally almost every day. We\u0026rsquo;ve just had a big change in the office situation, but I\u0026rsquo;m sure we\u0026rsquo;ll still find a way to be social. Nearly every week you can find students going to happy hour or on a hike.\nFinally the alumni of the department stay in touch and are quite friendly. Our alumni list includes familiar names in statistics like Nathan Yau of FlowingData, Jake Porway of DataKind, and Roger Peng of Simply Statistics, as well as graduates pursuing tenure-track professorships, like Mine Çetinkaya-Rundel at Duke and Andrew Bray at Mount Holyoke.\nDo you have more questions about statistics graduate school? Ask me in the comments!\n","date":"August 27, 2014","externalUrl":null,"permalink":"/part-2-of-statistics-grad-school-what-is-it-really-like/","section":"Posts","summary":"(Note: This is my second installment on grad school, you can read the first part on coursework, the difference between a Master’s and a Ph.D., and proving your worth, here.)\nAdvancement to candidacy # After finishing coursework, the next step in our PhD program is to “advance to candidacy.” This is the formal process of proposing your dissertation research. Our department has a diverse set of research interests, both theoretical and applied, so dissertations can take a variety of forms. This is yet another difference between Master’s and PhD projects– typically, Master’s projects are more directed by faculty members and PhD projects are chosen primarily by the students themselves. Either way, the process begins with a conversation between the student and his/her advisor.\n","title":"Part 2 of Statistics Grad school - What is it really like?","type":"posts"},{"content":"“At one point, the only source for daily R news was David Smith’s blog.” \u0026ndash; Szilard Pafka\nDavid Smith is an integral part of the R community. His background in computational statistics goes back to the early 90s . David has worked at Revolution R, one of the leading R companies, for nearly a decade. His name has been included in lists such as:\nTop 20 influencers in big data 30 most influencial Data Scientists on Twitter #3 Best Big Data Twitter account Top Big Data Executives and Experts to Follow on Twitter David Smith is a powerhouse of connections, information, and knowledge not only about the R community but also the Data Ecosystem as a whole. After having the privilege of a lovely conversation regarding the state of our industry at useR! 2014, however, I can tell you that above all things, David Smith cares about the happiness and well-being of the community he has seen flourish around him in recent years.\nIn this wide-ranging interview, David and I talk about how he became involved in mathematical software, his transition from a statistician to his current role as Chief Community Officer at Revolution Analytics, and what that role entails.\nVideo Breakdown # Please click here to see the video directly on youtube. Or if you prefer, subscribe to our podcast to get the audio!\nBirth of the Revolution Blog # We began this interview by discussing his blog for Revolution Analytics, and how it has grown into an amazing community resource. For a long time, this blog has been one of the most actively updated, thoughtfully curated centerpieces of R community discussion on the internet.\n▶ David Smith - Interview by DataScience.LA at useR 2014 Journey of R from Academia to Industry # We then recounted the long journey that R has traveled from a niche language used primarily in the academy by a select group of computational statisticians to the wide appeal it has currently found in industry. We talk about how originally insurance and pharma were the only two sectors interested in data, yet since the birth of Google\u0026rsquo;s business models and data-driven decision making most future looking businesses now see their futures intrinsically tied to their company\u0026rsquo;s ability to understand the relevance of the signals in their own data.\n▶ David Smith - Interview by DataScience.LA at useR 2014 Big Data, Hype and Reality # There is probably no term more overused in recent years than Big Data. A recent series of articles highlight how the \u0026ldquo;Big Data\u0026rdquo; hype cycle is now entering the \u0026ldquo;trough of disillusionment.\u0026rdquo;\nIs this trend our reality, though? Companies have worked with large data sets for a very long time, however these explorations were very regimented and controlled rather than the comprehensive data science processes known today. What has changed is the ability for individual analyst to manage large scale data-sets in an ad-hoc format and store their data at a much lower cost. Since the birth of Hadoop, the marginal cost of storage has decreased such that what was previously only available to companies at the highest end is now available to everyone.\n▶ David Smith - Interview by DataScience.LA at useR 2014 Data Science is Sexy! # This takes us to the loaded term \u0026ldquo;Data Scientist\u0026rdquo;. Whether you are a fan of the term or not (and David is), the fact is that a lot of folks continue to call \u0026ldquo;Data Scientist\u0026rdquo; the sexiest job of the 21st century. Even at the same time that people are wondering if it\u0026rsquo;s already time to kill the \u0026ldquo;Data Scientist\u0026rdquo; title. In the interview we discuss the importance of the term Data Scientist as a way of conveying the multidisciplinary nature of the job, even if this job is really what statisticians have been doing all along.\n▶ David Smith - Interview by DataScience.LA at useR 2014 The Many Sub-Communities in R # R is a language for programming with data, but you\u0026rsquo;d be hard pressed to find an industry that isn\u0026rsquo;t alive with data in 2014. In this section of the interview, we discuss the infrastructure that the R community has gotten right, from CRAN to the cross-pollination of ideas across industries and from biostatistics to marketing. We outline how statistical models used in one field can find new life in completely unrelated fields, both from the standpoint of simple model application and how ideas spur new and exciting innovations.\n▶ David Smith - Interview by DataScience.LA at useR 2014 R as a Thought Leader in Reproducibility # There are plenty of horror stories about how a researcher\u0026rsquo;s inability to reproduce scientific findings have hurt the science field as a whole. Some studies show that as little as 10% of published studies in scientific journals are truly reproducible. While this problem has plagued science since the birth of the scientific process, the past few years have seen the problem brought to the forefront. From the Economist\u0026rsquo;s stance that the problem may lie in the current funding model to the journal Nature focusing an entire special issue on the topic, it\u0026rsquo;s clear the problem has finally begun to draw the attention it deserves. From the non-reproducibility of syphillis bacteria to cold fusion, this problem with reproducibility won\u0026rsquo;t simply go away.\n▶ David Smith - Interview by DataScience.LA at useR 2014 In this section, David discusses what got him interested in the topic, a presentation at the BioConductor conference regarding a cancer study that was non-reproducible. The tooling wasn\u0026rsquo;t available for peer review, the code wasn\u0026rsquo;t available, the data wasn\u0026rsquo;t available, nothing was available. R has a number of interesting projects trying to address this reproducibility issue, from Yihui\u0026rsquo;s knitR to Revolution\u0026rsquo;s own Reproducible R Toolkit (RRT) and many others.\nR\u0026rsquo;s Coolest Features # David\u0026rsquo;s response to what makes R special is particularly interesting when taken as part of the larger theme at this year\u0026rsquo;s useR! 2014 conference. In the keynote, one of the biggest messages from John Chambers was R\u0026rsquo;s intention as an interface. An interface works in both directions, from the user down into the computer as we generally conceptualize, but it is also important to remember that the interface is also from the computer up to the user. David\u0026rsquo;s focus upon the features that make it possible for R users to truly excel definitely showcase his emphasis on the users of R and his role as their advocate.\n▶ David Smith - Interview by DataScience.LA at useR 2014 Advice to new R Programmers # While David admits that these days he doesn\u0026rsquo;t spend a lot of time programming in R, he does spend a lot of time writing the Revolution R blog. His advice to programmers is in line with his own progression: R programmers should write, blog, and use those two avenues as a scaffolding to grow their own learning over time.\n▶ David Smith - Interview by DataScience.LA at useR 2014 How to Build a Community # By David\u0026rsquo;s estimate, there are more than 2 million R users today. When you are considering such a large number of people, the underlying need for a community is definitely there. The question then becomes how do you provide the necessary scaffolding for a community to grow in a fashion that will be welcoming to new members? How are you supportive of members that have been there a long time? How do you provide a strong identity to the outside world? These are in addition to defining what being a community member means and sustaining the ecosystem over time.\n▶ David Smith - Interview by DataScience.LA at useR 2014 David Smith expresses his desire for us each to have fantastic job opportunities, have fun during the process of data science, and really feel like what we are doing is valuable to the world around us. This is clearly what motivates him every day, and the R community lucky to have someone like him in a visible position, putting himself out there as an example of approachability, contribution, and leadership.\n","date":"August 25, 2014","externalUrl":null,"permalink":"/david-smith-the-user-2014-interview/","section":"Posts","summary":"“At one point, the only source for daily R news was David Smith’s blog.” – Szilard Pafka\nDavid Smith is an integral part of the R community. His background in computational statistics goes back to the early 90s . David has worked at Revolution R, one of the leading R companies, for nearly a decade. His name has been included in lists such as:\n","title":"David Smith, the useR! 2014 Interview","type":"posts"},{"content":"","date":"August 25, 2014","externalUrl":null,"permalink":"/tag/revolution/","section":"Tags","summary":"","title":"revolution","type":"tags"},{"content":"R is really cool and people that use it can do a lot\u0026hellip;\n","date":"August 25, 2014","externalUrl":null,"permalink":"/david-smith/","section":"Posts","summary":"R is really cool and people that use it can do a lot…\n","title":"David Smith","type":"posts"},{"content":"","date":"August 21, 2014","externalUrl":null,"permalink":"/tag/panel/","section":"Tags","summary":"","title":"panel","type":"tags"},{"content":"(Please click here if the video above doesn’t load.)\nThe video above is from an event at useR! 2014 consisting of Yasmin Lucero’s presentation plus an incredible panel discussion, which is included here. The title of the panel was “R’s Place in the Production Environment”, and was moderated by Tareef Kawaf, President at RStudio. Five high-profile members of the R community were on this panel, spanning the gamut of the enterprise:\nPete Myer - Senior Ads Quality Statistician at Google Alex Toth - DevOps Architect with Gravity-Aol Ajay Gopal - VP of Growth and Data Science at CARD.com Vladimir Ryzhov - Game Analyst at Activision Hilary Parker - Senior Data Analyst at Etsy The panel provides invaluable insight into the day-to-day life of Data Scientists and analysts in the enterprise. Discussions range from the deep technical implications of configuration management across distributed systems to the diversity of backgrounds which are attracted to data as a career, and is an invaluable resource for organizations looking to operationalize their data. We hope your enjoy learning more about R and its uses in industry!\nThis event was co-located with useR!2014, and was organized by the LA R Meetup. If you would like to be receive announcements for the LA R Meetup or the DataScience.LA community, sign up to our mailing list here.\n","date":"August 21, 2014","externalUrl":null,"permalink":"/r-in-production-panel-discussion-la-r-meetup-user-2014/","section":"Posts","summary":"(Please click here if the video above doesn’t load.)\nThe video above is from an event at useR! 2014 consisting of Yasmin Lucero’s presentation plus an incredible panel discussion, which is included here. The title of the panel was “R’s Place in the Production Environment”, and was moderated by Tareef Kawaf, President at RStudio. Five high-profile members of the R community were on this panel, spanning the gamut of the enterprise:\n","title":"R in Production - Panel Discussion - LA R meetup @ useR! 2014","type":"posts"},{"content":"(Please click here if the video above doesn’t load.)\nDuring useR! 2014, the LA R Meetup (run by Szilard Pafka) organized an event focused on R’s status in industry. This event took the form of a presentation by Yasmin Lucero, Senior Statistician at AOL-Gravity, followed by a panel discussion. The video above captures the presentation, where Yasmin provides an excellent overview of the current status of R in the enterprise and breaks down the taxonomy of solutions for which R is commonly chosen as a development platform. From batch reporting and dashboards for internal deployment to full blown public facing analytics data products, R has certainly found a home in business. Yasmin’s talk concludes with specific, operational aspects of managing R in the enterprise, such as library and dependency management. Enjoy the video and don’t forget to tune in for Part 2 of this session where the panel discussed “R’s Place in the Production Environment.”\nIf you would like to receive real-time announcements for the LA R Meetup or the DataScience.LA community, please sign up for our mailing list here.\n▶ R in Production - Opening Talk by Yasmin Lucero - LA R meetup @ useR 2014 (Part 1) ","date":"August 21, 2014","externalUrl":null,"permalink":"/r-in-production-opening-talk-by-yasmin-lucero-la-r-meetup-user-2014/","section":"Posts","summary":"(Please click here if the video above doesn’t load.)\nDuring useR! 2014, the LA R Meetup (run by Szilard Pafka) organized an event focused on R’s status in industry. This event took the form of a presentation by Yasmin Lucero, Senior Statistician at AOL-Gravity, followed by a panel discussion. The video above captures the presentation, where Yasmin provides an excellent overview of the current status of R in the enterprise and breaks down the taxonomy of solutions for which R is commonly chosen as a development platform. From batch reporting and dashboards for internal deployment to full blown public facing analytics data products, R has certainly found a home in business. Yasmin’s talk concludes with specific, operational aspects of managing R in the enterprise, such as library and dependency management. Enjoy the video and don’t forget to tune in for Part 2 of this session where the panel discussed “R’s Place in the Production Environment.”\n","title":"R in Production - Opening Talk by Yasmin Lucero - LA R meetup @ useR! 2014","type":"posts"},{"content":"","date":"August 20, 2014","externalUrl":null,"permalink":"/tag/high-school-students/","section":"Tags","summary":"","title":"high school students","type":"tags"},{"content":"Another exciting development in data science coming from our department at UCLA is a high school class called Introduction to Data Science (IDS). This project has been made possible by a National Science Foundation grant to support Mobilize, for which Rob Gould (mentioned in my previous post on DataFest) is the Principal Investigator.\nThe year-long IDS course is piloting in 10 LAUSD high schools this academic year. For the pilot year, we\u0026rsquo;ve recruited 10 brave classroom teachers who were willing to learn a lot of new material and pedagogy. Most of the teachers have taught statistics before (either \u0026ldquo;regular\u0026rdquo; statistics or Advanced Placement) however they are not experienced with data science.\nIDS is a computation-heavy course, as a major component of the course are completed using hands-on labs using R within RStudio. The team has written an R package to simplify some of the R syntax issues that trip up beginners, and instead of teaching traditional statistical tests and formulas the course is focused on randomization as a basis for making inference, both formal and informal.\nAnother exciting component of the course is that it incorporates participatory sensing, an idea developed at UCLA to describe when people become participants in the data collection process using hand-held, ubiquitous sensors to collect data. Typically, when we speak of sensors in this context we actually mean smartphones, since they are embedded with many native sensors such as GPS and accelerometers. The Mobilize technology team has developed a mobile app that makes it easy to deploy surveys to phones, for example, and students get to collect data on themselves and their habits (for example, what they snack on during the day). When they work on their laboratory activities in class, the students’ own data is involved and they are better able to \u0026ldquo;see themselves\u0026rdquo; in the data and feel connected to the process.\nThe course has been approved by the University of California Office of the President (UCOP), the body that determines if a high school class in the state of California can be considered during college admissions. The UCOP has approved the course as a \u0026ldquo;C\u0026rdquo; course in the \u0026ldquo;A-G\u0026rdquo; requirements, which means that it will count toward the three required years of math. This is very exciting, as it means that the course won\u0026rsquo;t only be seen as an elective, but instead as an integral part of the standard college preparatory curriculum. Great news indeed for the next generation of data scientists!\nWith projects like IDS the future of data science seems bright! Send us a note with other projects that make you feel inspired and optimistic at info@datascience.la, we’d love to hear about them!\n","date":"August 20, 2014","externalUrl":null,"permalink":"/introduction-to-data-science-for-high-school-students/","section":"Posts","summary":"Another exciting development in data science coming from our department at UCLA is a high school class called Introduction to Data Science (IDS). This project has been made possible by a National Science Foundation grant to support Mobilize, for which Rob Gould (mentioned in my previous post on DataFest) is the Principal Investigator.\n","title":"Introduction to Data Science for High School Students","type":"posts"},{"content":"","date":"August 20, 2014","externalUrl":null,"permalink":"/tag/mobilize/","section":"Tags","summary":"","title":"mobilize","type":"tags"},{"content":"Data Science @ IRIS.TV\nby Dr. Thomas Sullivan, Chief Data Scientist and Joel Spitalnik, VP Engineering\nWhile video recommendations best serve the viewer experience when presented in real time, machine learning is a computationally intensive process that is often done on the back end. IRIS.TV’s utilizes a hybrid process to synchronize computational tasks for learning so that engineering and data science teams can shrink the latency between analyzing history and presentation of recommended videos to improve the experience.\nDr. Thomas Sullivan, Chief Data Scientist at IRIS.TV is a data scientist/statistician with extensive IP development experience and is the (co-)author of 8 books. Prior to joining IRIS.tv, he spent 13 years as a Senior Information Scientist at the RAND Corporation, taught Applied Statistics at the RAND\u0026rsquo;s graduate school, and then was Director of R\u0026amp;D at a tech company in Santa Monica. He holds a Ph.D. in Computational Sciences, an M.S. in Statistical Sciences, and an M.A. in Economics.\nJoel Spitalnik, VP Engineering at IRIS.TV has developed iOS apps for hospital laboratory testing as well as dynamic websites for arts and entertainment organizations. He is a veteran of the film and television industry, having worked for Disney, DreamWorks SKG, HDNet, and A-Line Pictures. He is a graduate of Princeton University and is an active member of multiple programming communities.\nDate: September 15, 2014\nTimeline:\n6:30pm food/drinks and networking 7:30pm talk starts promptly You must have a confirmed RSVP and please arrive by 7:25pm the latest. Please RSVP on Eventbrite.\nAlso, to receive updates on subsequent meetups along with other useful community information, please sign up for the DataScience.LA mailing list.\nVenue: Cross Campus in Santa Monica will kindly host this meetup. Factual will kindly sponsor this meetup and also provide food+drinks.\n","date":"August 19, 2014","externalUrl":null,"permalink":"/data-science-meetup-september-15-data-science-iris-tv/","section":"Posts","summary":"Data Science @ IRIS.TV\nby Dr. Thomas Sullivan, Chief Data Scientist and Joel Spitalnik, VP Engineering\nWhile video recommendations best serve the viewer experience when presented in real time, machine learning is a computationally intensive process that is often done on the back end. IRIS.TV’s utilizes a hybrid process to synchronize computational tasks for learning so that engineering and data science teams can shrink the latency between analyzing history and presentation of recommended videos to improve the experience.\n","title":"Data Science meetup - September 15: Data Science @ IRIS.TV","type":"posts"},{"content":"You’ve asked and we’ve listened. The audio content from our DataScience.LA interviews will now be released as a podcast! Check us out on iTunes or point your podcast app to here!\nWe’ve already released the audio from the userR! conference interviews of Hadley Wickham, Max Kuhn, and Tal Galili on iTunes and hope to release upcoming interviews in podcast form shortly after our video blog posts. Thanks for being a part of the DataScience.LA community and we look forward to hearing more from you!\nA huge thank you to DataScience.LA contributor Jeff Weakley who recorded such fantastic audio in our interviews that it made the conversion to podcast possible!\n","date":"August 18, 2014","externalUrl":null,"permalink":"/announcing-the-dsla-podcast/","section":"Posts","summary":"You’ve asked and we’ve listened. The audio content from our DataScience.LA interviews will now be released as a podcast! Check us out on iTunes or point your podcast app to here!\nWe’ve already released the audio from the userR! conference interviews of Hadley Wickham, Max Kuhn, and Tal Galili on iTunes and hope to release upcoming interviews in podcast form shortly after our video blog posts. Thanks for being a part of the DataScience.LA community and we look forward to hearing more from you!\n","title":"Announcing the DSLA Podcast!","type":"posts"},{"content":"","date":"August 18, 2014","externalUrl":null,"permalink":"/tag/podcast-2/","section":"Tags","summary":"","title":"podcast","type":"tags"},{"content":"","date":"August 18, 2014","externalUrl":null,"permalink":"/category/podcast/","section":"Categories","summary":"","title":"Podcast","type":"categories"},{"content":"What I can say is that I am joyful at looking at what people are doing!\n","date":"August 18, 2014","externalUrl":null,"permalink":"/tal-galili/","section":"Posts","summary":"What I can say is that I am joyful at looking at what people are doing!\n","title":"Tal Galili","type":"posts"},{"content":"“One can acquire everything in solitude except character.”\n― Stendhal\nThe Interview # Tal Galili is, in many ways, a central spoke of the R community. Both gregarious and thoughtful, he has grown his website R-bloggers into the definitive aggregation of the R community’s voice through his genuine, passion-driven intensity. Tal had a simple desire as a young programmer - to learn more about his chosen tools - and looked to the internet to find other voices like his. When googling for “R blogs”, Tal found numerous blogs about pirates, but only a handful about R. This interview details how Tal started R-bloggers and decided to challenge the status quo, as well as giving us a peek behind the curtain for his new projects.\nOn Community # There was a time in the not too distant past where in order to get anything accomplished you actually had to know and remember things using only your brain, your books, and nearby scribbled notes. It was a terrible and dark time, where knowledge and access were asymmetrically given to the minds of a few. If you were just starting as a programmer and your code gave you an error it was oftentimes an insurmountable obstacle. You could, of course, ask people in your local community if they’d ever faced a similar challenge, but you were likely to find an answer only if you were lucky enough to have a community of like-minded people nearby and they were actively working with similar tools. Fortunately, the collective web has managed to lift the veil of darkness that once held back an individual’s potential for progress. Rather than being limited to what a single person knows or by one’s geographic proximity to approachable experts, we now find ourselves more limited by the latency of the HTTP request/reponse cycle or our degree of google-fu.\nModern blogging can give everyone a voice. Sure, before the mid 1990’s it was theoretically possible for anyone to put up a web page and \u0026lt;html\u0026gt;\u0026lt;head\u0026gt;\u0026lt;title\u0026gt; themselves into Jerry and David’s Guide to the World Wide Web - however not everyone could truly take advantage. Back then, those venturing into this new web space were often academics with access to departmental or university-wide servers or hobbyists on the very cusp of early adoptership, not to mention that you also had to learn HTML without our modern conveniences like WordPress or Codeacademy. Blogging opened these previously obscured gates and provided simple interfaces to type up a post, disseminate it across the greater web, and make it discoverable for future readers. Blogs place the writer in the center, providing an avenue to share one’s thoughts with the rest world. In addition to providing a vehicle for communication, however, blogs can also serve to feed the writer’s ego.\nThis, among other things, is what makes Tal and R-bloggers special. Community-driven websites depend upon their technology’s integration into the existing human element. Tal understood this and knew that for an online culture to succeed, the people and their ideas should both be front-and-center. What could have morphed over the years to become a self serving, ego-bolstering engine, has instead remained an honest gift to the community. His name is listed only once on the entire page, if you scroll all the way to the bottom and are capable of somehow resisting the seduction to click from the many amazing posts from his over 500+ contributors on the site. The footer plainly, simply states:\nR-bloggers was founded by Tal Galili, with gratitude to the R community.\nDataScience.LA surely can’t speak for the whole R community, but we can speak for ourselves and we’d like to thank you for your work in raising up the profile of individual R bloggers everywhere through your site. We\u0026rsquo;d especially like to thank you for providing a single place that helps us find answers in the late of night as a deadline approaches. The community is truly strengthened by having the fruits of your labor.\n▶ Tal Galili – Full Interview – by DataScience.LA at useR 2014 ","date":"August 18, 2014","externalUrl":null,"permalink":"/a-conversation-with-tal-galili-at-user-2014/","section":"Posts","summary":"“One can acquire everything in solitude except character.”\n― Stendhal\nThe Interview # Tal Galili is, in many ways, a central spoke of the R community. Both gregarious and thoughtful, he has grown his website R-bloggers into the definitive aggregation of the R community’s voice through his genuine, passion-driven intensity. Tal had a simple desire as a young programmer - to learn more about his chosen tools - and looked to the internet to find other voices like his. When googling for “R blogs”, Tal found numerous blogs about pirates, but only a handful about R. This interview details how Tal started R-bloggers and decided to challenge the status quo, as well as giving us a peek behind the curtain for his new projects.\n","title":"A Conversation with Tal Galili at useR! 2014","type":"posts"},{"content":"","date":"August 18, 2014","externalUrl":null,"permalink":"/tag/community/","section":"Tags","summary":"","title":"community","type":"tags"},{"content":"","date":"August 18, 2014","externalUrl":null,"permalink":"/tag/r-bloggers/","section":"Tags","summary":"","title":"r-bloggers","type":"tags"},{"content":"Five LA R meetup members will present short talks of what they learned at useR! 2014\n(plus we have 5 more volunteers for the following meetup!)\n1. Timothy J. Triche, Jr., PhD\nFellow, Jane Anne Nohl Division of Hematology, USC/Norris Comprehensive Cancer Center\nComparing R and Python: Recent and promising developments, some perspective from teaching Python and R courses, and selecting the right tool for the jobs at hand\nShannon Callan\nFounder, Digital Trend Analytics Financial market applications at useR! 2014\nJuan J Natera\nSenior Software Engineer, Campus Explorer Inc. A Short Introduction to dplyr\nWayne Smith, Ph.D.\nDepartment of Management, David Nazarian College of Business and Economics\nCalifornia State University, Northridge The growing number of packages that depending on \u0026lsquo;rJava\u0026rsquo;, the latest R-related books from conference vendors, and using openintro.org and openstaxcollege.org as statistics textbooks for elementary statistics/data science instruction\nArno Vigen\nCFO, AireSpring, Inc. Business-oriented talks at UseR! 2014: Business R at Zillow, Sequencing tools for R, and R \u0026amp; SalesForce\nDate: September 4, 2014\nTimeline:\n6:00pm pizza arrives and networking 7:00pm talks starts promptly You must have a confirmed RSVP and please arrive by 6:55pm the latest. Please RSVP here on Eventbrite.\nAlso, to receive updates on new meetups and other useful community information, please sign up to the DataScience.LA mailing list.\nVenue: Adconion Media Group in Santa Monica will kindly host this meetup and provide pizza and drinks (with a big Thank You to Mikhail and AMG!)\n","date":"August 14, 2014","externalUrl":null,"permalink":"/la-r-meetup-highlights-from-the-user-2014-conference/","section":"Posts","summary":"Five LA R meetup members will present short talks of what they learned at useR! 2014\n(plus we have 5 more volunteers for the following meetup!)\n1. Timothy J. Triche, Jr., PhD\nFellow, Jane Anne Nohl Division of Hematology, USC/Norris Comprehensive Cancer Center\n","title":"LA R meetup - September 4: Highlights from the useR! 2014 Conference","type":"posts"},{"content":"Winston Chang\u0026rsquo;s \u0026ldquo;Interactive Graphics with ggvis\u0026rdquo; talk at useR! 2014 was in many ways a watershed moment for the the data science community, as the future directions of rich data visualization for analytics, communication, and the web were revealed. In this talk, Winston introduces the viewer to ggvis, lays out the fundamental architecture, and provides an overview of reactive architectures.\nThe commonality between science and art is in trying to see profoundly - to develop strategies of seeing and showing.\n-Edward Tufte\nData Science does not happen solely in the large matrices hidden inside the dark confines of servers, it happens through the practitioner’s senses and in their mind. A practicing data scientist collects data from any number of data sources, forms theories about its underlying structure, and proceeds to (attempt to) discover an inkling of the secrets contained therein. Visualization is the tool most often used to accomplish this process in data science, and it’s not hard to understand why. When you consider the senses available to us and how long human beings have spent desperately harnessing scarce resources to sustain ourselves, it makes even more sense.\nThe most critical tool our ancient ancestors used was not the sense of touch or taste, but instead was our vision. Through thousands of years of evolution, humans developed the ability to know which red berry was sweet, and which was toxic. We developed the ability to spot out of the corner of our eye the slight rustling of some leaves, which betrayed a fawn and our next meal. We depend on these senses to let us know when something is right, when something is wrong, and when something is dangerous. Human are born explorers, and we hone the sharpest of our tools towards that goal. We see new things, new connections, and infer what we can’t directly see from what we can see in order to predict the unknown around us.\nIntuition fails in high dimensions\n-Unknown\nThe field of data visualization piggybacks on this evolutionary masterpiece of vision-driven cognition. This understanding allows humans to use intuition by mapping from many-dimensional data down to the dimensions humans work well in - like the two dimensional plane of your monitor. For many years R has contained the premiere visualization system as a “baked in” offering. Hadley Wickham’s ggplot2 library, an implementation of Leland Wilkinson’s “The Grammar of Graphics”, has been cited repeatedly as one of R’s competitive advantages in the data science toolbox. However, ggplot2 was not designed for the modern computing age where the deployment target for all non-trivial development is the web browser.\nThe ggvis package provides a modern re-conceptualization of many of the core concepts of ggplot2 and reframes them in the context of a reactive framework. Whereas ggplot2 created beautiful visualizations which were static and framed in the moment of the creators mind, ggvis allows the creator to assemble a pipeline of ideas, package them, and present them to the consumer. With ggvis, the reader is empowered through interactivity and responsiveness through the data to gain new perspectives. We hope this video encourages you to develop new and innovative visualizations, and empowers your users to interact with data in new ways you\u0026rsquo;ve never before thought possible!\nUPDATE: Slides and code for this presentation are available here.\n","date":"August 14, 2014","externalUrl":null,"permalink":"/winston-chang-interactive-graphics-with-ggvis-user-2014/","section":"Posts","summary":"Winston Chang’s “Interactive Graphics with ggvis” talk at useR! 2014 was in many ways a watershed moment for the the data science community, as the future directions of rich data visualization for analytics, communication, and the web were revealed. In this talk, Winston introduces the viewer to ggvis, lays out the fundamental architecture, and provides an overview of reactive architectures.\n","title":"Winston Chang's \"Interactive Graphics with ggvis\" at useR! 2014 ","type":"posts"},{"content":"Eyeo festival highlights\nAmelia McNamara\nFrom the festival website, \u0026ldquo;Eyeo assembles an incredible set of creative coders, data designers and artists, and attendees \u0026ndash; expect enthralling talks, unique workshops and interactions with open source instigators and super fascinating practitioners.\u0026rdquo; I\u0026rsquo;ll talk about some of the most inspiring talks and projects I saw at Eyeo this year, including work by Santiago Ortiz, Lillian Schwartz, Cod.Act, Kate Crawford and Nicholas Felton. Most of this work can be described best by the term \u0026ldquo;art,\u0026rdquo; so come expecting provocations, not prototypes. To learn more about the festival, visit eyeofestival.com\nAmelia is a statistics PhD candidate at UCLA. Her research interests include statistics education, statistical computing, and data visualization. She attends events like Eyeo in order to gain inspiration for future work.\nDate: August 26, 2014\nTimeline:\n6:00pm networking 7:00pm talk start promptly You must have a confirmed RSVP and arrive between 6:00-6:55pm in order to pass security. Please RSVP here on Eventbrite.\nAlso, to get updated on subsequent meetups and get other useful community information, please sign up to the DataScience.LA mailing list.\nVenue: Cornerstone OnDemand in Santa Monica will kindly host this meetup.\n","date":"August 12, 2014","externalUrl":null,"permalink":"/datavis-la-meetup-eyeo-festival-highlights-august-26/","section":"Posts","summary":"Eyeo festival highlights\nAmelia McNamara\nFrom the festival website, “Eyeo assembles an incredible set of creative coders, data designers and artists, and attendees – expect enthralling talks, unique workshops and interactions with open source instigators and super fascinating practitioners.” I’ll talk about some of the most inspiring talks and projects I saw at Eyeo this year, including work by Santiago Ortiz, Lillian Schwartz, Cod.Act, Kate Crawford and Nicholas Felton. Most of this work can be described best by the term “art,” so come expecting provocations, not prototypes. To learn more about the festival, visit eyeofestival.com\n","title":"DataVis LA Meetup  - August 26: Eyeo Festival Highlights","type":"posts"},{"content":"","date":"August 11, 2014","externalUrl":null,"permalink":"/tag/caret/","section":"Tags","summary":"","title":"caret","type":"tags"},{"content":"We\u0026rsquo;ve made a lot of the same mistakes, and we want to protect people from doing them\n","date":"August 11, 2014","externalUrl":null,"permalink":"/max-kuhn-on-caret/","section":"Posts","summary":"We’ve made a lot of the same mistakes, and we want to protect people from doing them\n","title":"Max Kuhn on caret","type":"posts"},{"content":" The Interview # In the video above, Max provides some amazing insights into the why and how of caret, an R package he created. He also discusses his book on Applied Predictive Modeling which he co-authored with Kjell Johnson, including details on how he set out to write the book he wished he would have had. As a special bonus, Max also describes Quinlan\u0026rsquo;s C5.0, an alternate \u0026ldquo;forest of decision trees\u0026rdquo; algorithm, the secrets of which were hidden behind commercial licensing for years - and which has recently been ported and made available to the R ecosystem. Whether you are a beginner just getting your feet wet with R and predictive modeling, or a seasoned data scientist, this interview has something for everyone.\nExpanding Your Superpowers with Caret # “What is your favorite superpower?” is a classic icebreaker, and the answers will tell you quite a lot about the people answering. Someone in the group will immediately claim that the ability to fly is paramount. Someone else invariably brings up invisibility. Super strength, super speed, laser vision, the ability to talk to sea creatures - these are all fine choices. For me, however, the best answer has always been the ability to predict the future. No other superpower seems to compare - if you can predict the future, then you know how a Flying Superhero will attack, and that you’ll need to bring a mirror to battle Laser-Vision Man. If you can predict the future, perhaps you’ll skip out on that whale watching adventure if you know Sea-Creatures Guy has it out for you. Considering the (current) impossibility of choosing one’s superpower, it’s a fun thought exercise but not much else.\nIn the context of data science, however, there may be a little something to be done about this “predicting the future” thing, and maybe, just maybe, Max Kuhn is the guy to show you how to do it.\nA non-clinical statistician, is kind of exactly what it sounds like. - Max Kuhn\nMax is the Director of Nonclinical Statistics at Pfizer, a position that involves supporting a great many scientists with software tools, analysis, and machine learning during the creation and validation of molecules in the pipeline to become potentially life-saving and life-giving medicines. He is also the creator of the caret package for the R language. caret (short for Classification And REgression Training) is a set of functions which attempt to streamline the process of creating predictive models. Put succintly, the caret package provides the ‘train’ function. This function is your gateway to nearly every awesome machine learning model that can be implemented in R. Want to train a neural network to predict Species from all other variables in the iris data set?\ntrain(Species ~ ., data=iris, method=\u0026#34;nnet”) Turns out that a neural network didn’t provide the accuracy you wanted and instead you decide to try out a more powerful machine learning technique, like random forests?\ntrain(Species ~ ., data=iris, method=\u0026#34;rf\u0026#34;) Perhaps you’re willing to trade a little bit of predictive power in exchange for interpretability, in which case you’d switch over to traditional decision trees.\ntrain(Species ~ ., data=iris, method=\u0026#34;rpart\u0026#34;) That’s all it takes to get started. That’s it.\nInside of the train function, Max has taken his combined decades of experience and expertise creating predictive models and has hidden that complexity for the sake of usability. Each method has its own series of smart defaults and behavior, so that even if you only stick to the basics you can still hit the ground running and be productive. However, the caret package contains much, much more than just the train function.\nInside of the package, Max has encoded best-practice approaches for handling those pitfalls that both new and experienced data scientists might face. Perennial questions such as ‘How do you handle unbalanced classes?’ are answered in caret, providing functions to create balanced data partitions. How do you approach feature selection? Caret is helpful and provides recursive feature elimination. How do you make sure that scaling/centering/PCA pre-processing are properly handled during your cross-validation steps and that they don’t add bias to your results? Caret has your back. How do you test your newly-trained model on a held-back training set and view the accuracy metrics? Caret has a a buffet of options waiting patiently at your fingertips. Suddenly, you can start to predict the future… and you’re certainly a little closer to having a superpower with caret in your toolbox.\n","date":"August 11, 2014","externalUrl":null,"permalink":"/a-conversation-with-max-kuhn-the-user-2014-interview/","section":"Posts","summary":"The Interview # In the video above, Max provides some amazing insights into the why and how of caret, an R package he created. He also discusses his book on Applied Predictive Modeling which he co-authored with Kjell Johnson, including details on how he set out to write the book he wished he would have had. As a special bonus, Max also describes Quinlan’s C5.0, an alternate “forest of decision trees” algorithm, the secrets of which were hidden behind commercial licensing for years - and which has recently been ported and made available to the R ecosystem. Whether you are a beginner just getting your feet wet with R and predictive modeling, or a seasoned data scientist, this interview has something for everyone.\n","title":"A Conversation with Max Kuhn - The useR! 2014 Interview","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/history/","section":"Tags","summary":"","title":"history","type":"tags"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/john-chambers/","section":"Tags","summary":"","title":"John Chambers","type":"tags"},{"content":"At useR! 2014, John Chambers was generous enough to provide us with insight into the very early stages of user-centric interactive data exploration. He explains, step by step, how his insight to provide an interface into algorithms, putting the user first has placed us on the fruitful path which analysts, statisticians, and data scientists enjoy to this day. In his talk, John Chambers also does a fantastic job of highlighting a number of active projects, new and vibrant in the R ecosystem, which are helping to continue this legacy of \u0026ldquo;a software interface into the best algorithms.\u0026rdquo; The future is bright, and new and dynamic ideas are building off these thoughtful, well measured, solid foundations of the past.\nTo understand why this past is so important, I\u0026rsquo;d like to provide a brief view of the historical context that underpins these breakthroughs. In 1976, John Chambers was concerned with making software supported interactive numerical analysis a reality. Let’s talk about what other advances were happening in 1976 in the field of software and computing:\nIn January of 1976, the very first Cray-1 (a commercially available supercomputer), developed by Cray Research, is available. In April of 1976, Steve Jobs and Steve Wozniak decide to form a little company called Apple Computer. In November of 1976, some kid in New Mexico decides to start a company with the audacious goal of making money from selling software, he picks the name MicroSoft. In December of 1976, the very first word processor for personal computers is released, Electric Pencil. Suffice it to say, 1976 was clearly a very busy year, and at the very earliest dawn of the transition from a priesthood of centralized computing to a democratized land of individualized computing. At the crest of that wave, on May 5th, 1976, John Chambers sketched out an idea in a laboratory at the famed Murray Hill office of Bell Labs.\nStated simply, there should be an interface to the very best numerical algorithms available. Those algorithms should be implemented in the way is most natural for them, in whatever language is best for them (usually Fortran), but the user should be freed from the constraints of working at such a low level. The user should be able to think at a higher level, manipulate at a higher level, and be productive at a higher level.\nThus was born the S programming environment, the predecessor to R, and in no small part to almost every user-centric scientific software tool today. For this achievement, John Chambers received the coveted ACM Software System Award in 1998, for the feat of “forever altering the way people analyze, visualize, and manipulate data.”\n","date":"August 6, 2014","externalUrl":null,"permalink":"/john-chambers-user-2014-keynote/","section":"Posts","summary":"At useR! 2014, John Chambers was generous enough to provide us with insight into the very early stages of user-centric interactive data exploration. He explains, step by step, how his insight to provide an interface into algorithms, putting the user first has placed us on the fruitful path which analysts, statisticians, and data scientists enjoy to this day. In his talk, John Chambers also does a fantastic job of highlighting a number of active projects, new and vibrant in the R ecosystem, which are helping to continue this legacy of “a software interface into the best algorithms.” The future is bright, and new and dynamic ideas are building off these thoughtful, well measured, solid foundations of the past.\n","title":"John Chambers useR! 2014 Keynote","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/s/","section":"Tags","summary":"","title":"S","type":"tags"},{"content":"Hadley Wickham is famous. He\u0026rsquo;s not Kardashian famous, but walking around useR! and seeing the community\u0026rsquo;s reaction to him, there\u0026rsquo;s no question, he\u0026rsquo;s \u0026lsquo;R famous\u0026rsquo;. If you have the good fortune to see his talks, tutorials, or sessions in person, you owe it to yourself to do so. He projects depth and wisdom with a booming voice, which combines with a hard-won confidence brought about by years of honing his craft and developing his expertise. He takes as much time as is needed to answer questions, listens to every single bit of feedback and succeeds in making you feel that what you say indeed matters. Hadley Wickham has poise. It\u0026rsquo;s also quite obvious, if you watch him for long enough, that this fame suits him like an itchy sweater made by a loving grandparent. It brings warmth and it comes from a place of love, but it\u0026rsquo;s always a little uncomfortable regardless of how well it may fit.\nI\u0026rsquo;m definitely an introvert. - Hadley Wickham\nHadley and I had a long ranging interview at useR 2014, shown above, discussing R\u0026rsquo;s strengths and revealing its weaknesses together. We reveal Hadley\u0026rsquo;s evil plans for world domination, as well as his not-so-evil plans to help users better manage their workflow. He also discusses his approach for the design of APIs and interfaces, including his thought process and personal experiences which have led him to this philosophy. We get a peek at his regular workflow, and how he continues to create such fantastic packages and modules for release. Hadley gives us some fantastic insight into what advice he would give himself when he was starting out, and what advice he would give someone starting out in the field today. Finally, we talk about what the future holds for R the software, as well as R the community. I hope you enjoy watching it as much as I enjoyed making it!\nHow We Got Here # When the opportunity arose to attend useR 2014, I was ecstatic. My favorite new programming language was having it\u0026rsquo;s yearly conference practically in my backyard. This was serendipity, and I was going to take full advantage of it! The opportunity quickly morphed beyond my wildest dreams, however. Thanks to the amazing R and data science community in Los Angeles, and because some things only seem to happen in LA, DataScience.LA contributor Jeff Weakley brought up the idea: \u0026ldquo;What if we tape a few interviews with selected attendees?\u0026rdquo;\nIt was a great idea, since I already knew that I wanted to see these videos before they were even filmed! As we started discussing the project we realized that in order to tape interviews of attendees, we would need an interviewer. As the resident extrovert in our organization, I volunteered to lead the charge and interview those useR attendees willing to sit down with us. How hard could it be, after all? Interviewing is just asking questions and talking, and I\u0026rsquo;m already pretty solid at both of those. My first interview of the conference was Hadley Wickham, what could go wrong?\n\u0026ldquo;How to Interview Someone\u0026rdquo;\nIs a pretty awful feeling to start researching 24 hours before you have the opportunity to interview anyone - much less interviewing a set of luminaries in a field you love. Much less \u0026ldquo;the guy.\u0026rdquo; But yet, here I was, a single day before I was set to sit down with Hadley Wickham, trying to figure out how to conduct an interesting interview. In case you\u0026rsquo;re curious, here\u0026rsquo;s a helpful 2 page PDF I found. The basic advice was pretty straight forward:\nBe prepared Make the subject comfortable I knew I wasn\u0026rsquo;t going to be prepared enough, and I had no idea how to make any of my subjects comfortable. UseR 2014 was going to be a long scary week.\nI am relatively new to the R community, but I have been a programmer in a great many communities for a long time. Fortunate programming communities are able to find their Hadley, someone who comes in to an established ecosystem bringing new ideas with the boundless energy and intelligence to elegantly implement them, be it DHH for Ruby or Miyagawa for Perl. It has also always struck me as odd that often times, the way we thank these kinds of luminaries is by hoisting them into the Land of the Extrovert. Their fame, made through asynchronous email and forum communication and with patch after patch of lovingly handcrafted code, leaves them rewarded by placement on a public pedestal for all to see and regularly made the center of unyielding attention.\nAnd yet, here we were, sitting next to each other on a cloudy day at UCLA\u0026rsquo;s campus, Hadley with a patient smile, and me, hoping I would remember my list of questions and that I wouldn\u0026rsquo;t throw up on him.\nMy Introduction to the Hadleyverse # I was originally introduced to R through my wife\u0026rsquo;s doctoral dissertation she completed in 2009. As a \u0026ldquo;programmer\u0026rdquo; I didn\u0026rsquo;t get it. Why on earth would you index a data structure with a $? Why would you \u0026ldquo;attach\u0026rdquo; a data frame? Clearly this programming environment was not for \u0026ldquo;serious work\u0026rdquo;, was my immediate decision, and I promptly filed it away. Like most humans, I am historically awful at predicting the future. This was no exception.\nAs my own personal career began to shift from being a \u0026ldquo;regular programmer\u0026rdquo; to what some would call data science, I felt my trusty and reliable tools had begun to fall short and had started looking for solutions. The limitations of the code -\u0026gt; execute -\u0026gt; read results loop constrained my ability to explore ever larger data sets, and the ability to visualize complex relationships was laughable. As part of a General Assembly Data Science course, I was introduced to ggplot2 and my relationships with R, data, and exploration, were forever changed. I had been introduced to the Hadleyverse.\nAs I explored ggplot2 for visualization, to plyr for transformation, it became clear. From tidy data to lubridate, it seemed like this gentleman, Hadley Wickham, had addressed all the major problems in \u0026ldquo;programming R\u0026rdquo;, and had made it a kinder, less shocking ecosystem to explore. It made it easy for my concept of R to transform from an also-ran to one of my favorite programming ecosystems of all time.\nStay tuned for more interviews like this one, as we will be releasing one per week over the next few months!\n","date":"August 6, 2014","externalUrl":null,"permalink":"/a-conversation-with-hadley-wickham-the-user-2014-interview/","section":"Posts","summary":"Hadley Wickham is famous. He’s not Kardashian famous, but walking around useR! and seeing the community’s reaction to him, there’s no question, he’s ‘R famous’. If you have the good fortune to see his talks, tutorials, or sessions in person, you owe it to yourself to do so. He projects depth and wisdom with a booming voice, which combines with a hard-won confidence brought about by years of honing his craft and developing his expertise. He takes as much time as is needed to answer questions, listens to every single bit of feedback and succeeds in making you feel that what you say indeed matters. Hadley Wickham has poise. It’s also quite obvious, if you watch him for long enough, that this fame suits him like an itchy sweater made by a loving grandparent. It brings warmth and it comes from a place of love, but it’s always a little uncomfortable regardless of how well it may fit.\n","title":"A Conversation with Hadley Wickham - the useR! 2014 interview","type":"posts"},{"content":"I think it\u0026rsquo;s interesting that R is an extremely fixable language.\nHadley Wickham\n","date":"August 6, 2014","externalUrl":null,"permalink":"/hadley-on-r/","section":"Posts","summary":"I think it’s interesting that R is an extremely fixable language.\nHadley Wickham\n","title":"Hadley on R","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/humor/","section":"Tags","summary":"","title":"humor","type":"tags"},{"content":"I’ve had a lifelong fascination with maps, and working with R definitely enables my map habit. Perhaps there’s something about being an immigrant that made me particularly introspective, since from a very early age I was aware that I was “from” one place on the map and now lived at this \u0026ldquo;other place.”\nSome of my earliest family memories are of my father calling out a random city such as Reykjavik and my sister and I scrambling in response to be the first to find it on one of the many world atlases distributed throughout our house. They’re very fond memories, since playing Find the Wacky-Sounding City is a fantastic way to spend time with your kids.\nOne of my other favorite things to do is express my love of humor and my love of maps through ggplot. I love building facetious maps to amuse my Twitter followers, and recently discovered a lovely little data set containing states where it is illegal to have fireworks delivered.\nSo I wrote a little R function for the Fourth of July that put together a joke map and delivers a zinger:\njoke_map \u0026lt;- function(states_to_highlight, title=\u0026#34;\u0026#34;, highlight_colors=c(\u0026#34;#aaaaaa\u0026#34;, \u0026#34;#ee0000\u0026#34;)) { library(dplyr) library(ggplot2) library(RColorBrewer) library(maps) highlighting = data.frame(region=tolower(state.name[match(states_to_highlight, state.abb)]), highlight=TRUE) d = map_data(\u0026#34;state\u0026#34;) %\u0026gt;% left_join(highlighting, by=\u0026#34;region\u0026#34;) d[is.na(d$highlight),]$highlight \u0026lt;- FALSE p \u0026lt;- ggplot(d) + geom_polygon(aes(x=long, y=lat, group = group, fill=highlight), colour=\u0026#34;black\u0026#34;) + scale_fill_manual(values=highlight_colors) + ggtitle(title) + xlab(\u0026#34;\u0026#34;) + ylab(\u0026#34;\u0026#34;) + theme(axis.ticks = element_blank(), axis.text.x = element_blank(), axis.text.y = element_blank()) + guides(fill=FALSE) p } Source: gist ab49e04 by earino\nNot exactly groundbreaking R code, I know, but quite handy. I was able to simply call the function with dataset I had found at fireworks.us and have a little chuckle.\nEnjoy the code, and have fun generating imaginary internet points! As I get older, I\u0026rsquo;ve also been working on my Dad joke superpowers, so I\u0026rsquo;ll leave you with my latest masterpiece:\nmap \u0026lt;- joke_map(states_to_highlight = c(\u0026#34;WY\u0026#34;), title = \u0026#34;Squarest, Bluest State\u0026#34;, highlight_colors=c(\u0026#34;#aaaaaa\u0026#34;, \u0026#34;#0000ee\u0026#34;)) Source: gist 65623ce by earino\n","date":"August 6, 2014","externalUrl":null,"permalink":"/making-maps-with-a-punchline/","section":"Posts","summary":"I’ve had a lifelong fascination with maps, and working with R definitely enables my map habit. Perhaps there’s something about being an immigrant that made me particularly introspective, since from a very early age I was aware that I was “from” one place on the map and now lived at this “other place.”\n","title":"Making Maps with a Punchline","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/become-a-contributor/","section":"Tags","summary":"","title":"become a contributor","type":"tags"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/blogging-accelerator/","section":"Tags","summary":"","title":"blogging accelerator","type":"tags"},{"content":"By now I’m sure you’ve checked out our blog and have wondered, ‘How can I contribute my ideas and share my projects with the community?’ There are two ways to contribute content to DataScience.LA, either as a regular contributor or through our Blogging Accelerator. Our vision is to help unleash you, members of the community, as effective data science communicators while establishing Los Angeles as one of the leading places for data thinking, data journalism, and data science. (Ready to write a blog post right now? Send us an email at info@datascience.la to get started as a contributor.)\nWe know, however, that not everyone has had the same writing experiences and may find the process of contributing to a community blog to be full of anxiety (AND excitement!) We’re here to help with the anxiety part, since we’d like to try something new to help generate stellar content regardless of your writing experience. If you are a data science student, practitioner, or expert and you have always wanted to write a blog post but were afraid to “go it alone”, we’re going to be here to provide targeted support while developing your voice to creating high-quality data science writing. The Blogging Accelerator program will start as a small number of individuals interested in working over a 2 month period to develop their ideas into full-fledged blog posts for our site. We will meet weekly to discuss our ideas and progress, and you\u0026rsquo;ll have the chance to see behind the scenes here at DataScience.LA during the process. Writing support during the program will be customized to what each participant needs to become successful, and can include how to develop an idea on paper and get to a first draft, providing editing expertise, help with sticking to a timeline, or technical support with posting to a blog itself. Regardless, it all starts with a conversation - drop us a note at info@datascience.la and mention you’d like to get started with our Blogging Accelerator!\n","date":"August 6, 2014","externalUrl":null,"permalink":"/blogging-accelerator-for-datascience-la/","section":"Posts","summary":"By now I’m sure you’ve checked out our blog and have wondered, ‘How can I contribute my ideas and share my projects with the community?’ There are two ways to contribute content to DataScience.LA, either as a regular contributor or through our Blogging Accelerator. Our vision is to help unleash you, members of the community, as effective data science communicators while establishing Los Angeles as one of the leading places for data thinking, data journalism, and data science. (Ready to write a blog post right now? Send us an email at info@datascience.la to get started as a contributor.)\n","title":"Blogging Accelerator for DataScience.LA","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/writing/","section":"Tags","summary":"","title":"writing","type":"tags"},{"content":"As a PhD graduate student in statistics here at UCLA, I’d like to talk a bit about our graduate program to give you an idea of what to expect if you’re interested in UCLA or statistics graduate school in general. First, it’s important to note that UCLA does not have a specific \u0026ldquo;data science\u0026rdquo; degree program at this point; the department offers a PhD and a Master\u0026rsquo;s degree, both simply labeled \u0026ldquo;statistics.\u0026rdquo; Some statistics departments have made the decision to change their name to \u0026ldquo;data science,\u0026rdquo; but that is often just a marketing decision. The UCLA department is pretty forward-thinking and has a focus on computation, so it\u0026rsquo;s fairly \u0026ldquo;data science\u0026rdquo;-y without having that exact name.\nLet\u0026rsquo;s start with a general outline of graduate school: usually, the first component is coursework, and the second is writing your thesis or dissertation. For those pursuing a Master\u0026rsquo;s degree, the project is typically called a thesis, and for PhD students we call it a dissertation. These written projects differ in complexity and length, with dissertations being longer, more technical, and generally more focused on theory than applications. As a result, students chasing a PhD usually take quite a bit longer to complete their degrees than those getting a Masters.\nI just completed my fourth year of graduate school and am planning to defend my dissertation next June, making it an even 5 years from the start of my graduate career. This is pretty standard length for a PhD from our department, although there\u0026rsquo;s bound to be some variation (+/- 2 years).\nCoursework # The first year of graduate school is typically focused entirely upon coursework, and UCLA is no exception. For students pursuing the PhD track, there are three required sequences of classes (i.e. 200, 201, 202) with three classes contained in each (e.g. 200A, 200B, 200C). The 200 sequence contains topics in theoretical statistics and probability and is focused mostly on proofs. The 201 sequence is a little more applied, focusing on sampling, experimental design, regression, and the like. The 202 sequence contains statistical computing, from basic R and Python to Monte Carlo methods. (Side note: UCLA uses a course numbering scheme I hadn\u0026rsquo;t encountered before, as courses in the 10s are lower-division undergraduate classes, like STAT 10, STAT 13, etc., for non-statistics majors, the 100s are upper-division undergraduate courses for statistics majors, and the 200+ courses are graduate level.)\nIn addition to the technical course sequences discussed above, we also have a weekly departmental seminar course, which all graduate students are required to attend (although it\u0026rsquo;s open to the public as well). Every week in the seminar is a different invited speaker, presenting their work. The speakers tend to reflect the interests of the professor who is organizing the seminar \u0026ndash; when Mark Handcock is in charge, there are many talks on networks, for example. My first year in the program, Mark Hansen organized the seminar and we had a variety of not-traditionally-statistics talks, including both the technology head and a design technologist from Stamen design, a law professor, and a statistician who specializes in the ethics of data. The goal of the departmental seminar is to expose students to research interests that may be outside the expertise of the department, and allow them to see the variety of possible sub-fields available to them. It\u0026rsquo;s been likened to a required Meetup group for us grad students.\nTo complete the coursework requirements, PhD students must have 54 units, or about 13 classes. After completing the 200-series, most students have approximately 4 additional electives they must complete in order to have their required units. In my case, I chose Spatial Statistics, Data and the Media Arts, Programming Media, Computer Programming Languages and Systems, and two quarters of \u0026ldquo;Data, Data Practices, and Data Curation.\u0026rdquo; Most of my electives were related to data, computing, and data visualization/art. For those of you keeping track at home, that\u0026rsquo;s more than four courses, but I enjoyed coursework and had quite a variety of interests.\nMasters program # Thus far, I’ve only discussed the PhD program. For Master\u0026rsquo;s students, the requirements for coursework are slightly reduced (although they take the 200-level courses, too), and students typically spend only two years at UCLA (one year to complete coursework and one year to write their thesis). Once Master\u0026rsquo;s students have completed their coursework, they can move on to writing their thesis. I\u0026rsquo;m not as familiar with the Master\u0026rsquo;s program, but if you want to know more, you can check out the department\u0026rsquo;s information for applicants.\nProving your worth # After finishing coursework, the next step in a PhD program is typically to prove that you know the material from the required courses. This is often one of the things that separates Master\u0026rsquo;s and PhD programs\u0026ndash; Master\u0026rsquo;s students are required to take classes and write a thesis; PhD students have to take classes, prove they know the material, and then write a dissertation.\nIn many graduate programs in mathematics, proving you know the material requires students to complete a grueling series all-day exams covering all the content from the required courses. These are called qualifying exams (or \u0026ldquo;quals\u0026rdquo;). However, a few years before I joined the Statistics department the faculty voted to discontinue quals, even for PhD students. Instead, grading for courses was set to make the classes even more rigorous, meaning that once you\u0026rsquo;ve passed all your coursework with the required GPA you are considered qualified. This lack of quals makes the UCLA statistics program little outside the norm for math-related PhDs. However, when I was looking at graduate programs I was honestly drawn to the department by the lack of quals. It felt much less stressful for me to pass classes individually than to remember everything on one stressful, make-or-break day.\nWhat\u0026rsquo;s next? # In the next episode of this post, I\u0026rsquo;ll talk about the process for \u0026ldquo;advancing to candidacy,\u0026rdquo; writing and defending the dissertation, and show you some of the more fun aspects of grad school. For a taste, I\u0026rsquo;ll just tell you that we tend to have fun at conferences together:\nMe and my fellow UCLA statistics grad student Terri, at the 2013 Joint Statistical Meetings. ","date":"August 6, 2014","externalUrl":null,"permalink":"/statistics-grad-school-what-is-it-really-like/","section":"Posts","summary":"As a PhD graduate student in statistics here at UCLA, I’d like to talk a bit about our graduate program to give you an idea of what to expect if you’re interested in UCLA or statistics graduate school in general. First, it’s important to note that UCLA does not have a specific “data science” degree program at this point; the department offers a PhD and a Master’s degree, both simply labeled “statistics.” Some statistics departments have made the decision to change their name to “data science,” but that is often just a marketing decision. The UCLA department is pretty forward-thinking and has a focus on computation, so it’s fairly “data science”-y without having that exact name.\n","title":"Statistics grad school-- what is it really like?","type":"posts"},{"content":"Below is the first of several exciting data science developments for the younger generation, happening right here in Los Angeles. This project is unique because it engages undergraduates in real data analysis, something that happens quite rarely in a classroom. If projects like this catch on as a new trend, we\u0026rsquo;re going to have some real competition for our jobs in a few years!\nDataFest # 2014 UCLA DataFest contestants DataFest is an event for undergraduates that began at UCLA and is now spreading across the country. Each year, we find an interesting \u0026ldquo;data sponsor\u0026rdquo; (i.e. an active company willing to share their data) and give students 48 hours to come up with interesting insights. We provide the food, coffee, energy drinks and wifi, they do the data cleaning, exploratory data analysis, and presentations. This is similar to the ‘hackathons’ many in our group are familiar with, with a data twist.\nIn addition to the data and fuel, we provide the contestants with plenty of help. There are always roving graduate students to consult with, and we recruit \u0026ldquo;VIP consultants\u0026rdquo; (working data science practitioners) to come by throughout the event. Most of the students use R for their analysis, although it\u0026rsquo;s not a formal requirement. Statistics majors at UCLA will have seen R in one of their required courses, but many students haven\u0026rsquo;t worked with messy, real datasets before DataFest so they often need technical help. Our VIP consultants had expressed worry that they wouldn\u0026rsquo;t be able to answer students\u0026rsquo; questions, however asking \u0026ldquo;why are you trying to do that?\u0026rdquo; as a followup would get to the root of the problem and enabled these mentors to provide valuable insights toward a simpler path.\nPast years\u0026rsquo; data sponsors of DataFest have included the LAPD, eHarmony, Kiva.com, and GridPoint. Typically, we give three prizes\u0026ndash; \u0026ldquo;Best Insight,\u0026rdquo; \u0026ldquo;Best Visualization,\u0026rdquo; and \u0026ldquo;Best Use of External Data.\u0026rdquo; This year\u0026rsquo;s UCLA DataFest winners for Best Visualization were Lev Golod and Jonhngyun Lee. Golod and Lee created a graphic that showed when GridPoint users were mis-using air conditioning (e.g. using AC when the outside temperature was less than 60 degrees). See that winning entry here or see all this year\u0026rsquo;s winners on the UCLA DataFest website.\nThis year, a number of higher education institutions held their own DataFests, including Duke and the Five Colleges of Western Massachusetts (Amherst, Hampshire, Mount Holyoke, Smith, and UMass Amherst).\nFor more on DataFest, check out this post from fivethirtyeight, this writeup from the American Statistical Association (Note that the ASA is taking over sponsorship of the event as of next year), or my post about DataFest and Hadley Wickham\u0026rsquo;s R packages. If you’d like to be involved next year (donating data, swag, prizes, money, or your time as a VIP consultant), contact me or Rob Gould.\nWith projects like DataFest the future of data science seems bright! Send us a note with other projects that make you feel inspired and optimistic at info@datascience.la, we’d love to hear about them!\n","date":"August 6, 2014","externalUrl":null,"permalink":"/data-science-goes-to-college-with-datafest/","section":"Posts","summary":"Below is the first of several exciting data science developments for the younger generation, happening right here in Los Angeles. This project is unique because it engages undergraduates in real data analysis, something that happens quite rarely in a classroom. If projects like this catch on as a new trend, we’re going to have some real competition for our jobs in a few years!\n","title":"Data science goes to college with DataFest","type":"posts"},{"content":"","date":"August 6, 2014","externalUrl":null,"permalink":"/tag/datafest/","section":"Tags","summary":"","title":"DataFest","type":"tags"},{"content":"Data visualization helps us understand data. The DataVis LA meetup discusses the many aspects of data visualization: exploratory data analysis, presentation graphics, infographics, design, art, etc., and brings together a diverse community of data scientists, statisticians, software developers, BI specialists, business analysts, journalists, designers and artists with the goal of learning more about the art and science of data visualization from each other.\n","date":"August 4, 2014","externalUrl":null,"permalink":"/meetup-groups-datavis-la/","section":"Posts","summary":"Data visualization helps us understand data. The DataVis LA meetup discusses the many aspects of data visualization: exploratory data analysis, presentation graphics, infographics, design, art, etc., and brings together a diverse community of data scientists, statisticians, software developers, BI specialists, business analysts, journalists, designers and artists with the goal of learning more about the art and science of data visualization from each other.\n","title":"Meetup Groups: DataVis LA","type":"posts"},{"content":"The LA Machine Learning Meetup began with events discussing a large variety of machine learning topics such as classification, clustering, neural networks, graphical algorithms, information retrieval, search, game theory, computational learning theory, reinforcement learning, collaborative filtering etc. More recently, a Data Science Track was launched with the focus on business applications and the entire process of data mining (e.g. business understanding, data collection, exploratory data analysis, data transformations, feature engineering, modeling, model validation, deployment, communication of results). While in fact Machine Learning is a part of Data Science and not the other way around, rather than starting a new meetup group, Data Science is featured within the already existing Machine Learning meetup.\n","date":"August 4, 2014","externalUrl":null,"permalink":"/meetup-groups-la-machine-learning-data-science/","section":"Posts","summary":"The LA Machine Learning Meetup began with events discussing a large variety of machine learning topics such as classification, clustering, neural networks, graphical algorithms, information retrieval, search, game theory, computational learning theory, reinforcement learning, collaborative filtering etc. More recently, a Data Science Track was launched with the focus on business applications and the entire process of data mining (e.g. business understanding, data collection, exploratory data analysis, data transformations, feature engineering, modeling, model validation, deployment, communication of results). While in fact Machine Learning is a part of Data Science and not the other way around, rather than starting a new meetup group, Data Science is featured within the already existing Machine Learning meetup.\n","title":"Meetup Groups: LA Machine Learning","type":"posts"},{"content":"Many of us here in the DataScience.LA Team were also involved in organizing the useR! 2014 conference, the annual get-together of the wonderful R community, which brought over 700 R users and developers from all over the world to Los Angeles this past July.\nThanks to the volunteer efforts of some of our Team members (see below), we will be able to publish some of these useR session recordings over the next couple of weeks. DataScience.LA also had the opportunity to conduct a series of interviews with prominent members of the R community during useR, and we’ll be publishing those videos as well.\nThe first two videos are live, John Chambers’ keynote talk and the interview we conducted with Hadley Wickham.\nWe are planning a weekly release for the rest of the videos we have (interviews on Mondays, sessions on Thursdays). We recommend you follow us on Twitter @wwwDSLA in order to get updates on each new release.\nVolunteers for the video project:\nJeff Weakley, videography\nTimothy Phan, videography\nEduardo Arino de la Rubia, interviewer\nAll materials were filmed as part of useR! 2014 at UCLA, supervised by David L McArthur PhD MPH within the guidelines of UCLA Policy 863 Section D.\n","date":"August 4, 2014","externalUrl":null,"permalink":"/user-2014-conference-videos-and-release-plan/","section":"Posts","summary":"Many of us here in the DataScience.LA Team were also involved in organizing the useR! 2014 conference, the annual get-together of the wonderful R community, which brought over 700 R users and developers from all over the world to Los Angeles this past July.\n","title":"useR! 2014 Conference Videos and Release Plan","type":"posts"},{"content":"R is an open source software and programming language for data analysis, statistical modeling, visualization and data mining among other things. Recently, as the term “data science” has emerged to describe collectively these tasks, R has also become the primary tool of choice for data scientists. The LA R meetup began before this data science boom, back in 2009, by bringing together local users of R and featuring talks with world-class speakers while creating fruitful networking opportunities for its members. In retrospect, this group served as the very first data science meetup in Los Angeles and has significantly contributed to the development of our local data science community.\n","date":"August 4, 2014","externalUrl":null,"permalink":"/meetups-r/","section":"Posts","summary":"R is an open source software and programming language for data analysis, statistical modeling, visualization and data mining among other things. Recently, as the term “data science” has emerged to describe collectively these tasks, R has also become the primary tool of choice for data scientists. The LA R meetup began before this data science boom, back in 2009, by bringing together local users of R and featuring talks with world-class speakers while creating fruitful networking opportunities for its members. In retrospect, this group served as the very first data science meetup in Los Angeles and has significantly contributed to the development of our local data science community.\n","title":"Meetup Groups: LA R Meetup","type":"posts"},{"content":"","date":"August 2, 2014","externalUrl":null,"permalink":"/category/site/","section":"Categories","summary":"","title":"Site","type":"categories"},{"content":"Welcome! We are launching DataScience.LA to provide a home for all the content (slides, code, videos etc.) for several meetups in LA (Data Science/Machine Learning, R, DataVis etc.) and to engage the Los Angeles data science community in an open, active discussion about best practices and new developments in the field.\nDataScience.LA was founded by a team of data science professionals, including a few who have been involved in building the LA data science community for years, and it is helped by a growing number of contributors. Our goal is to build DataScience.LA into the premiere information resource and virtual meeting place for the Los Angeles data science community.\nIn order to maximize the benefits of being part of this community, we encourage you to join our mailing list, which will allow you to receive priority announcements/RSVPs to our meetups, participate in raffles (books, T-shirts etc.), get access to community news, and much, much more.\nTo keep up-to-date, please follow us on Twitter @wwwDSLA (to get notifications on new posts), and use the #dsla hashtag (especially at our meetups since the best tweets will be featured on the blog).\n","date":"August 2, 2014","externalUrl":null,"permalink":"/welcome-to-data-science-la/","section":"Posts","summary":"Welcome! We are launching DataScience.LA to provide a home for all the content (slides, code, videos etc.) for several meetups in LA (Data Science/Machine Learning, R, DataVis etc.) and to engage the Los Angeles data science community in an open, active discussion about best practices and new developments in the field.\n","title":"Welcome to Data Science LA!","type":"posts"},{"content":"We are a team of volunteers dedicated to building a world-class data science community in Los Angeles for the benefit of all.\nLeadership: # Szilard Pafka # Szilard is an expert data scientist and a leader of the Los Angeles data community. He brings extensive theoretical knowledge (PhD) along with more than 15 years of practical experience in advanced data analysis and developing analytical systems. He is currently the Chief Scientist at Epoch and the founder/organizer/co-organizer of several popular data science meetups in LA (R, DataVis, Data Science/Machine Learning, Big Data). Follow him on Twitter @DataScienceLA. Update: Szilard moved from California to Texas in July 2021 and is no longer involved with any activities in LA or California.\nAmelia McNamara # Amelia is a statistics PhD candidate at the University of California, Los Angeles (UCLA). Her research interests include statistical computing, data visualization, spatial statistics, and statistics education. She is currently working with teachers in the Los Angeles Unified School District to bring \u0026ldquo;data science\u0026rdquo; into high school classrooms as part of the Mobilize project. You can follow her on Twitter @AmeliaMN, or read her personal blog.\nEduardo Arino de la Rubia # Eduardo is a lifelong technologist with a passion for data science who thrives on effectively communicating data-driven insights throughout an organization. He is a graduate of the MTSU Computer Science department, General Assembly’s Data Science program, and is a graduate of the Johns Hopkins Coursera Data Science specialization. Follow him on Twitter @earino.\nLeigh Arino de la Rubia # Leigh is an interdisciplinary writer, project manager, and coach who combines an eye for detail with a sense of humor. A former science education researcher, roller derby coach and private investigator, Leigh enjoys working collaboratively to help people accomplish more than they think is possible. Follow her on Twitter @leighadlr or check out ppm.io to find out what\u0026rsquo;s been inspiring her lately.\nContributors: # Jeff Weakley # Jeff is an Emmy-winning Copywriter/ Creative Director with over 25 years experience working with Ad Agencies in Los Angeles. He’s a graduate of UCLA Cinematography extension and the owner of an ever-growing studio of lights, cameras, and sound equipment. Producer/ Editor/ VFX, Jeff also works in R, Python and SQL and aspires to bring more data science into the advertising creative arena through his endeavors such as ChocolateCoveredFocusGroups.com and CreativeAlgorithm.org.\nTimothy Phan # Tim is a data enthusiast with a passion for public policy. He previously studied Political Science and Development Economics at UC Santa Barbara and the Monterey Institute, and recently worked at the Center for Global Development in Washington DC. His interests include data visualization, new media, and R. Follow his Twitter at @timothy_phan, or see what he’s been into lately at his personal site.\nSubash D’Souza # Subash is a leader of the Big Data community in Los Angeles. He speaks regularly at conferences, has been recognized by Cloudera as a Champion of Big Data, and envisions himself as a Big Data Evangelist. He is the founder of Big Data Camp LA, a free community conference, and he runs the LA Big Data, LA HBase, and LA Apache Spark meetups. He is currently the Principal Hadoop Architect at Lynda.com. Follow him on Twitter @sawjd22.\nCarl Mullins # Carl Mullins is a long time Research Engineer with his first job at the Hughes Space and Communications Group and the Hughes Space Systems Division and TRW Space Division working on the Defense Support Program (DSP). He has now returned back to school and apply himself to Astronautical Engineering, Computer Science, Electrical Engineering, Mathematics and Physics. He is presently at the University of Southern California pursuing a Masters degree in Data Informatics and he is also volunteering his time to the LA Data Science community.\nGergely Daróczi # Gergely is a passionate R developer with around 10 years of experience using R in finance, social and medical sciences; PhD candidate in Sociology; technical founder of an R-based reporting web application at rapporter.net and the Hungarian R meetup; main organizer of the first satRday conference; currently the Director of Analytics at CARD.com mixing R with AWS services for a scalable data infrastructure. Follow him on Twitter and GitHub.\n","date":"July 15, 2014","externalUrl":null,"permalink":"/team/","section":"Data Science LA","summary":"We are a team of volunteers dedicated to building a world-class data science community in Los Angeles for the benefit of all.\nLeadership: # Szilard Pafka # Szilard is an expert data scientist and a leader of the Los Angeles data community. He brings extensive theoretical knowledge (PhD) along with more than 15 years of practical experience in advanced data analysis and developing analytical systems. He is currently the Chief Scientist at Epoch and the founder/organizer/co-organizer of several popular data science meetups in LA (R, DataVis, Data Science/Machine Learning, Big Data). Follow him on Twitter @DataScienceLA. Update: Szilard moved from California to Texas in July 2021 and is no longer involved with any activities in LA or California.\n","title":"Team","type":"page"},{"content":"This page used to host the sign-up form for the DataScience.LA mailing list, which announced meetups and events in the Los Angeles area. DataScience.LA is no longer active and the mailing list is closed; this site is a read-only archive of the original blog.\n","externalUrl":null,"permalink":"/join-membership/","section":"Data Science LA","summary":"This page used to host the sign-up form for the DataScience.LA mailing list, which announced meetups and events in the Los Angeles area. DataScience.LA is no longer active and the mailing list is closed; this site is a read-only archive of the original blog.\n","title":"Join / Membership","type":"page"}]