Summary: Every organization should provide their analysts and data scientists with a few key tools: A Data Dictionary, a Metric Dictionary, a Research Repository, and a Code Repository. All of these tools need to be searchable to make it easy for analysts to find and use previous work.
Summary: Advanced analyses can be simplified by calling out which variables are most important. Decision Trees, Random Forests, Regression, and Chi-Square tests can quickly reveal what variables carry a lot of weight.
Summary: Spark has an implementation of Alternating Least Squares (ALS) along with a set of very simple functions to create recommendations based on past data.
Summary: Spark (and Pyspark) use map, mapValues, reduce, reduceByKey, aggregateByKey, and join to transform, aggregate, and connect datasets. Each function can be stringed together to do more complex tasks.
Summary: Always check your numbers with smaller, simpler queries and figures. Use total sales as a reality check for comparison to sales queries. When creating models, compare performance to a simpler model. Don’t assume complexity equals accuracy. Be prepared to compare against existing “gold standard” models.
