Summary: Before your new analyst arrives, make sure they have access to the data, hardware, and software they’ll need. Set and discuss expectations on what the analyst should know and learn. Get others involved in the development with mentors and formalized “get to know” meetings.
Summary: Every organization should provide their analysts and data scientists with a few key tools: A Data Dictionary, a Metric Dictionary, a Research Repository, and a Code Repository. All of these tools need to be searchable to make it easy for analysts to find and use previous work.
Summary: Advanced analyses can be simplified by calling out which variables are most important. Decision Trees, Random Forests, Regression, and Chi-Square tests can quickly reveal what variables carry a lot of weight.
Summary: Spark has an implementation of Alternating Least Squares (ALS) along with a set of very simple functions to create recommendations based on past data.
Summary: Spark (and Pyspark) use map, mapValues, reduce, reduceByKey, aggregateByKey, and join to transform, aggregate, and connect datasets. Each function can be stringed together to do more complex tasks.
