Keep learning with DataSource

The best data science and machine learning articles. Written by data scientists for data scientists (and business people)

• 

  N/A
  Model Evaluation Metrics in Machine Learning

  Data Science Machine Learning

  CreditsPredictive models have become a trusted advisor to many businesses and for a good reason. These models can “foresee the future”, and there are many different methods available, meaning any industry can find one that fits their particular challenges.When we talk about predictive models, we are talking either about a regression model (continuous output) or a classification model (nominal or binary output). In classification problems, we use two types of algorithms (dependent on the kind of output it creates):Class output: Algorithms like SVM and KNN create a class output. For instance, in a binary classification problem, the outputs will be either 0 or 1. However, today we have algorithms that can convert these class outputs to probability.Probability output: Algorithms like Logistic Regression, Random Forest, Gradient Boosting, Adaboost, etc. give probability outputs. Converting probability outputs to class output is just a matter of creating a threshold probability.IntroductionWhile data preparation and training a machine learning model is a key step in the machine learning pipeline, it’s equally important to measure the performance of this trained model. How well the model generalizes on the unseen data is what defines adaptive vs non-adaptive machine learning models.By using different metrics for performance evaluation, we should be in a position to improve the overall predictive power of our model before we roll it out for production on unseen data.Without doing a proper evaluation of the ML model using different metrics, and depending only on accuracy, it can lead to a problem when the respective model is deployed on unseen data and can result in poor predictions.This happens because, in cases like these, our models don’t learn but instead memorize;hence, they cannot generalize well on unseen data.Model Evaluation MetricsLet us now define the evaluation metrics for evaluating the performance of a machine learning model, which is an integral component of any data science project. It aims to estimate the generalization accuracy of a model on the future (unseen/out-of-sample) data.Confusion MatrixA confusion matrix is a matrix representation of the prediction results of any binary testing that is often used to describe the performance of the classification model (or “classifier”) on a set of test data for which the true values are known.The confusion matrix itself is relatively simple to understand, but the related terminology can be confusing.Confusion matrix with 2 class labels.Each prediction can be one of the four outcomes, based on how it matches up to the actual value:True Positive (TP): Predicted True and True in reality.True Negative (TN): Predicted False and False in reality.False Positive (FP): Predicted True and False in reality.False Negative (FN): Predicted False and True in reality.Now let us understand this concept using hypothesis testing.A Hypothesis is speculation or theory based on insufficient evidence that lends itself to further testing and experimentation. With further testing, a hypothesis can usually be proven true or false.A Null Hypothesis is a hypothesis that says there is no statistical significance between the two variables in the hypothesis. It is the hypothesis that the researcher is trying to disprove.We would always reject the null hypothesis when it is false, and we would accept the null hypothesis when it is indeed true.Even though hypothesis tests are meant to be reliable, there are two types of errors that can occur.These errors are known as Type 1 and Type II errors.For example, when examining the effectiveness of a drug, the null hypothesis would be that the drug does not affect a disease.Type I Error:- equivalent to False Positives(FP).The first kind of error that is possible involves the rejection of a null hypothesis that is true.Let’s go back to the example of a drug being used to treat a disease. If we reject the null hypothesis in this situation, then we claim that the drug does have some effect on a disease. But if the null hypothesis is true, then, in reality, the drug does not combat the disease at all. The drug is falsely claimed to have a positive effect on a disease.Type II Error:- equivalent to False Negatives(FN).The other kind of error that occurs when we accept a false null hypothesis. This sort of error is called a type II error and is also referred to as an error of the second kind.If we think back again to the scenario in which we are testing a drug, what would a type II error look like? A type II error would occur if we accepted that the drug hs no effect on disease, but in reality, it did.A sample python implementation of the Confusion matrix.import warnings import pandas as pd from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.metrics import confusion_matrix import matplotlib.pyplot as plt %matplotlib inline   #ignore warnings warnings.filterwarnings('ignore') # Load digits dataset url = "http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data" df = pd.read_csv(url) # df = df.values X = df.iloc[:,0:4] y = df.iloc[:,4] #test size test_size = 0.33 #generate the same set of random numbers seed = 7 #Split data into train and test set. X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=test_size, random_state=seed) #Train Model model = LogisticRegression() model.fit(X_train, y_train) pred = model.predict(X_test)  #Construct the Confusion Matrix labels = ['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'] cm = confusion_matrix(y_test, pred, labels) print(cm) fig = plt.figure() ax = fig.add_subplot(111) cax = ax.matshow(cm) plt.title('Confusion matrix') fig.colorbar(cax) ax.set_xticklabels([''] + labels) ax.set_yticklabels([''] + labels) plt.xlabel('Predicted Values') plt.ylabel('Actual Values') plt.show()Confusion matrix with 3 class labels.The diagonal elements represent the number of points for which the predicted label is equal to the true label, while anything off the diagonal was mislabeled by the classifier. Therefore, the higher the diagonal values of the confusion matrix the better, indicating many correct predictions.In our case, the classifier predicted all the 13 setosa and 18 virginica plants in the test data perfectly. However, it incorrectly classified 4 of the versicolor plants as virginica.There is also a list of rates that are often computed from a confusion matrix for a binary classifier:1. AccuracyOverall, how often is the classifier correct?Accuracy = (TP+TN)/totalWhen our classes are roughly equal in size, we can use accuracy, which will give us correctly classified values.Accuracy is a common evaluation metric for classification problems. It’s the number of correct predictions made as a ratio of all predictions made.Misclassification Rate(Error Rate): Overall, how often is it wrong. Since accuracy is the percent we correctly classified (success rate), it follows that our error rate (the percentage we got wrong) can be calculated as follows:Misclassification Rate = (FP+FN)/totalWe use the sklearn module to compute the accuracy of a classification task, as shown below.#import modules import warnings import pandas as pd import numpy as np from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn import datasets from sklearn.metrics import accuracy_score #ignore warnings warnings.filterwarnings('ignore') # Load digits dataset iris = datasets.load_iris() # # Create feature matrix X = iris.data # Create target vector y = iris.target #test size test_size = 0.33 #generate the same set of random numbers seed = 7 #cross-validation settings kfold = model_selection.KFold(n_splits=10, random_state=seed) #Model instance model = LogisticRegression() #Evaluate model performance scoring = 'accuracy' results = model_selection.cross_val_score(model, X, y, cv=kfold, scoring=scoring) print('Accuracy -val set: %.2f%% (%.2f)' % (results.mean()*100, results.std()))  #split data X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=test_size, random_state=seed) #fit model model.fit(X_train, y_train) #accuracy on test set result = model.score(X_test, y_test) print("Accuracy - test set: %.2f%%" % (result*100.0))The classification accuracy is 88% on the validation set.2. PrecisionWhen it predicts yes, how often is it correct?Precision=TP/predicted yesWhen we have a class imbalance, accuracy can become an unreliable metric for measuring our performance. For instance, if we had a 99/1 split between two classes, A and B, where the rare event, B, is our positive class, we could build a model that was 99% accurate by just saying everything belonged to class A. Clearly, we shouldn’t bother building a model if it doesn’t do anything to identify class B; thus, we need different metrics that will discourage this behavior. For this, we use precision and recall instead of accuracy.3. Recall or SensitivityWhen it’s actually yes, how often does it predict yes?True Positive Rate = TP/actual yesRecall gives us the true positive rate (TPR), which is the ratio of true positives to everything positive.In the case of the 99/1 split between classes A and B, the model that classifies everything as A would have a recall of 0% for the positive class, B (precision would be undefined — 0/0). Precision and recall provide a better way of evaluating model performance in the face of a class imbalance. They will correctly tell us that the model has little value for our use case.Just like accuracy, both precision and recall are easy to compute and understand but require thresholds. Besides, precision and recall only consider half of the confusion matrix:4. F1 ScoreThe F1 score is the harmonic mean of the precision and recall, where an F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.Why harmonic mean? Since the harmonic mean of a list of numbers skews strongly toward the least elements of the list, it tends (compared to the arithmetic mean) to mitigate the impact of large outliers and aggravate the impact of small ones.An F1 score punishes extreme values more. Ideally, an F1 Score could be an effective evaluation metric in the following classification scenarios:When FP and FN are equally costly — meaning they miss on true positives or find false positives — both impact the model almost the same way, as in our cancer detection classification exampleAdding more data doesn’t effectively change the outcome effectivelyTN is high (like with flood predictions, cancer predictions, etc.)A sample python implementation of the F1 score.import warnings import pandas from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.metrics import log_loss from sklearn.metrics import precision_recall_fscore_support as score, precision_score, recall_score, f1_score  warnings.filterwarnings('ignore')  url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" dataframe = pandas.read_csv(url) dat = dataframe.values X = dat[:,:-1] y = dat[:,-1] test_size = 0.33 seed = 7  model = LogisticRegression() #split data X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=test_size, random_state=seed) model.fit(X_train, y_train) precision = precision_score(y_test, pred) print('Precision: %f' % precision) # recall: tp / (tp + fn) recall = recall_score(y_test, pred) print('Recall: %f' % recall) # f1: tp / (tp + fp + fn) f1 = f1_score(y_test, pred) print('F1 score: %f' % f1)5. SpecificityWhen it’s no, how often does it predict no?True Negative Rate=TN/actual noIt is the true negative rate or the proportion of true negatives to everything that should have been classified as negative.Note that, together, specificity and sensitivity consider the full confusion matrix:6. Receiver Operating Characteristics (ROC) CurveMeasuring the area under the ROC curve is also a very useful method for evaluating a model. By plotting the true positive rate (sensitivity) versus the false-positive rate (1 — specificity), we get the Receiver Operating Characteristic (ROC) curve. This curve allows us to visualize the trade-off between the true positive rate and the false positive rate.The following are examples of good ROC curves. The dashed line would be random guessing (no predictive value) and is used as a baseline; anything below that is considered worse than guessing. We want to be toward the top-left corner:A sample python implementation of the ROC curves.#Classification Area under curve import warnings import pandas from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.metrics import roc_auc_score, roc_curve  warnings.filterwarnings('ignore')  url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" dataframe = pandas.read_csv(url) dat = dataframe.values X = dat[:,:-1] y = dat[:,-1] seed = 7 #split data X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=test_size, random_state=seed) model.fit(X_train, y_train)  # predict probabilities probs = model.predict_proba(X_test) # keep probabilities for the positive outcome only probs = probs[:, 1]  auc = roc_auc_score(y_test, probs) print('AUC - Test Set: %.2f%%' % (auc*100))  # calculate roc curve fpr, tpr, thresholds = roc_curve(y_test, probs) # plot no skill plt.plot([0, 1], [0, 1], linestyle='--') # plot the roc curve for the model plt.plot(fpr, tpr, marker='.') plt.xlabel('False positive rate') plt.ylabel('Sensitivity/ Recall') # show the plot plt.show()In the example above, the AUC is relatively close to 1 and greater than 0.5. A perfect classifier will have the ROC curve go along the Y-axis and then along the X-axisLog LossLog Loss is the most important classification metric based on probabilities.As the predicted probability of the true class gets closer to zero, the loss increases exponentially:It measures the performance of a classification model where the prediction input is a probability value between 0 and 1. Log loss increases as the predicted probability diverge from the actual label. The goal of any machine learning model is to minimize this value. As such, smaller log loss is better, with a perfect model having a log loss of 0.A sample python implementation of the Log Loss.#Classification LogLoss import warnings import pandas from sklearn import model_selection from sklearn.linear_model import LogisticRegression from sklearn.metrics import log_loss  warnings.filterwarnings('ignore') url = "https://raw.githubusercontent.com/jbrownlee/Datasets/master/pima-indians-diabetes.data.csv" dataframe = pandas.read_csv(url) dat = dataframe.values X = dat[:,:-1] y = dat[:,-1] seed = 7 #split data X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, test_size=test_size, random_state=seed) model.fit(X_train, y_train) #predict and compute logloss pred = model.predict(X_test) accuracy = log_loss(y_test, pred) print("Logloss: %.2f" % (accuracy))Logloss: 8.02 Jaccard IndexJaccard Index is one of the simplest ways to calculate and find out the accuracy of a classification ML model. Let’s understand it with an example. Suppose we have a labeled test set, with labels as –y = [0,0,0,0,0,1,1,1,1,1]And our model has predicted the labels as –y1 = [1,1,0,0,0,1,1,1,1,1]The above Venn diagram shows us the labels of the test set and the labels of the predictions, and their intersection and union.Jaccard Index or Jaccard similarity coefficient is a statistic used in understanding the similarities between sample sets. The measurement emphasizes the similarity between finite sample sets and is formally defined as the size of the intersection divided by the size of the union of the two labeled sets, with formula as –Jaccard Index or Intersection over Union(IoU)So, for our example, we can see that the intersection of the two sets is equal to 8 (since eight values are predicted correctly) and the union is 10 + 10–8 = 12. So, the Jaccard index gives us the accuracy as –So, the accuracy of our model, according to Jaccard Index, becomes 0.66, or 66%.Higher the Jaccard index higher the accuracy of the classifier.A sample python implementation of the Jaccard index.import numpy as np def compute_jaccard_similarity_score(x, y):     intersection_cardinality = len(set(x).intersection(set(y)))     union_cardinality = len(set(x).union(set(y)))     return intersection_cardinality / float(union_cardinality)  score = compute_jaccard_similarity_score(np.array([0, 1, 2, 5, 6]), np.array([0, 2, 3, 5, 7, 9])) print "Jaccard Similarity Score : %s" %score passJaccard Similarity Score : 0.375Kolomogorov Smirnov chartK-S or Kolmogorov-Smirnov chart measures the performance of classification models. More accurately, K-S is a measure of the degree of separation between positive and negative distributions.The cumulative frequency for the observed and hypothesized distributions is plotted against the ordered frequencies. The vertical double arrow indicates the maximal vertical difference.The K-S is 100 if the scores partition the population into two separate groups in which one group contains all the positives and the other all the negatives. On the other hand, If the model cannot differentiate between positives and negatives, then it is as if the model selects cases randomly from the population. The K-S would be 0.In most classification models the K-S will fall between 0 and 100, and that the higher the value the better the model is at separating the positive from negative cases.The K-S may also be used to test whether two underlying one-dimensional probability distributions differ. It is a very efficient way to determine if two samples are significantly different from each other.A sample python implementation of the Kolmogorov-Smirnov.from scipy.stats import kstest import random   # N = int(input("Enter number of random numbers: ")) N = 10   actual =[] print("Enter outcomes: ")   for i in range(N):     # x = float(input("Outcomes of class "+str(i + 1)+": "))     actual.append(random.random())   print(actual) x = kstest(actual, "norm")    print(x)The Null hypothesis used here assumes that the numbers follow the normal distribution. It returns statistics and p-value. If the p-value is < alpha, we reject the Null hypothesis.Alpha is defined as the probability of rejecting the null hypothesis given the null hypothesis(H0) is true. For most of the practical applications, alpha is chosen as 0.05.Gain and Lift ChartGain or Lift is a measure of the effectiveness of a classification model calculated as the ratio between the results obtained with and without the model. Gain and lift charts are visual aids for evaluating the performance of classification models. However, in contrast to the confusion matrix that evaluates models on the whole population gain or lift chart evaluates model performance in a portion of the population.The higher the lift (i.e. the further up it is from the baseline), the better the model.The following gains chart, run on a validation set, shows that with 50% of the data, the model contains 90% of targets, Adding more data adds a negligible increase in the percentage of targets included in the model.Gain/lift chartLift charts are often shown as a cumulative lift chart, which is also known as a gains chart. Therefore, gains charts are sometimes (perhaps confusingly) called “lift charts”, but they are more accurately cumulative lift charts.It is one of their most common uses is in marketing, to decide if a prospective client is worth calling.Gini CoefficientThe Gini coefficient or Gini Index is a popular metric for imbalanced class values. The coefficient ranges from 0 to 1 where 0 represents perfect equality and 1 represents perfect inequality. Here, if the value of an index is higher, then the data will be more dispersed.Gini coefficient can be computed from the area under the ROC curve using the following formula:Gini Coefficient = (2 * ROC_curve) — 1ConclusionUnderstanding how well a machine learning model is going to perform on unseen data is the ultimate purpose behind working with these evaluation metrics. Metrics like accuracy, precision, recall are good ways to evaluate classification models for balanced datasets, but if the data is imbalanced and there’s a class disparity, then other methods like ROC/AUC, Gini coefficient perform better in evaluating the model performance.Well, this concludes this article. I hope you guys have enjoyed reading it, feel free to share your comments/thoughts/feedback in the comment section.Thanks for reading !!!

  16 minutes read
• 

  N/A
  5 Tips To Ace Your Job Interview For A Data Scientist Opening

  Data Science

  5 Tips To Ace Your Job Interview For A Data Scientist Opening.PNG 795.94 KBImage SourceAspiring data scientists have a bright future ahead of them. They’re about to enter a field that’s exponentially expanding in terms of job growth and career opportunities. Reports say that the sector has seen a 650% job growth since 2012 and, according to predictions, there will be an estimated 11.5 million new jobs by 2026. All that’s left for data scientist hopefuls is to develop their skills and ace their job interviews. While that may be the most daunting part, we're here to give you five tips on how to impress your interviewer and grab that opportunity. #1- Prepare answers for potential interview questions In our list of 10 highly probable data scientist interview questions, we highlight some of the most asked questions that you’ll want to prepare for. These include situations related to machine learning, Python, and SQL. For example, you could get asked about the difference between classification and clustering, or the important features of dictionaries. While these questions can depend on your interviewer and the company you’re trying for, it won’t hurt to have prepared answers for these basic questions. Brush up on these topics and do your own research. #2- Recall your technical abilities Companies often have a separate technical screening portion prepared for you, but it would also be helpful to run over your technical abilities. This also depends on the specifications of the position you’re applying for; as a data scientist, they might inquire about your efficiency in developing algorithms for the collation and cleaning of datasets. Your interviewer could ask if you’ve had the chance to create an original algorithm of your own. If you’ve done any data projects, they might also inquire about the challenges you faced and how you were able to deal with them. #3- Communicate your strengths It would be helpful if you could confidently explain and articulate what kind of data scientist you are. To do this, you must know where your strengths lie and what your niche is. In any job interview, companies often ask about an applicant’s strengths because the way they approach this question says a lot about them. Think about what you could contribute to a team and what type of role you see yourself thriving in. Then, figure out a way to communicate why you think your unique strengths are an asset to the company. #4- When prompted, ask questions of your own Interviewers love when an applicant shows engagement and interest in the company. Throughout the process, jot down the questions that might come to you, and don’t be afraid to ask away when prompted. The questions you ask in an interview could be a chance for you to learn more about your potential employer and the work environment. You could ask them simple questions like, “what is the most enjoyable part of working here?”, or “what are the company’s goals over the next few years?” This shows them your passion and dedication for the role. #5- Stay updated on trends in the data science space The data science industry is always changing, and there’s always something new to learn every day. If you want to gain an edge against your competitors, make sure you’re on top of the latest data science trends and news. One way to do so is to always be on the lookout for upcoming data science conferences and seminars you can attend. Attending events could earn you connections and help you learn things you won’t find in textbooks. You could also do some supplementary reading of the latest research papers. This will show your interviewers that you’re a motivated self-starter.With time, effort, and these five tips in mind, you’ll be ready to answer any question thrown at you. Interviews are just the first step towards the career of your dreams, so make sure you prepare for every opportunity presented to you.

  3 minutes read
• 

  N/A
  When to Avoid Deep Learning

  Deep learning

  IntroductionThis article is intended for data scientists who may consider using deep learning algorithms, and want to know more about the cons of implementing these type of models into your work. Deep learning algorithms have many benefits, are powerful, and can be fun to show off. However, there are a few times when you should avoid them. I will be discussing those times when you should stop using deep learning below, so keep on reading if you would like a deeper dive into deep learning.When You Want to Easily ExplainPhoto by Malte Helmhold on Unsplash [2].Because other algorithms have been around longer, they have countless amounts of documentation, including examples and functions that make interpretability easier. It is also how the other algorithms work themselves. Deep learning can be intimidating to data scientists for this reason as well, it can be a turn-off to use a deep learning algorithm when you are unsure of how to explain it to a stakeholder.Here are 3 examples of when you would have trouble explaining deep learning:When you want to describe the top features of your model — the features become hidden inputs, so you will not know what caused a certain prediction to happen, and if you need to prove to stakeholders or customers why a certain output was achieved, it can be more of a black boxWhen you want to tune your hyperparameters like learning rate and batch sizeWhen you want to explain how the algorithm works itself — for example, if you were to present the algorithm itself to stakeholders, they might get lost, because even a simplified approach is still difficult to understandHere are 3 examples of how you could explain those same situations from above from non-deep learning algorithms:When you want to explain your top features, you can easily access SHAP libraries, say for the algorithm CatBoost, once your model is fitted, you can simply make a summary plot from feat = model.get_feature_importance() and then use the summary_plot() to rank the features by feature name, so that you can present a nice plot to stakeholders (and yourself for that matter)Example of ranked SHAP output from a non-deep learning model [3].As a solution, some other algorithms have made it plenty easy to tune your hyperparameters by randomized grid search or a more structured, set grid search method. There are even some algorithms that tune themselves so you do not have to worry about complicated tuningExplaining how other algorithms work can be a lot easier, like decision trees, for example, you can easily show a yes or no, 0/1 chart that shows a simple answer for features that lead to a prediction, like yes it is raining, yes it is winter, would provide for yes it is going to be coldOverall, deep learning algorithms are useful and powerful, so there is definitely a time and place for them, but there are other algorithms that you can use instead, as we will discuss below.When You Can Use Other AlgorithmsPhoto by Luca Bravo on Unsplash [4].To be frank, there are a few go-to algorithms that can give you a great model with great results rather quickly. Some of these algorithms include Linear Regression, Decision Trees, Random Forest, XGBoost, and CatBoost. These are alternatives that are more simple.Here are examples of why you would want to use a non-deep learning algorithm, becuase you have so many other, simpler, non-deep learning options:They can be easier and faster to set up, for example, deep learning can require you to have your model add sequential, dense layers, and compile it, which can be more complex, and take longer than simply having a regressor or classifier and fitting it with non-deep learning algorithmsI personally find more errors that can result from this more complex deep learning code and documentation for how to fix it can be confusing or old and not be applicable, using an algorithm like Random Forest instead, can have much more documentation on errors that are easy to understandTraining on a deep learning algorithm may not be complicated sometimes, but when predicting from an endpoint, it might be confusing on how to feed values to predict on, whereas some models, you can simply have the values in an encoded list of ordered valuesI would say that you can of course try out deep learning algorithms, but before you do that, it might be best to start with a simpler solution. It can depend on things like how often you will train and make predictions, or if it is a one-off task. There are some other reasons why you would not want to use a deep learning algorithm, like when you have a small dataset and small budget, as we will discuss below.When You Have a Small Dataset and BudgetPhoto by Hello I’m Nik on Unsplash [5].Oftentimes, you can be working as a data scientist at a smaller company, or perhaps at a startup. In these cases, you would not have much data and you might not have a big budget. You would, therefore, try to avoid the use of deep learning algorithms. Sometimes you can even have a small dataset that is just a few thousand rows and few features, you could simply run an alternative model instead locally, rather than spending a lot of money by serving it frequently.Here is when you should second guess using a deep learning algorthim based on costs and data availability:Small data availability is usually the case for a lot of companies (but is not always the case), and deep learning performs better on information with a lot of dataYou might be performing a one-off task, as in the model only predicts one time — and you can run it locally for free (not all models will be running in production frequently), like a simple Decision Tree Classifier. It might not be worth investing time in a deep learning model.Your company is interested in data science applications but wants to keep the budget small, rather than perform costly executions from a deep learning model, and rather, use a tree-based model with early-stopping-rounds to prevent overfitting, shorten training time, and ultimately reduce costsThere have been times where I brought up deep learning and it was shot down for a variety of reasons, and these reasons were usually the case. But, I do not want to dissuade someone from using deep learning completely, as it is something you should use sometimes in your career, and can be something you do frequently or mainly depending on the circumstances and where you are working.SummaryOverall, before you dive deep into deep learning, realize that there are some times when you should avoid using it for a variety of reasons. There are, of course, more reasons for avoiding it, but there are also reasons for using it too. It is ultimately up to you to look at the pros and cons of deep learning yourself.Here are three times/reasons when you should not use deep learning:* When You Want to Easily Explain * When You Can Use Other Algorithms * When You Have Small Dataset and BudgetI hope you found my article both interesting and useful. Please feel free to comment down below if you agree or disagree with reasons for avoiding deep learning. Why or why not? What other reasons do you think you should avoid using deep learning as a data scientist? These can certainly be clarified even further, but I hope I was able to shed some light on deep learning. Thank you for reading!I am not affiliated with any of these companies.Please feel free to check out my profile, Matt Przybyla, and other articles, as well as subscribe to receive email notifications for my blogs by following the link below, or by clicking on the subscribe icon on the top of the screen by the follow icon, and reach out to me on LinkedIn if you have any questions or comments.Subscribe link: https://datascience2.medium.com/subscribeReferences[1] Photo by Nadine Shaabana on Unsplash, (2018)[2] Photo by Malte Helmhold on Unsplash, (2021)[3] M.Przybyla, Example of ranked SHAP output from a non-deep learning model, (2021)[4] Photo by Luca Bravo on Unsplash, (2016)[5] Photo by Hello I’m Nik on Unsplash, (2021)

  7 minutes read
• 

  N/A
  6 Advanced Statistical Concepts in Data Science

  Data Science Programming

  The article contains some of the most commonly used advanced statistical concepts along with their Python implementation.In my previous articles Beginners Guide to Statistics in Data Science and The Inferential Statistics Data Scientists Should Know we have talked about almost all the basics(Descriptive and Inferential) of statistics which are commonly used in understanding and working with any data science case study. In this article, lets go a little beyond and talk about some advance concepts which are not part of the buzz.Concept #1 - Q-Q(quantile-quantile) PlotsBefore understanding QQ plots first understand what is a Quantile?A quantile defines a particular part of a data set, i.e. a quantile determines how many values in a distribution are above or below a certain limit. Special quantiles are the quartile (quarter), the quintile (fifth), and percentiles (hundredth).An example:If we divide a distribution into four equal portions, we will speak of four quartiles. The first quartile includes all values that are smaller than a quarter of all values. In a graphical representation, it corresponds to 25% of the total area of distribution. The two lower quartiles comprise 50% of all distribution values. The interquartile range between the first and third quartile equals the range in which 50% of all values lie that are distributed around the mean. In Statistics, A Q-Q(quantile-quantile) plot is a scatterplot created by plotting two sets of quantiles against one another. If both sets of quantiles came from the same distribution, we should see the points forming a line that’s roughly straight(y=x).Q-Q plotFor example, the median is a quantile where 50% of the data fall below that point and 50% lie above it. The purpose of Q Q plots is to find out if two sets of data come from the same distribution. A 45-degree angle is plotted on the Q Q plot; if the two data sets come from a common distribution, the points will fall on that reference line.It’s very important for you to know whether the distribution is normal or not so as to apply various statistical measures on the data and interpret it in much more human-understandable visualization and their Q-Q plot comes into the picture. The most fundamental question answered by the Q-Q plot is if the curve is Normally Distributed or not.Normally distributed, but why?The Q-Q plots are used to find the type of distribution for a random variable whether it is a Gaussian Distribution, Uniform Distribution, Exponential Distribution, or even Pareto Distribution, etc. You can tell the type of distribution using the power of the Q-Q plot just by looking at the plot. In general, we are talking about Normal distributions only because we have a very beautiful concept of the 68–95–99.7 rule which perfectly fits into the normal distribution So we know how much of the data lies in the range of the first standard deviation, second standard deviation and third standard deviation from the mean. So knowing if a distribution is Normal opens up new doors for us to experiment with  Types of Q-Q plots. Source Skewed Q-Q plotsQ-Q plots can find skewness(measure of asymmetry) of the distribution. If the bottom end of the Q-Q plot deviates from the straight line but the upper end is not, then the distribution is Left skewed(Negatively skewed).Now if upper end of the Q-Q plot deviates from the staright line and the lower is not, then the distribution is Right skewed(Positively skewed).Tailed Q-Q plotsQ-Q plots can find Kurtosis(measure of tailedness) of the distribution.The distribution with the fat tail will have both the ends of the Q-Q plot to deviate from the straight line and its centre follows the line, where as a thin tailed distribution will term Q-Q plot with very less or negligible deviation at the ends thus making it a perfect fit for normal distribution.Q-Q plots in Python(Source)Suppose we have the following dataset of 100 values:import numpy as np #create dataset with 100 values that follow a normal distribution np.random.seed(0) data = np.random.normal(0,1, 1000) #view first 10 values data[:10] array([ 1.76405235, 0.40015721, 0.97873798, 2.2408932 , 1.86755799, -0.97727788, 0.95008842, -0.15135721, -0.10321885, 0.4105985 ])To create a Q-Q plot for this dataset, we can use the qqplot() function from the statsmodels library:import statsmodels.api as sm import matplotlib.pyplot as plt #create Q-Q plot with 45-degree line added to plot fig = sm.qqplot(data, line='45') plt.show()In a Q-Q plot, the x-axis displays the theoretical quantiles. This means it doesn’t show your actual data, but instead, it represents where your data would be if it were normally distributed.The y-axis displays your actual data. This means that if the data values fall along a roughly straight line at a 45-degree angle, then the data is normally distributed.We can see in our Q-Q plot above that the data values tend to closely follow the 45-degree, which means the data is likely normally distributed. This shouldn’t be surprising since we generated the 100 data values by using the numpy.random.normal() function.Consider instead if we generated a dataset of 100 uniformly distributed values and created a Q-Q plot for that dataset:#create dataset of 100 uniformally distributed values data = np.random.uniform(0,1, 1000) #generate Q-Q plot for the dataset fig = sm.qqplot(data, line='45') plt.show()The data values clearly do not follow the red 45-degree line, which is an indication that they do not follow a normal distribution.Concept #2- Chebyshev's InequalityIn probability, Chebyshev’s Inequality, also known as “Bienayme-Chebyshev” Inequality guarantees that, for a wide class of probability distributions, only a definite fraction of values will be found within a specific distance from the mean of a distribution.Source: https://www.thoughtco.com/chebyshevs-inequality-3126547 Chebyshev’s inequality is similar to The Empirical rule(68-95-99.7); however, the latter rule only applies to normal distributions. Chebyshev’s inequality is broader; it can be applied to any distribution so long as the distribution includes a defined variance and mean.So Chebyshev’s inequality says that at least (1-1/k^2) of data from a sample must fall within K standard deviations from the mean (or equivalently, no more than 1/k^2 of the distribution’s values can be more than k standard deviations away from the mean).Where K --> Positive real numberIf the data is not normally distributed then different amounts of data could be in one standard deviation. Chebyshev’s inequality provides a way to know what fraction of data falls within K standard deviations from the mean for any data distribution.Also read: 22 Statistics Questions to Prepare for Data Science InterviewsCredits: https://calcworkshop.com/joint-probability-distribution/chebyshev-inequality/ Chebyshev’s inequality is of great value because it can be applied to any probability distribution in which the mean and variance are provided.Let us consider an example, Assume 1,000 contestants show up for a job interview, but there are only 70 positions available. In order to select the finest 70 contestants amongst the total contestants, the proprietor gives tests to judge their potential. The mean score on the test is 60, with a standard deviation of 6. If an applicant scores an 84, can they presume that they are getting the job?The results show that about 63 people scored above a 60, so with 70 positions available, a contestant who scores an 84 can be assured they got the job.Chebyshev's Inequality in Python(Source) Create a population of 1,000,000 values, I use a gamma distribution(also works with other distributions) with shape = 2 and scale = 2.import numpy as np import random import matplotlib.pyplot as plt #create a population with a gamma distribution shape, scale = 2., 2. #mean=4, std=2*sqrt(2) mu = shape*scale #mean and standard deviation sigma = scale*np.sqrt(shape) s = np.random.gamma(shape, scale, 1000000)Now sample 10,000 values from the population.#sample 10000 values rs = random.choices(s, k=10000)Count the sample that has a distance from the expected value larger than k standard deviation and use the count to calculate the probabilities. I want to depict a trend of probabilities when k is increasing, so I use a range of k from 0.1 to 3.#set k ks = [0.1,0.5,1.0,1.5,2.0,2.5,3.0] #probability list probs = [] #for each k for k in ks: #start count c = 0 for i in rs: # count if far from mean in k standard deviation if abs(i - mu) > k * sigma : c += 1 probs.append(c/10000)Plot the results:plot = plt.figure(figsize=(20,10)) #plot each probability plt.xlabel('K') plt.ylabel('probability') plt.plot(ks,probs, marker='o') plot.show() #print each probability print("Probability of a sample far from mean more than k standard deviation:") for i, prob in enumerate(probs): print("k:" + str(ks[i]) + ", probability: " \ + str(prob)[0:5] + \ " | in theory, probability should less than: " \ + str(1/ks[i]**2)[0:5])From the above plot and result, we can see that as the k increases, the probability is decreasing, and the probability of each k follows the inequality. Moreover, only the case that k is larger than 1 is useful. If k is less than 1, the right side of the inequality is larger than 1 which is not useful because the probability cannot be larger than 1.Concept #3- Log-Normal DistributionIn probability theory, a Log-normal distribution also known as Galton's distribution is a continuous probability distribution of a random variable whose logarithm is normally distributed.Thus, if the random variable X is log-normally distributed, then Y = ln(X) has a normal distribution. Equivalently, if Y has a normal distribution, then the exponential function of Y i.e, X = exp(Y), has a log-normal distribution. Skewed distributions with low mean and high variance and all positive values fit under this type of distribution. A random variable that is log-normally distributed takes only positive real values. The general formula for the probability density function of the lognormal distribution is:The location and scale parameters are equivalent to the mean and standard deviation of the logarithm of the random variable.The shape of Lognormal distribution is defined by 3 parameters:σis the shape parameter, (and is the standard deviation of the log of the distribution)θ or μ is the location parameter (and is the mean of the distribution)m is the scale parameter (and is also the median of the distribution)The location and scale parameters are equivalent to the mean and standard deviation of the logarithm of the random variable as explained above.If x = θ, then f(x) = 0. The case where θ = 0 and m = 1 is called the standard lognormal distribution. The case where θ equals zero is called the 2-parameter lognormal distribution.The following graph illustrates the effect of the location(μ) and scale(σ) parameter on the probability density function of the lognormal distribution: Source: https://www.sciencedirect.com/topics/mathematics/lognormal-distribution Log-Normal Distribution in Python(Source)Let us consider an example to generate random numbers from a log-normal distribution with μ=1 and σ=0.5 using scipy.stats.lognorm function.import numpy as np import matplotlib.pyplot as plt from scipy.stats import lognorm np.random.seed(42) data = lognorm.rvs(s=0.5, loc=1, scale=1000, size=1000) plt.figure(figsize=(10,6)) ax = plt.subplot(111) plt.title('Generate wrandom numbers from a Log-normal distribution') ax.hist(data, bins=np.logspace(0,5,200), density=True) ax.set_xscale("log") shape,loc,scale = lognorm.fit(data) x = np.logspace(0, 5, 200) pdf = lognorm.pdf(x, shape, loc, scale) ax.plot(x, pdf, 'y') plt.show()Concept #4- Power Law distributionIn statistics, a Power Law is a functional relationship between two quantities, where a relative change in one quantity results in a proportional relative change in the other quantity, independent of the initial size of those quantities: one quantity varies as a power of another. For instance, considering the area of a square in terms of the length of its side, if the length is doubled, the area is multiplied by a factor of four.A power law distribution has the form Y = k Xα, where:X and Y are variables of interest,α is the law’s exponent,k is a constant.Source: https://en.wikipedia.org/wiki/Power_law Power-law distribution is just one of many probability distributions, but it is considered a valuable tool to assess uncertainty issues that normal distribution cannot handle when they occur at a certain probability.Many processes have been found to follow power laws over substantial ranges of values. From the distribution in incomes, size of meteoroids, earthquake magnitudes, the spectral density of weight matrices in deep neural networks, word usage, number of neighbors in various networks, etc. (Note: The power law here is a continuous distribution. The last two examples are discrete, but on a large scale can be modeled as if continuous).Also read: Statistical Measures of Central TendencyPower-law distribution in Python(Source) Let us plot the Pareto distribution which is one form of a power-law probability distribution. Pareto distribution is sometimes known as the Pareto Principle or ‘80–20’ rule, as the rule states that 80% of society’s wealth is held by 20% of its population. Pareto distribution is not a law of nature, but an observation. It is useful in many real-world problems. It is a skewed heavily tailed distribution.import numpy as np import matplotlib.pyplot as plt from scipy.stats import pareto x_m = 1 #scale alpha = [1, 2, 3] #list of values of shape parameters plt.figure(figsize=(10,6)) samples = np.linspace(start=0, stop=5, num=1000) for a in alpha: output = np.array([pareto.pdf(x=samples, b=a, loc=0, scale=x_m)]) plt.plot(samples, output.T, label='alpha {0}' .format(a)) plt.xlabel('samples', fontsize=15) plt.ylabel('PDF', fontsize=15) plt.title('Probability Density function', fontsize=15) plt.legend(loc='best') plt.show()Concept #5- Box cox transformationThe Box-Cox transformation transforms our data so that it closely resembles a normal distribution.The one-parameter Box-Cox transformations are defined as In many statistical techniques, we assume that the errors are normally distributed. This assumption allows us to construct confidence intervals and conduct hypothesis tests. By transforming your target variable, we can (hopefully) normalize our errors (if they are not already normal).Additionally, transforming our variables can improve the predictive power of our models because transformations can cut away white noise.Original distribution(Left) and near-normal distribution after applying Box cox transformation. Source  At the core of the Box-Cox transformation is an exponent, lambda (λ), which varies from -5 to 5. All values of λ are considered and the optimal value for your data is selected; The “optimal value” is the one that results in the best approximation of a normal distribution curve. The one-parameter Box-Cox transformations are defined as:and the two-parameter Box-Cox transformations as:Moreover, the one-parameter Box-Cox transformation holds for y > 0, i.e. only for positive values and two-parameter Box-Cox transformation for y > -λ, i.e. negative values. The parameter λ is estimated using the profile likelihood function and using goodness-of-fit tests.If we talk about some drawbacks of Box-cox transformation, then if interpretation is what you want to do, then Box-cox is not recommended. Because if λ is some non-zero number, then the transformed target variable may be more difficult to interpret than if we simply applied a log transform.A second stumbling block is that the Box-Cox transformation usually gives the median of the forecast distribution when we revert the transformed data to its original scale. Occasionally, we want the mean and not the median.Box-Cox transformation in Python(Source)SciPy’s stats package provides a function called boxcox for performing box-cox power transformation that takes in original non-normal data as input and returns fitted data along with the lambda value that was used to fit the non-normal distribution to normal distribution.#load necessary packages import numpy as np from scipy.stats import boxcox import seaborn as sns #make this example reproducible np.random.seed(0) #generate dataset data = np.random.exponential(size=1000) fig, ax = plt.subplots(1, 2) #plot the distribution of data values sns.distplot(data, hist=False, kde=True, kde_kws = {'shade': True, 'linewidth': 2}, label = "Non-Normal", color ="red", ax = ax[0]) #perform Box-Cox transformation on original data transformed_data, best_lambda = boxcox(data) sns.distplot(transformed_data, hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 2}, label = "Normal", color ="red", ax = ax[1]) #adding legends to the subplots plt.legend(loc = "upper right") #rescaling the subplots fig.set_figheight(5) fig.set_figwidth(10) #display optimal lambda value print(f"Lambda value used for Transformation: {best_lambda}") Concept #6- Poisson distributionIn probability theory and statistics, the Poisson distribution is a discrete probability distribution that expresses the probability of a given number of events occurring in a fixed interval of time or space if these events occur with a known constant mean rate and independently of the time since the last event.In very simple terms, A Poisson distribution can be used to estimate how likely it is that something will happen "X" number of times. Some examples of Poisson processes are customers calling a help center, radioactive decay in atoms, visitors to a website, photons arriving at a space telescope, and movements in a stock price. Poisson processes are usually associated with time, but they do not have to be. The Formula for the Poisson Distribution Is:Where:e is Euler's number (e = 2.71828...)k is the number of occurrencesk! is the factorial of kλ is equal to the expected value of kwhen that is also equal to its varianceLambda(λ) can be thought of as the expected number of events in the interval. As we change the rate parameter, λ, we change the probability of seeing different numbers of events in one interval. The below graph is the probability mass function of the Poisson distribution showing the probability of a number of events occurring in an interval with different rate parameters.   Probability Mass function for Poisson Distribution with varying rate parameters.Source The Poisson distribution is also commonly used to model financial count data where the tally is small and is often zero. For one example, in finance, it can be used to model the number of trades that a typical investor will make in a given day, which can be 0 (often), or 1, or 2, etc.As another example, this model can be used to predict the number of "shocks" to the market that will occur in a given time period, say over a decade.Poisson distribution in Pythonfrom numpy import random import matplotlib.pyplot as plt import seaborn as sns lam_list = [1, 4, 9] #list of Lambda values plt.figure(figsize=(10,6)) samples = np.linspace(start=0, stop=5, num=1000) for lam in lam_list: sns.distplot(random.poisson(lam=lam, size=10), hist=False, label='lambda {0}'.format(lam)) plt.xlabel('Poisson Distribution', fontsize=15) plt.ylabel('Frequency', fontsize=15) plt.legend(loc='best') plt.show()As λ becomes bigger, the graph looks more like a normal distribution.I hope you have enjoyed reading this article, If you have any questions or suggestions, please leave a comment. Also read: False Positives vs. False NegativesFeel free to connect me on LinkedIn for any query.Thanks for reading!!!Referenceshttps://calcworkshop.com/joint-probability-distribution/chebyshev-inequality/ https://corporatefinanceinstitute.com/resources/knowledge/data-analysis/chebyshevs-inequality/ https://www.itl.nist.gov/div898/handbook/eda/section3/eda3669.htm https://www.statology.org/q-q-plot-python/ https://gist.github.com/chaipi-chaya/9eb72978dbbfd7fa4057b493cf6a32e7 https://stackoverflow.com/a/41968334/7175247 

  17 minutes read
• 

  N/A
  Top 10 Python Extensions for Visual Studio Code

  Python

  In this new post we want to talk about the most useful Python extensions for Visual Studio Code. Visual Studio Code is an integrated development environment created by Microsoft for Windows, Linux and macOS. Among its features are debugging, syntax highlighting, smart code completion, snippets, code refactoring and integrated Git. Users can change the theme, keyboard shortcuts, preferences and install extensions that add additional functionality.Precisely we are going to talk about the extensions you can install for VS. Here is a list of our favorites1- PythonLink: https://github.com/Microsoft/vscode-pythonPython extension for Visual Studio CodeA Visual Studio Code extension with rich support for the Python language (for all actively supported versions of the language: >=3.6), including features such as IntelliSense (Pylance), linting, debugging, code navigation, code formatting, refactoring, variable explorer, test explorer, and more!NOTE: Web support -- e.g., github.dev -- is limited.Installed extensionsThe Python extension will automatically install the Pylance and Jupyter extensions to give you the best experience when working with Python files and Jupyter notebooks. However, Pylance is an optional dependency, which means that the Python extension will remain fully functional if it is not installed. You can also uninstall it at the expense of some features if you are using a different language server.2- Python IndentLink: https://github.com/kbrose/vsc-python-indentIt is used to correct Python indentation in Visual Studio Code. How it worksEvery time you press the Enter key in a Python context, this extension will parse your Python file down to the location of your cursor, and determine exactly how much to indent the next line (or two in the case of hanging indents) and how much to indent nearby lines.There are three main cases when determining the correct indent. Review the documentation here: https://github.com/kbrose/vsc-python-indent3- Python Doctring GeneratorLink: https://github.com/NilsJPWerner/autoDocstringVisual Studio Code extension to quickly generate docstrings for python functions.FeaturesQuickly generate a docstring fragment that can be tabbed.Choose from several types of docstring formats.Infer parameter types via pep484 type hints, default values and var names.Support for args, kwargs, decorators, errors and parameter types.Docstring FormatsGoogle (default)docBlockrNumpySphinxPEP0257 (coming soon)UsageThe cursor must be on the line directly below the definition to generate a complete auto-populated docstring.Press enter after opening the docstring with triple quotes ("""" or ''')Keyboard shortcut: ctrl+shift+2 or cmd+shift+2 for macCan be changed in Preferences -> Keyboard shortcuts -> extension.generateDocstringCommand: Generate DocstringRight-click menu: Generate DocstringAlso read: 4 Must-Know Python Pandas Functions for Time Series Analysis4- Python ExtendedLink: https://github.com/tushortz/vscode-Python-ExtendedPython Extended is a vscode snippet that makes it easy to write Python code by providing completion options along with all arguments.UsageRun vscode and in a python file, type the name of the method to complete and press tab or enter on selection.How to installOpen vscode. Press F1, search for "ext install" followed by the extension name, in this case "ext install Python Extended" without the ">". Or if you prefer ">ext install", press enter, search for "Python Extended".5- Python PreviewLink: https://github.com/dongli0x00/python-previewA Visual Studio Code extension with debug preview support for the Python language.RequirementsInstall a version of Python 3.6 or Python 2.7. Make sure that the location of your Python interpreter is included in your PATH environment variable.It is best to install the Python extension for Python Intellisense.6- AREPL for PythonLink: https://github.com/almenon/arepl-vscodeAREPL automatically evaluates Python code in real time as you type.UsageFirst, make sure you have python 3.7 or higher installed.Open a python file and click on the cat in the top right bar to open AREPL. You can click the cat again to close it.Or run AREPL via the search command: control-shift-por use the shortcuts: control-shift-a (current document) / control-shift-q (new document)FeaturesReal-time evaluation: no need to run - AREPL evaluates your code automatically. You can control this (or even disable it) in the settings.Variable display: The final state of your local variables is displayed in a collapsible JSON format.Error display: The moment you make a mistake an error is displayed with the stack trace.Settings: AREPL offers many settings to suit your user experience. Customize the look and feel, bounce time, python options and much more.Aldo read: 3 Python Tricks That Will Improve Your Code7- Python PathLink: https://github.com/mgesbert/vscode-python-pathThis extension adds a set of tools to help generate internal import statements in a Python project.Features"Copy Python Path" is accessible from:Command lineExplorer context menuEditor context menuEditor title context menu8- Python Test ExplorerLink: https://github.com/kondratyev-nv/vscode-python-test-adapterThis extension allows you to run your Python Unittest, Pytest or Testplan tests with the Test Explorer user interface.How to get startedInstall the extensionConfigure Visual Studio Code to discover your tests (see the Configuration section and the documentation for the test framework of your choice:Unittest documentationPytest documentationTestplan documentationOpen the sidebar of the test viewExecute your tests via the Run icon in the Test ExplorerFeaturesDisplays a Test Explorer in the test view in the VS Code sidebar with all detected tests and suites and their statusConvenient error reporting during test detectionUnittest, Pytest and Testplan debuggingDisplays the log of a failed test when the test is selected in the explorerTest rerun when saving testsSupports multi-root workspacesSupports Unittest, Pytest and Testplan test frameworks and their plugins9- Python SnippetsLink: https://github.com/ylcnfrht/vscode-python-snippet-packA snippet package to make working with Python more productive This snippet package contains all of the following Python methodsall built-in python snippets and contains at least one example for each methodall python string snippets contain at least one example for each methodall python list snippets contain at least one example for each methodall Python set snippets contain at least one example for each methodall Python tuple snippets contain at least one example for each methodall python dictionary snippets contain at least one example for each methodAnd it contains many other code snippets (such as if/else, for, while, while/else, try/catch, file process, andclass snippets and class examples for oop (polymorphism, encapsulation, inheritance, etc.).If you don't use a method don't worry this extension contains a lot of code examples for each python method.This extension is not just a code snippet, it will also be useful for learning the python programming language.You will learn all python methods with a lot of code examples.For example, if you want to use the string replacement method, you just need to use .replace.But if you don't know how to use the replace method then use string.replace =>10- JupyterLink: https://github.com/Microsoft/vscode-jupyterA Visual Studio Code extension that provides basic notebook support for language kernels that are compatible with Jupyter Notebooks today. Many language kernels will work without any modifications. To enable advanced features, modifications to the VS Code language extensions may be necessary.Notebook supportThe Jupyter Extension uses VS code's built-in notebook support. This interface offers a number of advantages to notebook users:Out-of-the-box support for VS Code's wide range of basic code editing functions, such as hot output, search and replace, and code folding.Editor extensions such as VIM, bracket coloring, linters and many more are available while editing a cell.Deep integration with the general workbench and file-based features of VS Code, such as outline view (table of contents), breadcrumbs, and other operations.Fast load times for Jupyter notebook (.ipynb) files. Any notebook file is loaded and rendered as quickly as possible, while execution-related operations are initialized behind the scenes.Includes a notebook diff tool, which makes it easy to compare and visualize differences between code cells, results and metadata.Extensibility beyond what the Jupyter extension provides. Extensions can now add their own specific language or runtime to notebooks, such as the .NET and Gather interactive notebooks.Although the Jupyter extension comes with a comprehensive set of the most commonly used renderers for output, the marketplace supports installable custom renderers to make working with your notebooks even more productive. To get started writing your own, check out the VS Code renderer api documentation.You can also read data science posts in Spanish here.ConclusionThere are many extensions that you can use with your Visual Studio Code, and deciding which one to use will involve testing, reviewing utilities, use cases and so on in order to make your work easier while coding!Also read: Why Decorators In Python Are Pure Genius?

  7 minutes read
• 

  N/A
  10 Highly Probable Data Scientist Interview Questions

  Machine Learning

  The popularity of data science attracts a lot of people from a wide range of professions to make a career change with the goal of becoming a data scientist.Despite the high demand for data scientists, it is a highly challenging task to find your first job. Unless you have a solid prior job experience, interviews are where you can show you skills and impress your potential employer.Data science is an interdisciplinary field which covers a broad range of topics and concepts. Thus, the number of questions that you might be asked at an interview is very high.However, there are some questions about the fundamentals in data science and machine learning. These are the ones you do not want to miss. In this article, we will go over 10 questions that are likely to be asked at a data scientist interview.The questions are grouped into 3 main categories which are machine learning, Python, and SQL. I will try to provide a brief answer for each question. However, I suggest reading or studying each one in more detail afterwards.Machine Learning1. What is overfitting?Overfitting in machine learning occurs when your model is not generalized well. The model is too focused on the training set. It captures a lot of detail or even noise in the training set. Thus, it fails to capture the general trend or the relationships in the data. If a model is too complex compared to the data, it will probably be overfitting.A strong indicator of overfitting is the high difference between the accuracy of training and test sets. Overfit models usually have very high accuracy on the training set but the test accuracy is usually unpredictable and much lower than the training accuracy.2. How can you reduce overfitting?We can reduce overfitting by making the model more generalized which means it should be more focused on the general trend rather than specific details.If it is possible, collecting more data is an efficient way to reduce overfitting. You will be giving more juice to the model so it will have more material to learn from. Data is always valuable especially for machine learning models.Another method to reduce overfitting is to reduce the complexity of the model. If a model is too complex for a given task, it will likely result in overfitting. In such cases, we should look for simpler models.3. What is regularization?We have mentioned that the main reason for overfitting is a model being more complex than necessary. Regularization is a method for reducing the model complexity.It does so by penalizing higher terms in the model. With the addition of a regularization term, the model tries to minimize both loss and complexity.Two main types of regularization are L1 and L2 regularization. L1 regularization subtracts a small amount from the weights of uninformative features at each iteration. Thus, it causes these weights to eventually become zero.On the other hand, L2 regularization removes a small percentage from the weights at each iteration. These weights will get closer to zero but never actually become 0.4. What is the difference between classification and clustering?Both are machine learning tasks. Classification is a supervised learning task so we have labelled observations (i.e. data points). We train a model with labelled data and expect it to predict the labels of new data.For instance, spam email detection is a classification task. We provide a model with several emails marked as spam or not spam. After the model is trained with those emails, it will evaluate the new emails appropriately.Clustering is an unsupervised learning task so the observations do not have any labels. The model is expected to evaluate the observations and group them into clusters. Similar observations are placed into the same cluster.In the optimal case, the observations in the same cluster are as close to each other as possible and the different clusters are as far apart as possible. An example of a clustering task would be grouping customers based on their shopping behavior.PythonThe built-in data structures are of crucial importance. Thus, you should be familiar with what they are and how to interact with them. List, dictionary, set, and tuple are 4 main built-in data structures in Python.5. What is the difference between lists and tuplesThe main difference between lists and tuples is mutability. Lists are mutable so we can manipulate them by adding or removing items.mylist = [1,2,3] mylist.append(4) mylist.remove(1) print(mylist) [2,3,4]On the other hand, tuples are immutable. Although we can access each element in a tuple, we cannot modify its content.mytuple = (1,2,3) mytuple.append(4) AttributeError: 'tuple' object has no attribute 'append'One important point to mention here is that although tuples are immutable, they can contain mutable elements such as lists or sets.mytuple = (1,2,["a","b","c"]) mytuple[2] ['a', 'b', 'c'] mytuple[2][0] = ["A"] print(mytuple) (1, 2, [['A'], 'b', 'c'])6. What is the difference between lists and setsLet’s do an example to demonstrate the main difference between lists and sets.text = "Python is awesome!" mylist = list(text) myset = set(text) print(mylist) ['P', 'y', 't', 'h', 'o', 'n', ' ', 'i', 's', ' ', 'a', 'w', 'e', 's', 'o', 'm', 'e', '!'] print(myset) {'t', ' ', 'i', 'e', 'm', 'P', '!', 'y', 'o', 'h', 'n', 'a', 's', 'w'}As we notice in the resulting objects, the list contains all the characters in the string whereas the set only contains unique values.Another difference is that the characters in the list are ordered based on their location in the string. However, there is no order associated with the characters in the set.Here is a table that summarizes the main characteristics of lists, tuples, and sets.(image by author)7. What is a dictionary and what are the important features of dictionaries?A dictionary in Python is a collection of key-value pairs. It is similar to a list in the sense that each item in a list has an associated index starting from 0.mylist = ["a", "b", "c"] mylist[1] "b"In a dictionary, we have keys as the index. Thus, we can access a value by using its key.mydict = {"John": 24, "Jane": 26, "Ashley": 22} mydict["Jane"] 26The keys in a dictionary are unique which makes sense because they act like an address for the values.SQLSQL is an extremely important skill for data scientists. There are quite a number of companies that store their data in a relational database. SQL is what is needed to interact with relational databases.You will probably be asked a question that involves writing a query to perform a specific task. You might also be asked a question about general database knowledge.8. Query example 1Consider we have a sales table that contains daily sales quantities of products.SELECT TOP 10 * FROM SalesTable(image by author)Find the top 5 weeks in terms of total weekly sales quantities.SELECT TOP 5 CONCAT(YEAR(SalesDate), DATEPART(WEEK, SalesDate)) AS YearWeek, SUM(SalesQty) AS TotalWeeklySales FROM SalesTable GROUP BY CONCAT(YEAR(SalesDate), DATEPART(WEEK, SalesDate)) ORDER BY TotalWeeklySales DESC (image by author)We first extract the year and week information from the date column and then use it in the aggregation. The sum function is used to calculate the total sales quantities.9. Query example 2In the same sales table, find the number of unique items that are sold each month.SELECT MONTH(SalesDate) AS Month, COUNT(DISTINCT(ItemNumber)) AS ItemCount FROM SalesTable GROUP BY MONTH(SalesDate) Month ItemCount 1 9 1021 2 8 102110. What is normalization and denormalization in a database?These terms are related to database schema design. Normalization and denormalization aim to optimize different metrics.The goal of normalization is to reduce data redundancy and inconsistency by increasing the number of tables. On the other hand, denormalization aims to speed up the query execution. Denormalization decreases the number of tables but at the same time, it adds some redundancy.ConclusionIt is a challenging task to become a data scientist. It requires time, effort, and dedication. Without having prior job experience, the process gets harder.Interviews are very important to demonstrate your skills. In this article, we have covered 10 questions that you are likely to encounter in a data scientist interview.Thank you for reading. Please let me know if you have any feedback.Soner Yıldırım

  7 minutes read
• 

  N/A
  5 Google Chrome Extensions Every Data Scientist Should Know About

  Machine Learning

  In this new post we will talk about the best Google Chrome extensions that as data scientists make certain tasks easier for us. You should at least try them out and verify the results on your own, it is possible that they will become part of your arsenal of tools to work with on a daily basis. If you have any additional recommendations feel free to post them in the comments, or if you have any reviews on the mentioned extensions we would also like to hear from you!#1- AI/ML Papers with Code Everywhere - CatalyzeXExtension description:💻 Get the implementation code of machine learning research papers with code directly on Google, ArXiv, Scholar, Twitter, Github, etc.CatalyzeX free browser extension instantly finds and displays the implementation code of any machine learning, artificial intelligence, natural language processing, computer vision, reinforcement learning, and deep learning research papers with code you find while browsing Google, ArXiv, Twitter, Scholar, Github, and other websites.▶ Browse the web as usual and you'll start seeing [CODE] buttons next to documents everywhere.▶ The [CODE] buttons will automatically appear directly on the page of any implementation link found.▶ Click any code button to instantly jump to open source and explore!Powered by the CatalyzeX.com repository used by thousands of engineers, researchers, developers and technology leaders from around the world - to browse ML algorithms, models and articles with code for relevant and cutting-edge developments in machine learning, data science, artificial intelligence, computer vision, natural language processing, speech recognition, reinforcement learning and deep learning.Download link: https://chrome.google.com/webstore/detail/aiml-papers-with-code-eve/aikkeehnlfpamidigaffhfmgbkdeheil#2- Instant Data ScraperExtension description:Instant Data Scraper extracts data from web pages and exports it as Excel or CSV files.Instant Data Scraper is an automated data extraction tool for any website. It uses AI to predict what data is most relevant on an HTML page and allows you to save it to an Excel or CSV file (XLS, XLSX, CSV).This tool does not require website-specific scripts, but uses AI heuristic analysis of the HTML structure to detect the data to extract. If the prediction is not satisfactory, it allows the user to customize the selections for greater accuracy. This type of scrape technology is much more convenient, because it does not require large libraries of user-created scrape scripts, which are often filled with obsolete and redundant versions. This means that our scrape method works just as well with small, lesser-known websites as it does with global giants like Amazon. Plus, our users don't need to have any coding, json or xml knowledge!Instant Data Scraper features:* Data detection for extraction with AI.* Detection of dynamic data when it has been loaded.* Customization of delay and timeout for desired crawl speed.* Support for pagination on web sites.* Automatic navigation to the next page via buttons or links.* Support for infinite scroll.* Preview of extracted data with copy and paste support.* Data export to Excel spreadsheet or CSV file.* Rename and filter extracted data columns.This extension is completely FREE. Instant Data Scraper works well in unison with SEO tools, CRM recruiting systems, sales lead management tools or email marketing campaigns. Web scraping and data downloading is made easy with our tool. Plus, you have data security and privacy as scraped data does not leave your browser.Instant Data Scraper USE CASES:* Lead generation for companies and freelancers.* Growth hackers looking for easy ways to collect data.* Recruiters looking for job candidates.* Obtaining search engine results.* Obtaining product pricing data from e-commerce websites.* Amazon sellers, distributors and review analysts.* Obtain email and directory addresses and phone numbers.* Get contact information from professional association websites.* Collect reviews and ratings.* Analyze posts for likes, comments, connection and contacts.* Extract emails and IDs from social network profiles.Developer's promise: This extension does not contain any malware or spyware beyond the standard Google Analytics.Download link: https://chrome.google.com/webstore/detail/instant-data-scraper/ofaokhiedipichpaobibbnahnkdoiiah#3- EquatIO - Math made digitalExtension description:Easily create mathematical equations, formulas and quizzes. Type intuitively or by hand, without the need to learn complicated mathematical code.Until now, writing mathematical equations and expressions on the computer was slow and laborious.EquatIO makes mathematics digital, helping teachers and students of all levels to create mathematical expressions quickly and easily.Type or handwrite virtually any mathematical expression directly on your keyboard or touch screen. There is no need for any complicated code or programming languages.EquatIO instantly converts your thoughts into clear and precise formulas on the screen. When you type "sq", it knows you mean "square root". And it listens to you, too. When you dictate a formula out loud, EquatIO is smart enough to ignore "umms," "errs," and other unwanted non-math words.More than a substitute for pencil and paper, EquatIO makes math and other STEM subjects engaging and rewarding for all students.Features:- Easily create mathematical expressions, including equations and formulas.- Compatible with Google Docs, Forms*, Slides*, Sheets* and Drawings*.- Keyboard input, handwriting recognition* (via touchscreen or touchpad), and voice dictation- Guess what you're writing or typing, just like predictive text on your smartphone*. - Hear your mathematical expressions read aloud.- Compatible with LaTeX input for advanced users.- Create interactive digital math quizzes*.- Enter student responses directly into digital quizzes*.- Chemistry formula prediction*.Download link: https://chrome.google.com/webstore/detail/equatio-math-made-digital/hjngolefdpdnooamgdldlkjgmdcmcjnc4- Challenge HuntExtension description:One application. All programming competitions. Keeps you up to date with all active and upcoming programming competitions and hackathons from around the world.Challenge Hunt brings you the best programming challenges from around the world. Keeps you up to date with all active and upcoming coding competitions, hackathons, hiring and data science challenges. It allows you to:1. see all active and upcoming: * Coding competitions * Hackathons * Data science challenges * Hiring challenges2. Set reminders about upcoming challenges so you don't miss any challenges.3. Set your preferred online challenge platforms and Challenge Hunt will show you challenges only from those platforms.Download link: https://chrome.google.com/webstore/detail/challenge-hunt/cgbnhngdlmhjbmgkekedepoapmoingjj5- Open in ColabExtension description:Open a notebook hosted on Github in Google Colab. Simple browser extension to quickly open Jupyter notebook hosted on GitHub in Google Colab.Download link: https://chrome.google.com/webstore/detail/open-in-colab/iogfkhleblhcpcekbiedikdehleodpjoThanks for reading us!

  5 minutes read
• 

  N/A
  Data Scientists are Really Just Product Managers. Here’s Why.

  Data Science

  Unpopular opinion?Table of ContentsIntroductionBusiness and Product UnderstandingStakeholder CollaborationSummaryReferencesIntroductionAs mentioned above, the title of this article may be an unpopular opinion, but let me lay it out for you to see what you think with a new perspective. If you are in school still, this statement might sound quite surprising, however, if have been in the tech business for a few years, this statement may come as no surprise to you. Of course, this statement will not be true for everyone, all of the time, but it gets pretty close when you point out the similarities between these two product roles. With that being said, I will outline some of the similarities between data science and product manager roles, skills required, and goals below.Business and Product UnderstandingPhoto by Hugo Rocha on Unsplash [2].When you work as a data scientist, it is essential that you know the business, its goals, as well as its shortcomings, the same can be said about product managers. The main difference between these roles is the method by which their specific goals are reached, but overall, the main goals are shared. Here is an example:Data Science Goal:Identify problems in the productUse data science models as a solution to a product problemUse data science models as a productAnalyze data to predict future data with algorithmsProduct Manager Goal:Identify problems in the productRank which problems are of most concernLook into data to learn about future data trendsOf course, data science can have a focus on coding, but without an understanding of the business and product, a data science model can be completely useless. It is essential to have this understanding for both roles, as this knowledge is how you come up with next steps, a process, problem diagnosis, and eventual solutions.On the company structure level, you will see that data scientists are often the bridge between departments or a part of both engineering and product. That statement alone can show that there is considerable overlap.Here are some skills that both roles share:Data explorationProblem diagnosisVisualization tools (Tableau, Looker, Lucidchart, etc.)Querying (SQL)In addition to some of the skills shared between roles, there is also a similar process:Product problem isolation or product improvementData analysisCurrent process analysisSolutions (this is where data science differs by being the ones who will create the model, product managers will organize this step, with things like results, costs, timeline, and how it will ultimately affect the business, but data scientists can also share some of that work)Results presentationTestingExecutive approval/approval in generalImplementationAs you can see, some skills, process steps, and data science and product management goals are shared. Sometimes, there are no product managers at smaller companies, and therefore, data scientists will have to work that role as well.Stakeholder CollaborationPhoto by Mimi Thian on Unsplash [3].When you work as a data scientist, you will have to collaborate with several verticals of the business and their respective stakeholders. The same can be said for product managers. Both data scientists and product managers can also be stakeholders themselves.Here is where these roles share stakeholder collaboration:Proof of concepts with software engineers, data analysts, and executives, etc.Level of effort analysis with the same roles from above, and moreSetting up meetings with those rolesUpdating those roles on stepsAllocating work to othersOverall, both data scientists and product managers can prove to be cross-functional in their work, as well as in who they collaborate with, whether that be a data analyst, salesperson, or software engineer (etc.).SummaryThe goal of this article is not to say one role is better than another, but to highlight that both roles have a considerable amount of overlap in their daily tasks, skills required, who they work with, and overall goals. To be more specific, maybe data scientists are just product managers who have special skills and focus on algorithms.To summarize, here are some ways that data scientists and product managers are similar:* Business and Product Understanding * Stakeholder CollaborationI hope you found my article both interesting and useful. Please feel free to comment down below if you agree or disagree with these comparisons between roles. Why or why not? What other comparisons (or differences) do you think are important to point out? These can certainly be clarified even further, but I hope I was able to shed some light on some of the common similarities between data scientists and product managers. Thank you for reading!I am not affiliated with any of these companies.Please feel free to check out my profile,  Matt Przybyla, and other articles, as well as subscribe to receive email notifications for my blogs by following the link below, or by clicking on the subscribe icon on the left of the screen, and reach out to me on LinkedIn if you have any questions or comments.

  4 minutes read
• 

  N/A
  16 Underrated Pandas Series Methods And When To Use Them

  Libraries Pandas

  In this article, we’re going to explore some lesser-known but very useful pandas methods for manipulating Series objects. Some of these methods are related only to Series, the others — both to Series and DataFrames, having, however, specific features when used with both structure types.1. is_uniqueAs its name sugests, this method checks if all the values of a Series are unique:import pandas as pd print(pd.Series([1, 2, 3, 4]).is_unique) print(pd.Series([1, 2, 3, 1]).is_unique) Output: True False 2 & 3. is_monotonic and is_monotonic_decreasingWith these 2 methods, we can check if the values of a Series are in ascending/descending order:print(pd.Series([1, 2, 3, 8]).is_monotonic) print(pd.Series([1, 2, 3, 1]).is_monotonic) print(pd.Series([9, 8, 4, 0]).is_monotonic_decreasing) Output: True False TrueBoth methods work also for a Series with string values. In this case, Python uses a lexicographical ordering under the hood, comparing two subsequent strings character by character. It’s not the same as just an alphabetical ordering, and actually, the example with the numeric data above is a particular case of such an ordering. As the Python documentation says,Lexicographical ordering for strings uses the Unicode code point number to order individual characters.In practice, it mainly means that the letter case and special symbols are also taken into account:print(pd.Series(['fox', 'koala', 'panda']).is_monotonic) print(pd.Series(['FOX', 'Fox', 'fox']).is_monotonic) print(pd.Series(['*', '&', '_']).is_monotonic) Output: True True FalseA curious exception happens when all the values of a Series are the same. In this case, both methods return True:print(pd.Series([1, 1, 1, 1, 1]).is_monotonic) print(pd.Series(['fish', 'fish']).is_monotonic_decreasing) Output: True TrueAlso Read: 4 Must-Know Python Pandas Functions for Time Series Analysis4. hasnansThis method checks if a Series contains NaN values:import numpy as np print(pd.Series([1, 2, 3, np.nan]).hasnans) print(pd.Series([1, 2, 3, 10, 20]).hasnans) Output: True False5. emptySometimes, we might want to know if a Series is completely empty, not containing even NaN values:print(pd.Series().empty) print(pd.Series(np.nan).empty) Output: True FalseA Series can become empty after some manipulations with it, for example, filtering:s = pd.Series([1, 2, 3]) s[s > 3].empty Output: True 6 & 7. first_valid_index() and last_valid_index()These 2 methods return index for first/last non-NaN value and are particularly useful for Series objects with many NaNs:print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).first_valid_index()) print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).last_valid_index()) Output: 2 4If all the values of a Series are NaN, both methods return None:print(pd.Series([np.nan, np.nan, np.nan]).first_valid_index()) print(pd.Series([np.nan, np.nan, np.nan]).last_valid_index()) Output: None None 8. truncate()This method allows truncating a Series before and after some index value. Let’s truncate the Series from the previous section leaving only non-NaN values:s = pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]) s.truncate(before=2, after=4) Output: 2 1.0 3 2.0 4 3.0 dtype: float64The original index of the Series was preserved. We may want to reset it and also to assign the truncated Series to a variable:s_truncated = s.truncate(before=2, after=4).reset_index(drop=True) print(s_truncated) Output: 0 1.0 1 2.0 2 3.0 dtype: float64Also Read: Pandas vs SQL. When Data Scientists Should Use One Over the Other9. convert_dtypes()As the pandas documentation says, this method is used toConvert columns to best possible dtypes using dtypes supporting pd.NA.If to consider only Series objects and not DataFrames, the only application of this method is to convert all nullable integers (i.e. float numbers with a decimal part equal to 0, such as 1.0, 2.0, etc.) back to “normal” integers. Such float numbers appear when the original Series contains both integers and NaN values. Since NaN is a float in numpy and pandas, it leads to the whole Series with any missing values to become of float type as well.Let’s take a look at the example from the previous section to see how it works:print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan])) print('\n') print(pd.Series([np.nan, np.nan, 1, 2, 3, np.nan]).convert_dtypes()) Output: 0 NaN 1 NaN 2 1.0 3 2.0 4 3.0 5 NaN dtype: float64 0 <NA> 1 <NA> 2 1 3 2 4 3 5 <NA> dtype: Int64 10. clip()We can clip all the values of a Series at input thresholds (lower and upper parameters):s = pd.Series(range(1, 11)) print(s) s_clipped = s.clip(lower=2, upper=7) print(s_clipped) Output: 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 dtype: int64 0 2 1 2 2 3 3 4 4 5 5 6 6 7 7 7 8 7 9 7 dtype: int64 11. rename_axis()In the case of a Series object, this method sets the name of the index:s = pd.Series({'flour': '300 g', 'butter': '150 g', 'sugar': '100 g'}) print(s) s=s.rename_axis('ingredients') print(s) Output: flour 300 g butter 150 g sugar 100 g dtype: object ingredients flour 300 g butter 150 g sugar 100 g dtype: object 12 & 13. nsmallest() and nlargest()These 2 methods return the smallest/largest elements of a Series. By default, they return 5 values, in ascending order for nsmallest() and in descending - for nlargest().s = pd.Series([3, 2, 1, 100, 200, 300, 4, 5, 6]) s.nsmallest() Output: 2 1 1 2 0 3 6 4 7 5 dtype: int64It’s possible to specify another number of the smallest/largest values to be returned. Also, we may want to reset the index and assign the result to a variable:largest_3 = s.nlargest(3).reset_index(drop=True) print(largest_3) Output: 0 300 1 200 2 100 dtype: int64Also Read: Pandas vs SQL. When Data Scientists Should Use One Over the Other14. pct_change()For a Series object, we can calculate percentage change (or, more precisely, fraction change) between the current and a prior element. This approach can be helpful, for example, when working with time series, or for creating a waterfall chart in % or fractions.s = pd.Series([20, 33, 14, 97, 19]) s.pct_change() Output: 0 NaN 1 0.650000 2 -0.575758 3 5.928571 4 -0.804124 dtype: float64To make the resulting Series more readable, let’s round it:s.pct_change().round(2) Output: 0 NaN 1 0.65 2 -0.58 3 5.93 4 -0.80 dtype: float64 15. explode()This method transforms each list-like element of a Series (lists, tuples, sets, Series, ndarrays) to a row. Empty list-likes will be transformed in a row with NaN. To avoid repeated indices in the resulting Series, it’s better to reset index:s = pd.Series([[np.nan], {1, 2}, 3, (4, 5)]) print(s) s_exploded = s.explode().reset_index(drop=True) print(s_exploded) Output: 0 [nan] 1 {1, 2} 2 3 3 (4, 5) dtype: object 0 NaN 1 1 2 2 3 3 4 4 5 5 dtype: object 16. repeat()This method is used for consecutive repeating each element of a Series a defined number of times. Also in this case, it makes sense to reset index:s = pd.Series([1, 2, 3]) print(s) s_repeated = s.repeat(2).reset_index(drop=True) print(s_repeated) Output: 0 1 1 2 2 3 dtype: int64 0 1 1 1 2 2 3 2 4 3 5 3 dtype: int64If the number of repetitions is assigned to 0, an empty Series will be returned:s.repeat(0) Output: Series([], dtype: int64) ConclusionTo sum up, we investigated 16 rarely used pandas methods for working with Series and some of their application cases. If you know some other interesting ways to manipulate pandas Series, you’re very welcome to share them in the comments.Thanks for reading!Also read: Using Python And Pandas Datareader to Analyze Financial Data

  7 minutes read
• 

  N/A
  How to Build Your Data Analytics Team

  Business

  Peer reviewed by Kat Holmes — Data Director ITV‍As businesses recognize the decisive power of data to achieve business goals, most are hoping to put data in the driver’s seat of their business and product strategies. This entails putting together a strong data team which can effectively propagate its insights across different areas of the business. Unfortunately, this is no easy task.To be truly data driven, companies need to build three capabilities: data strategy, data governance and data analytics.3 pillars for data-driven companies — Image from PitchStrategy: Data strategy is your organization’s roadmap for using data to achieve its goals. It requires a clear understanding of the data needs inherent to the business strategy. Why are you collecting data? Are you trying to make money, save money, manage risk, deliver exceptional customer experience, all the above?Governance: Data governance is a collection of processes, roles, policies, standards, and metrics that ensure the efficient use of information in enabling your organization to achieve its goals. A well-crafted data governance strategy ensures that data in your company is trusted, accurate and available.Analytics: The term data analytics refers to the process of analyzing raw data to draw conclusions about the information they contain. Typically, those involved with data analytics in an organization are data engineers, data analysts and data scientists.Ultimately, your ability to leverage data will depend on these three pillars. If you’re reading this and realizing that your organization possesses none of these, don’t worry. That’s why we’re here. A good place to start is to build a strong analytics team, one that is closely tied with the strategic goals of your business. It is the first pillar of your data organization, and the focus of the article.When building a data analytics team, heads of data typically grapple with the following questions:How big should this team be?How many data engineers, data analysts, data scientists?How does the team interact with the rest of the organization?Which structure for the data team? Centralized or embedded?They rightly do so; having a strong data team is not a luxury anymore, but essential to the very survival of a company today.Let’s start with the basics though.Where are you in your data journey?Before building a data team, it’s important that you realize where you are in your “data journey”, because this will directly affect the structure of your team. This part is thus dedicated to a simplified data maturity assessment. Beware, company size and data maturity are two different things. Your organization can be large but immature on a data level.Data maturity is the journey towards seeing tangible value from your data assets. We propose a simple framework of data maturity assessment, in which you measure your ability to understand your past, know your present and predict your future. What do I mean by this?Well, in most companies each department has its own set of KPIs that support the execution of the corporate strategy. It’s not enough just to define them, they must also be clearly tracked, and you must also have the ability to predict future outcomes against these KPIs. This ability rests on a clear knowledge of your present, which, in turn, builds on a strong understanding of the past. Do this, and you have found a simple way to assess your data maturity. For example, if you’re unable to identify the revenue drivers for your company ( your past), it means you need to work on your data maturity by bringing visibility to your business before you seek to predict future outcomes. We don’t recommend skipping steps. It’s like Maslow’s hierarchy of needs, but for data.Read Also: What Are the Expected Results of a Data Science Project?Data hierarchy of needs — Image by Louise de LeyritzLet’s look at a couple of practical examples:Marketing ROI. Define your ROI, across multiple channels, by using an identified attribution model. Then understand its evolution in the previous 12 months, and especially its drivers (identify performing channels, time of the year, product, ….). Then track on a daily/weekly/monthly basis its evolution thanks to a reporting tool you trust ( present). Forecast your marketing budget based on these predictive models ( future).Customer Satisfaction. Define your customer satisfaction measure. Is it NPS, CSAT? Everyone in your company should share a common understanding of how it is computed. As with our previous example, compute its evolution in the previous 12 months, find its drivers ( past). Then track daily the satisfaction of your customers with trusted dashboards. Identify action to take from today to increase it. Your understanding of the past and the present state of customers satisfaction will allow to predict churn efficiently ( future)Understanding your past and present is commonly referred to as performing descriptive analytics. Descriptive analytics helps an organization understand its performance by providing context to help key stakeholders interpret information. This context is usually in the form of data visualization, including graphs, dashboards, reports and charts. When you are analysing data to forecast the future, you’re engaging in predictive analytics. The idea with predictive analytics is to take historical data, feed it into a machine learning model that considers key patterns. Apply this model to current data, and hope that it will forecast the future. We’ll use the terms of descriptive and predictive analytics throughout the article to refer to understanding the past, present or predicting the future.If you realize that your organization is not fully mature (ie. you don’t have a clear understanding of your past and present), here are our recommendations for what should be the next steps of your data team.Key players on a data analytics teamA data analytics team is usually composed of four core functions, which are detailed below.Data engineer: They are responsible for designing, building, and maintaining datasets that can be leveraged in data projects. As such, data engineers closely work with both data scientists and data analysts. We also include the new role of analytics engineer here, although, in practice, this role lies between analytics and engineering.Data scientist: They use advanced mathematics and statistics, and programming tools to build predictive models. The roles of data scientists and data analysts are pretty similar, but data scientists focuse more on predictive analytics than descriptive analytics.Data analyst: They use data to perform reporting and direct analysis. Whereas data scientists and engineers typically interact with data in its raw or unrefined states, analysts work with data that’s already been cleaned and transformed into more user-friendly formats.Business analyst/ops analyst: They help the organization improve its processes and systems. They focus on dashboarding, answer business questions and propose their interpretation. They are agile and straddle the line between IT and the business to help bridge the gap and improve efficiency. They frequently work with a specific business area such as marketing or finance, and their SQL literacy can range from basic dashboarding to advanced analysis.Head of data analytics: They provide strategic oversight to the data team. Their goal is to create an environment that allows all different parties to access the data they need painlessly, build the skills of the business to draw meaningful insights from the data, and ensure data governance. They also act as a bridge between the data team and the main business unit, acting both as a visionary and a technical lead.‍Read Also: What Is Open Innovation In Data Science?How large should the team be?Different companies will build data teams of different sizes, no one size fits all. We have studied the data team’s structure of 300+ companies, with a 300–1000 employee range and derived the following insights:As a general rule, you should aim to have a total of 5–10% of data analysis savvy employees in your company. Some companies such as Amazon or Facebook are training a huge portion of their employees, but we have excluded them for our analysis.The first hires of a brand-new data teams are often a data engineer and a data analyst. With just these two roles, organizations can already engage in some basic descriptive analytics. When building a larger team, think in terms of the skillset you need. A typical data project requires the following skills: database, software development, machine learning, visualization, collaboration, and communication skills. It is very rare to find individuals who possess all these skills. You should thus be aware of which skill each candidate brings to the table. Regardless of how many people you decide to hire, your team should ideally cover this skill set. Where you are in your data journey also impacts who you hire and at which stage. Generally, data analysts focus on understanding the past. That is, they take the data you have and try to understand the drivers of growth and other metrics. Business analysts/obs analysts are oriented towards the present (dashboarding). Finally, data scientists focus on predicting future outcomes. So, if you have trouble understanding your past, hire a data analyst instead ahead of a data scientist.What should ultimately guide the size of your data team is the number of business problem statements and the complexity of the most serious problems. Look at the size of your roadmap and establish how many people you need to complete your data projects within a reasonable amount of time. If you realize it would take more than a year for your data team to complete its projects, then it’s probably time to expand the team. We also encourage you to look at your run vs build ratio. Members of your data team ‘run’ when they work on daily business operations, focusing on the present performance of the organization. They ‘build’ when they work on long-term projects, such as adding new features to the product. Your data team should be running 2/3 of the time and building 1/3 of the time. If your data team spends all its time focusing on day-to-day needs, you are jeopardising the future of your company, and it is probably time to expand the team.Finally, you might have to make some project-specific hirings. If you’re a fintech conducting a project on fraud detection, or a company specialising in dispatching for logistics, you might want to hire someone who knows the specifics of your industry.How does the data team integrate with the company?There is no perfect structure for an analytics team, and your structure is likely to change many times. If your data team structure hasn’t changed for the past 2 years, then it’s likely to be a sub-optimal structure. Why? Because the data needs of your company are evolving rapidly, calling for an adaptation of your data team’s structure. Also keep in mind that the more static your organization, the harder the next change will be. For this reason, we don’t prescribe a given structure, but rather present the most common models and how they can be suited to different types of businesses.The very first step to take when structuring your data team is to find the data people that already exist in your organization. They might not be just the people with the term “data” in their title, but they could be any employee who’s not afraid of data analysis or has SQL skills already, such as business analysts/ ops analysts. If you don’t take the time to locate pre-existing data people carefully, you are likely to end up with an unplanned data team structure, unlikely to fit your business needs.‍Centralized modelCentralized model for data teams — Image by Louise de LeyritzThe centralized model is the most straightforward structure to implement, and it is usually the first step for companies who aim to be data driven. There are, however, a few drawbacks to this model, which are referenced below. This structure usually leads to a centralized data “platform”, where the data team has access to all the data, and services the whole organization in a variety of projects. All data engineers, analysts and scientists within this team are managed directly by the head of data. With this structure, the data team is reporting in a dotted line to data stakeholders based in business units, in a consultant/client-type relationship.Read Also: The 3 Basic Principles of a Data-Driven CompanyThis flexible model is adaptable to the continuously evolving needs of a growing business. If you’re at the beginning of your data journey, that is, you still struggle to have a clear vision of your past and present, this is the structure we recommend. The data team’s first projects will seek to bring visibility to the business, ensuring all departments in your organization have KPIs and dashboards they can trust. This kind of structure is particularly good for analytics where reusability and data governance are important.Advantages✅ The data team can help with other teams’ projects while working on its own agenda.✅ The team can prioritise projects across the company.✅ There are more opportunities for talent and skillset development in a centralized team. In fact, the data team works on a broader variety of projects, and data engineers, scientists and analysts can benefit from their peers’s insights.✅ The head of data has a centralized view of the company’s strategy and can assign data people to projects that are the most suited to their capabilities.✅ Encourages career growth, as data engineers, scientists and have clear perspectives of seniority roles.‍Drawbacks❌ High chance of disconnect between the data analytics team and other business units. In this model, data engineers and data scientists are not immersed in the day-to-day activities of other teams, making it difficult for them to identify the most relevant problems to tackle.❌ Risk for the analytics group to be reduced to a “support” function, with other departments not taking their responsibilities.❌ As the data team serves the rest of the business, other business units might feel like their needs are not properly addressed, or that the planning process is too bureaucratic and slow.‍Decentralized/Embedded modelDecentralized model for data teams — Image by Louise de LeyritzIn a decentralized model, each department hires its “own” data people, with a centralized data platform. In this model, data analysts and scientists focus on the problems faced by their specific business unit, with little interaction with data people from other areas of the company. With this structure, data analysts report directly to the head of their respective business unit.Advantages✅ Embedded teams of data people are agile and responsive, because they are dedicated to their respective business functions and have good domain knowledge.✅ Product managers can assign data tasks to the people most qualified to work on them.✅ Business data teams don’t have to fight for resources to build their data project because the resources sit in the teams.Drawbacks❌ Lack of source of truth, duplication of data content❌ Data people end up working on redundant issues due to a lack of communication between different teams.❌ The creation of silos leads to productivity erosion since data people can’t draw on their colleagues’ expertise as they do in the centralized model.❌ This model makes it harder to optimally staff data people on different projects.❌ Business managers, usually lacking technical backgrounds, will find it hard to manage data people and understand the quality of their work.‍Federated model/ Centre of excellenceA federated model is most suited to companies that have reached data maturity, have a clear data strategy and engage in predictive analytics.Center of excellence mode l- Image by Louise de LeyritzIn the Centre of Excellence model (COE), data people are embedded in business units, but a centralized group that provides leadership, support and training remains. If data analysts and scientists are deployed across business departments, you would still have a data leader (or a core of data leaders according to company size) who prioritizes and supervises data projects. This ensures that the most beneficial data projects are tackled first.This strategy is most suited to larger, enterprise-scale companies with a clear data roadmap. The centre of excellence model entails a larger data team, as you need data scientists both in the COE and in the different business branches. If you are a small or medium company, your needs might not require a data team of this size.Read Also: How to Make Your Company a Data-driven Organization?This approach retains the advantages of both the centralized and the embedded model. It is a more balanced structure in which the data team’s actions are coordinated, but also keeps the data experts embedded in business units.Again, it’s extremely important that you know who your data people are. When building a centralized team at the beginning of your data journey, make sure you don’t have business analysts/ops embedded in other departments. Otherwise, you will end up with an unwanted mixed model, creating complete chaos in your organization. When creating a COE, you need to ensure it’s wanted and planned.Advantages✅ The Centre of Excellence model provides the advantages of both the centralized and the embedded models.It still presents some drawbacks, though:Drawbacks❌ This model requires an additional layer of coordination and communication needed to ensure alignment between COE and business units.❌ Not fit for purpose for small — medium sized organizations, so these companies can then hook it to the benefits that can come with this hub and spoke model.Final wordsBuilding a strong analytics team is a key pillar you need to build if your company is to become data-driven. The extent to which you will extract business value from data ultimately depends on the strength of this team, and how symbiotic it is with the rest of your business. There is no made-to-order advice for the size, composition and structure of your data team. That’s why you need to understand the data maturity level of your organization, so that you can build a data team suited to your business’ needs and aligned with your business strategy.At Castor, we write about all the processes involved when leveraging data assets: from the modern data stack, to data teams composition, to data governance. Our blog covers the technical and the less technical aspects of creating tangible value from data.At Castor, we are building a data documentation tool for the Notion, Figma, Slack generation. Or data-wise for the Fivetran, Looker, Snowflake, DBT aficionados. We designed our catalog to be easy to use, delightful and friendly.Want to check it out? Reach out to us and we will show you a demo.Originally published at https://www.castordoc.com.

  16 minutes read