Dear readers, Foremost, thank you for your support. I would like to announce that this blog has a new home now. The address is https://duttashi.github.io/blog/ I hope you will continue to support me by reading and providing your invaluable comments. Thank you for your time. Cheers Ashish
In this post, I will discuss on the subtle features of the Scrapy framework responsible for data extraction including building a basic spider. But first, a key points to remember, that are as follows; Preliminaries From the documentation, “Scrapy spiders can return the extracted data as Python dicts” therefore to separate the key-value pair the…
Disclaimer: The objective of this post is purely educational in nature. There are no monetary benefits associated. Introduction In this digital age, we are surrounded by data and a majority of it is in unstructured format. The oxford dictionary defines unstructured as “Without formal organization or structure.”. Websites are a rich source of this unstructured…
“Scrapy is an open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.” This article will walk you through installing Scrapy (on a windows operating system). 1. Preliminaries First, ensure the following dependencies exist on your machine; Step 1: Python version 2.7 as scrapy only…
I begin with a new project. It is from the Kaggle playground wherein the objective is to build a regression model (as the response variable or the outcome or dependent variable is continuous in nature) from a given set of predictors or independent variables. My motivation to work on this project are the following; Help me…
Continuing from my previous post, in this post I will discuss on the inferential and predictive analysis. About the dataset and the problem to solve: a brief The dataset is derived from UCI Machine learning repository and the task is to predict if a donor has donated blood in March 2007 (1 stand for donating blood; 0…
Off lately a considerable number of winner machine learning enthusiasts have used XGBoost as their predictive analytics solution. This algorithm has taken a preceedence over the traditional tree based algorithms like Random Forests and Neural Networks. The acronym Xgboost stands for eXtreme Gradient Boosting package. The creators of this algorithm presented its implementation by winning the Kaggle Otto…
Before starting to build on a predictive model in R, the following assumptions should be taken care off; Assumption 1: The parameters of the linear regression model must be numeric and linear in nature. If the parameters are non-numeric like categorical then use one-hot encoding (python) or dummy encoding (R) to convert them to numeric. Assumption…
A number of reasons can be attributed to when a predictive model crumples such as: Inadequate data pre-processing Inadequate model validation Unjustified extrapolation Over-fitting (Kuhn, 2013) Before we dive into data preprocessing, let me quickly define a few terms that I will be commonly using. Predictor/Independent/Attributes/Descriptors – are the different terms that are used as…
A few common steps in data model building are; Pre-processing the predictor data (predictor – independent variable’s) Estimating the model parameters Selecting the predictors for the model Evaluating the model performance Fine tuning the class prediction rules “One of the first decisions to make when modeling is to decide which samples will be used to…
In a bid to explore and detect variables of interest that were most strongly correlated with each other, I proceeded with the regression analysis (see here for the previous article). But there were challenges; One of the fundamental problem was that the variables were categorical and for regression analysis it is important that the variables…
In a previous article, I had discussed the topic in detail. I will revisit it again for the sake of brevity. I want to find out the predictors that are highly correlated to each other in reference to the operating conditions of water points in Tanzania. Continuing further, I begin the data exploration phase by…
At this point on the project, I am still working on the data preperation for exploration and subsequent analysis. Therefore, I provide below brief guidelines on how I will be performing the data analysis especially the sub chapter on Analysis wherein its not final but only provides the approach that I should consider. I now…
Draft Title Save water! Save life! The association between water point maintenance factors and water consumption Introduction to the Research Question In this research study, I would like to explore the factors that contribute to the depletion of a water point. A water point is an installation such as a well or a tap from…
Introduction The title of this week’s essay is actually derived from the infamous speech (“Give me blood and I promise you freedom!”) by the Indian nationalist Subhash Chandra Bose’s speech delivered in Burma on July 4th 1944. An essay makes more sense if its title can relate to its contents. Thus, after considerable debate on how to aptly title it,…
My thoughts & learnings 
You must be logged in to post a comment.