Introduction

Let’s start with Machine Learning. So the first Machine Learning Algorithm which we are going to observe is Linear Regression. Linear Regression is one of the important and easiest Machine Learning algorithm. Here we are going to see each and every topic related to Linear Regression in detail. So let’s start our journey for Linear Regression.

The name Linear Regression itself tells us that it belongs to Regression algorithm from Supervised Learning. It has the origin from statistics, where it is used to study the relationship between the input and output numerical variables. Linear Regression is used to show the linear relationship between the independent variable (predictor) i.e. on X-axis and the dependent variable (output) i.e. on y-axis. In other words, (y) can be calculated from a linear combination of the input variable (x).

If there is a single dependent variable it is called as Simple or Univariate Linear Regression. If there are more than one (multiple) dependent variable it is called as Multiple or Multivariate Linear Regression. 

Before getting into Mathematics part of Linear Regression. Let’s understand with an easy example.

Can we find out the salary of a person with 3 years of experience?

Yes, we can definetly find out the salary. So if we plot the graph for the above data we get something like this.

Now if we draw a straight line from the data points we can compute the results as below.

If a raise a line from years of experience on x-axis till Salary on y-axis we get the results that the salary for the professional with 3 years of experience is approximately 65,00,000.

from sklearn.linear_model import LinearRegression
#Representing LinearRegression as lr (creating LinearRegression object)
lr = LinearRegression()
#Fit the model using lr.fit()
lr.fit( X_train_lm , y_train_lm 

Introduction

Till now we have learnt about the introduction and saw the importance of Business and Data Analytics. Also we saw how we can select a proper BI tool and various factors of it.

This article marks an inflection point where we will learn how to solve business problems using data analysis. Analytics problem solving involved multiple steps like data cleaning, preparation, modelling, model evaluation etc. The structure used for solving an analytics problem is called as CRISP-DM framework which is known as Cross Industry Standard Process for Data Mining.

As a data analytics professional, you will face many challenges ranging from understanding various business problems to choosing the best technique to solve it. To avoid getting lost, data professionals have developed a robust process to solve virtually any analytics problem in any industry using CRISP-DM framework.

The flow of the framework is shown in below figure –

It involves a series of steps which are quite interesting

1. Business understanding
2. Data understanding
3. Data Preparation
4. Data Modelling
5. Model Evaluation
6. Model Deployment

Lets try to understand each step in a proper manner.

Business Understanding

We now have a framework to solve about various problems, but where do we exactly start? Do we directly go on data? Or do we ask some fundamental questions to understand the problem better?

Imagine you are in going to picnic and your car stop suddenly. You have your toolbox and you want to repair your car. To do so, you need to know first what exactly have gone wrong.

For a data professional, understanding the business is its specific problem is the most important. If you understand the problem clearly you can convert it into a well defined analytics problem. If you understand that business problem, only then you can lay out the brilliant strategy to solve it.

If you don’t understand the business and and jump directly to solve it then your strategies may definitely go wrong.

To understand the business problems, one has to undertake the following steps :

• Determine your business objectives clearly.
• Determine the goal of data analysis.

Data Understanding

After business understanding the next important step is data understanding. When you get your hand on the data for the first time, you would want to know the structure of your data (number of files, rows, columns, etc.), understand how are they related to each other and whether something look weird like negative values, outliers, etc. This step is also crucial because when you undertsnad your data properly you can perform further steps more effectively.

Data understanding may include following steps :

• The type of data sets that are available for analysis.
• The information you can get from the datasets.
• Exploring your data and understanding the depth.
• Performing quality check on the data sets.

Data Preparation

Across various data analytics project, data analysts spend almost 50% to 80% of time on data cleaning and preparation, and therefore data preparation becomes one of the most crucial steps.

Data is vast and are in various files. Collecting all the required data from the files togther and selecting the required columns and rows based on business understanding is a major step in data preparation. After data collection we have to deal with missing values and outliers in the data. Outliers can heavily effect the data and if not treated it can also effect your insights. It is considered as the most important step because the model will be built on the data sets created by in this step.

So some steps which include in data preparation are :

• Select relevant Data
• Integrate Data
• Clean Data
• Construct Data : Derive new features
• Format Data

Data Modelling

The Data Modelling is called as “Heart of Data Analysis”. One can think model as a magical box which takes relevant data as input and gives output you are interested in.

In Data Modelling, various Machine Learning and Deep Learning algorithms are used to make data models to answer your question.

We will study about Data Modelling further in our articles.

Model Evaluation and Deployment

In data analytics, evaluation is when you put everything you have done to litmus tests. If the results obtained from model evaluation is not satisfactory and you re-create the whole process. If the model performs well and gives you accurate results then your data modelling process is successful.

Evaluation is necessary to ensure that your model is robust and effective. Once your evaluation is successful you can further deploy your model on various platforms like cloud, local platforms, software’s, etc.

Conclusion

One of the interesting feature of CRISP-DM framework is that the whole process is iterative in nature. This completes the typical life cycle of a data analytics project.

Introduction

In this group of articles we are going to learn everything about Structured Query Language (SQL) in detail. The articles are bit different as we are not going to repeat everything which we can see in the traditional textbooks or other books, the articles here are going to be 100% practical oriented.

Instead we will provide you with SQL work tools which you will need in your work space. No useless concepts everything is straight to the point and very easy to understand. The topics included in the group of SQL articles are the topics which are used by Data analysts and Data managers in their daily work space. SQL allows this professionals to manipulate large sets of data.

The advantage of learning SQL is that, its syntax is intuitive and used to solve sophisticated tasks quickly.

Why SQL?

Imagine and put yourself in the shoes of the employee or manager who is handling huge amount of data. The professional can be working in various domains like Business Intelligence (BI), Data Science, Database administration or backend web development. This jobs are related to storing large amount of data. Before carrying out any analysis on data first you have to retrieve it. To achieve all this you need SQL.

SQL is known as Structured Query Language. Among different types of programming language SQL is a declarative (non-procedural) programming language. This means while coding you don’t have to focus on how task will be done, but need to focus on what task is to be done.

Using a procedural (imperative) language “HOW”. The examples are C, Java, etc. In this you explicitly need to define the solution of the task.
For example:

• Please, open the door.
• Go outside
• Take the bucket I forgot there.
• Bring it back to me.

On other hand, in declarative(non-procedural) language “WHAT” this would sound like Fetch the bucket, please. We don’t need to go the process step by step. The algorithm is made in and there is a optimizer which divided the steps into smaller tasks to achieve that task.

It is a programming language specially designed for working with databases. It is used to Create, manipulate and share the Data. Especially data from Relational Database management systems. The way the computer extract data from database is by writing Query.

Query is a piece of code intuiting computer to perform certain operations that can deliver the desired output. The entire process is called as the querying the database. SQL allows you to write queries that the computer can execute and then provide database insights in return.

The good thing about SQL is that it is intuitive and easy to learn language. But be careful it does not means using it is going to be a simple task. Sometimes the business problems involves the processing of large amounts of data. Despite it is relatively simple it has power to perform complex tasks.

Introduction to Databases

Databases plays a very important role in SQL. Writing SQL queries become easier if you understand databases well. Each row in database has certain value and each row makes a record. A record is each entry which exist in a table. A field is a column in a table containing specific information about every record in the table. When data are in the form of rows and columns it means you are dealing with stored tabular data. Relational algebra allows us to retrieve data efficiently.

The smallest unit that can contain a meaningful set of data is called entity. The rows represent the horizontal entity and columns represent the vertical entity. The table is itself an entity but it can also be called as database object.

Different types of Database languages

Introduction to Data analytics

Data analytics is the science of extracting trends, patterns, and relevant information from raw data to draw conclusions. It has multiple approaches, multiple dimensions, and diverse techniques. It helps in various scientific decision making and effective business operations. It is used for analyzing data, gaining profits, making better use of resources and improving managerial operations.

Data analytics is the process of examining and analyzing raw data sets to:

• Draw conclusions
• Derive more information
• Improve businesses, products, and services

In addition to making business decisions, it is used by data scientists and researchers to verify scientific models and theories.

The given visual shows the data analytics process flow which we will study in the future articles.

Types of Data analytics

According to the visual shown above we can observe that as we are going further in X-axis the Complexity of various types of analytics is increasing and as we are going above in Y-axis the value of the analytics is also increasing. Also the range of information is increasing as we go further in various types of analytics, starting from Descriptive analytics to that of Prescriptive analytics.

There are four main types of analytics based on the workflow and requirements of data analytics:

• Descriptive analytics
• Diagnostic analytics
• Predictive analytics
• Prescriptive analytics

Descriptive analytics

Descriptive analytics help us to answer the question like what has happened. In descriptive analytics existing data is analyzed to understand what is happened in the past or is happening currently. This analytics is a simplest form of analytics as it deals with data aggregation and mining techniques. The insights gathered by this analytics can be useful for planning various strategies in targeting marketing.

Some of the points of descriptive analytics

Data aggregation is the process of gathering and expressing information in a summarized form. Tools used for data aggregation include MS Excel, MATLAB, SPSS and STATA. Company report is an example of descriptive analytics.

Diagnostic analytics

Diagnostic analytics helps to answer the question about why this things happened. In diagnostic analytics it focuses more on the current events rather than past and to determine which are the factors which are influencing the current trends. In order to explore the data into much deep different techniques like data mining, drill down, data discovery, etc. are used.

Some of the key points of diagnostic analytics are:

They can be used to discover a causal relationship between two or more data sets. Diagnostic analytics is helpful for those concerned with day-to-day operations. For example, it helps identify why a sales representative has sold fewer items than usual.

Predictive analytics

Predictive analytics help answer the question about what will happen in the future. This analytics use the past historical data to another and analyze it and give the insights which can recur in the near future. It uses various statistical models and ML techniques. They can achieve higher level of accuracy. One of the most common example is regression analysis.

Predictive analytics is used in:

• Predicting future outcomes in terms of probablity of an event to occur.
• Analyzing sentiments where all opinions posted on social media are collected to predict a person’s sentiments.
• Identifying target audience for the promotional campaign.
• Forecasting weather, plan-failure prediction, and various recommendation systems.

A predictive model is built on the preliminary descriptive analytics stage.

Prescriptive analytics

The Prescriptive analytics helps answer questions about what is to be done. By the insights gained form predictive analytics company can make different data driven decisions. Through this company can take various decisions through facts and insights gained. Prescriptive analytics mostly depends on the patterns you get in previous analytics.

Predictive analytics is at the budding stage of implementation and companies have not used its full potentials. Advancements in predictive analytics is paving the way for its development.

The above mentioned types of analytics provide the insights that various organizations and businesses need to make effective and data driven decisions. If the analytics techniques are used properly they provide accurate insights according to company’s need and opportunities.

Benefits of Data analytics:

Benefits in Decision making:

• Companies use business analytics to enable faster and facts-based decision making.
• Data-driven organizations make better strategic decisions.
• Companies enjoy high operational efficiency, improved customer satisfaction, robust profit and revenue level.

Data analytics helps you define your target audience based on

• Customer age group
• Customer preferences
• Location-based purchases
• Popular brands or products people seek

Benefits in Cost Reduction:

• Data analytics helps understand shopper behaviors by monitoring their browsing interest.
• Seller identifies shopping patterns and customer demand.
• Customer data helps companies minimize failed campaigns and reduce cost associated with them.
• Data analytics helps in reducing marketing and logistic costs.
• Marketers use technologies to evaluate customer behaviors and make strategic decisions.
• Predictive analytics is used for better performance, higher ROI, and faster success.
• Marketing campaigns use measured activities to plan campaigns.

Let us consider a case study of Amazon to know how it uses various techniques of analytics:

Amazon used data analytics to improve efficiency and reduce cost. Analytics help customer to predict what to buy and anticipate shopping.

Such predictions help increase sales and reduce shipping, inventory, and supply chain costs.

Amazon has more than 200 fulfillment centers worldwide. Supply chain and logistic optimization helps companies reduce costs and improve performance.

Amazon used data analytics for choosing the warehouse closest to the customer and reduces shipping costs by 10-40 percent.

It uses data analytics to attract customers and increase profits by an average of 25 percent annually.

Prices are based on customer activity on a website, competitor’s pricing, and product availability.

Product prices typically change every 10 minutes as data is updated and analyzed.

Amazon typically offers discounts on the best-selling items and earns larger profits on less popular items.

Examples:

Descriptive analytics:

• Spent $20M in different sales training the previous year.

Diagnostic analytics:

• Amazon revenue increased in the West Coast during the past one year
• Increased spending on sales training

Predictive analytics:

• Purchase factor: price, time, weather and festive seasons.
• Predicted 10-12 percent increase in revenue.

Prescriptive analytics:

• Sales training fetched good ROI.
• Implemented a suitable optimization plan to maximize profit.

Core advantages of Data analytics

• Data analytics helps in identifying potential opportunities to streamline operations.
• It identifies potential problems and gives time to take actions.
• It allows companies to identify operations that yield the best results.
• It identifies and improves error-prone operational areas.
• Organizations implement data analytics in product or service development.
• Data analytics helps in understanding current state of business.
• It provides valuable insights to predict future outcomes.
• It helps businesses align new process or products with market needs.
• Data analytics tools are capable of handling heterogenous data and providing insights.

In the next blog we will understand about different types of data and data analytics process. Still than stay tuned.

Happy Learning !! 🔥🔥

Hello folks,

In the previous article we read about different types of statistics coming to more basics let’s discuss about different types of data.

So there are two types of statistics Descriptive and Inferential.

Similarly, Data is divided into two types shown in the visual below.

Types of Data

In the above visual Data is divided as Numerical and Categorical and Numerical is further divides as Discrete and Continuous

Numerical Data:

Numerical data has meaning which has certain measurement. For example person’s height, weight, student’s marks, blood group or they are count as the numbers for example number of properties a person owns.

Discrete data represents items that can be counted, they have certain possible values that can be listed. The values can either be finite or infinite. For example: Number of children in school, number of books in library, etc.

Continuous data represents measurements and they cannot be counted they are measured using range or intervals on the number line. For example under 18 people staying in Mumbai.

Categorical Data:

Categorical Data contains information about a category. Categorical data represents certain characteristics. Lets take an example, gender of person either male or female, marital status either single or married, Yes or No, types of movies, etc.

Also note that categorical values can take on numerical values (such as male is denoted by “1” and female is denoted by “0”) but these numbers does not have any mathematical meaning.

Level of Measurements

Level of measurement also known as scale of measure is types that give information and describes the nature of data with the particular values that are assigned to a variable.

By observing the above visual we have 4 different types of Level of measurement.

1. Nominal
2. Ordinal
3. Interval
4. Ratio

Nominal Data

Nominal Data also knows as Qualitative / Categorical data. It us used for labelling different types of classification and does not have a particular quantitative value of order.

For example: Where do you leave? Options are 1- Village     2- City     3- Town

Similarly, What is your gender? Options are 1- Male    2- Female

Nominal scale of measurement are mostly used in research surveys and Questionnaires where only variable label has certain significance. It  is the most fundamental level of measurement.

There can be two ways through which this type of data can be collected either by open ended question to which answers are coded by a label decided by researcher.

The alternative method is to include MCQ’s in which answers will be labelled.

Ordinal Data

Ordinal Scale is defined as a variable measurement scale used for rank of variables but difference between or distance between each variable does not matter.

They are mostly used to convey non-mathematical ideas satisfaction, degree of main, happiness, etc.

Here rank is important but degree does not matter.

Let’s take an example of race, in a race a person can come 1st, 2nd, 3rd, 4th … and so on here irrespective of distance person who comes first is first.

Another example grading system where we get ranks i.e. rank 1, rank 2, rank 3. If you get highest marks you are going to be the topper. If a person who is second is 5 Marks below then so he/she is going to be 2nd only.

Interval Data

Interval Scale is defined as a variable measurement scale use for rank of variables on basis of difference or distance. Here both rank and distance has importance.

For example, rank based on percentage, week days i.e. there are fixed set of intervals after every 24 hours. Another similar example is class periods here there are fixed set of time for all classes for 30 min each period.

Ratio Data

It is a variable measurement scale that not only produces the order of variable but also makes the difference between the known variable along with information about the value of true zero.

For example: What is your weight in Kg?

Options can be Less than 50 kg, 51 – 70 kg, 71 – 90 kg, 91 – 110 kg or more than 110 kg.