Exploratory data analysis (EDA) is used to analyze and investigate data sets and summarize their main characteristics numerically as well as visually.

The primary aim of exploratory data analysis is to examine the data for distribution, outliers and anomalies to direct specific testing of your hypothesis. It helps look at the data before making any assumptions. It can help identify obvious errors, as well as better understand patterns within the data and find interesting relations among the variables.

10 Quick Steps in Data Exploration and Preprocessing

Identification of variables and data types: This step helps in understanding whether the variable is numeric or categorical. Within Numeric we can check if it’s Discrete (results from counting for example – number of employees in a line of business) or Continuous (the number which can take any value such as the  daily expenses of a household). Within Categorical we can check if the variable is Ordinal (if the categories can be ordered logically for example – rating from customer on product satisfaction) or Nominal (this has levels that cannot be ordered such as Gender-Male, Female).

Analyzing the basic metrics: This includes understanding the data and what are the possible measurement strategies that could be employed to measure those variables.

Non-Graphical Univariate Analysis: This process numerically explores the data by looking at summary statistics of each variable. Summary statistics provide various measures such as minimum, maximum, mean, 25 percentile score, 50th percentile score, 75th percentile score, count, etc.

Graphical Univariate Analysis: This includes a detailed study of each variable used in the analysis. The variables are graphically explored using histogram, box plot, etc. to understand the data distribution.

Bivariate Analysis: This includes taking 2 variables at a time and assessing their correlation. It also includes mapping each independent variable against the dependent variable to see if it can influence the dependent variable significantly.

Variable transformations: Transformation is a mathematical operation that changes the measurement scale of a variable. This is usually done to make the dataset useable with a particular statistical test or method. Many statistical methods require data that follow a particular kind of distribution, usually a normal distribution.

Missing value treatment: Missing data in your data set can reduce the power/fit of a model or can lead to a biased model. It can lead to wrong prediction or classification. Missing values can be imputed using various methods or algorithms. Basic imputation includes replacing missing values with mean, median, or mode depending upon the data type and distribution.

Outlier treatment: Outliers are extremely low/high values in your data set. It usually is calculated using a box plot and values outside the range of < Q1-1.5*IQR or >Q3+ 1.5*IQR are considered as outliers commonly. Having said this, outliers are very sensitive and must be carefully excluded, included, or imputed. This becomes an easy task if you have strong domain knowledge and know the metrics used for analysis.

Correlation Analysis: Correlation analysis is used to quantify the degree to which two variables are related. Through the correlation analysis, you evaluate the correlation coefficient that tells you how much one variable chchangeshen the other one does. Correlation analysis provides you with a linear relationship between two variables.

Dimensionality Reduction: Dimensionality reduction is the transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties of the original data, ideally close to its intrinsic dimension. There are many ways to reduce the dimension. One most popular way is by application of Principal Component Analysis (PCA).

Contact Information

Website :> https://datapatrons.com/

Contact No :> +91-8800196438

E- Mail :> support@datapatrons.com

Address :> Gautam Buddha Nagar, Uttar Pradesh – 201309