Discovered in the 1970s by American mathematician John Tukey, exploratory data analysis (EDA) is a method of analysing and investigating the data sets to summarise their main characteristics. Scientists often use data visualisation methods to discover patterns, spot anomalies, check assumptions or test a hypothesis through summary statistics and graphical representations.
EDA goes beyond the formal modelling or hypothesis to give maximum insight into the data set and its structure, and in identifying influential variables. It can also help in selecting the most suitable data analysis technique for a given project. Specific knowledge, such as the creation of a ranked list of relevant factors to be used as guidelines, can also be obtained using EDA.
Types of EDA
The EDA types of techniques are either graphical or quantitative (non-graphical). While the graphical methods involve summarising the data in a diagrammatic or visual way, the quantitative method, on the other hand, involves the calculation of summary statistics. These two types of methods are further divided into univariate and multivariate methods.
Univariate methods consider one variable (data column) at a time, while multivariate methods consider two or more variables at a time to explore relationships. Thus, there are four types of EDA in all — univariate graphical, multivariate graphical, univariate non-graphical, and multivariate non-graphical. The graphical methods provide more subjective analysis, and quantitative methods are more objective.
- Univariate non-graphical: This is the simplest form of data analysis among the four options. In this type of analysis, the data that is being analysed consists of just a single variable. The main purpose of this analysis is to describe the data and to find patterns.
- Univariate graphical: Unlike the non-graphical method, the graphical method provides the full picture of the data. The three main methods of analysis under this type are histogram, stem and leaf plot, and box plots. The histogram represents the total count of cases for a range of values. Along with the data values, the stem and leaf plot shows the shape of the distribution. The box plots graphically depict a summary of minimum, first quartile median, third quartile, and maximum.
- Multivariate non-graphical: The multivariate non-graphical type of EDA generally depicts the relationship between multiple variables of data through cross-tabulation or statistics.
- Multivariate graphical: This type of EDA displays the relationship between two or more set of data. A bar chart, where each group represents a level of one of the variables and each bar within the group represents levels of other variables.
EDA Tools
Python and R language are the two most commonly used data science tools to create an EDA.
Python: EDA can be done using python for identifying the missing value in a data set. Other functions that can be performed are — the description of data, handling outliers, getting insights through the plots. Due to its high-level, built-in data structure, and dynamic typing and binding make it an attractive tool for EDA. Analyzing a dataset is a hectic task that takes a lot of time. Python provides certain open-source modules that can automate the whole process of EDA and help in saving time.
R: The R language is used widely by data scientists and statisticians for developing statistical observations and data analysis. R is an open-source programming language which provides a free software environment for statistical computing and graphics that is supported by the R Foundation for Statistical Computing.
Wrapping Up
Apart from the functions described above, EDA can also:
- Perform k-means clustering. It is an unsupervised learning algorithm where the data points are assigned to clusters, also known as k-groups. K-means clustering is commonly used in market segmentation, image compression, and pattern recognition.
- EDA can be used in predictive models such as linear regression, where it is used to predict outcomes.
- It is also used in univariate, bivariate, and multivariate visualization for summary statistics, establishing relationships between each variable, and for understanding how different fields in the data interact with each other.
If you loved this story, do join our Telegram Community.
Also, you can write for us and be one of the 500+ experts who have contributed stories at AIM. Share your nominations here.
