In this guide, we will review popular machine learning techniques with a brief overview of their applications, complexity levels, and some historical context. This list provides a starting point for understanding core machine learning methods and their practical uses.
1. Linear Regression
- Overview: A fundamental statistical method used to predict a dependent variable based on an independent variable. Commonly applied in forecasting and trend analysis.
- Complexity: ⭐
- Inventor: Sir Francis Galton (DOB: 1822-1911) and Karl Pearson (DOB: 1857-1936)
- Key Publication:
2. Logistic Regression
- Overview: Primarily used for binary classification, logistic regression estimates the probability of an outcome and is used widely in classification tasks like spam detection.
- Complexity: ⭐⭐
- Inventor: David Cox (DOB: 1924-2022)
- Key Publication: Cox, D.R. (1958). “The Regression Analysis of Binary Sequences.”
3. Decision Trees
- Overview: A tree-like structure for decision-making and predictive modeling. Suitable for classification and regression problems.
- Complexity: ⭐⭐
- Inventor: Ross Quinlan (DOB: 1944)
- Key Publication: Quinlan, J.R. (1986). “Induction of Decision Trees.”
4. Random Forests
- Overview: An ensemble method that builds multiple decision trees and combines them to improve accuracy. Known for its robustness and ease of use.
- Complexity: ⭐⭐⭐
- Inventor: Leo Breiman (DOB: 1928-2005)
- Key Publication: Breiman, L. (2001). “Random Forests.”
5. Support Vector Machines (SVM)
- Overview: An effective classification technique that finds the hyperplane maximizing the margin between data classes. Widely used in image and text classification.
- Complexity: ⭐⭐⭐
- Inventor: Vladimir Vapnik (DOB: 1936)
- Key Publication: Book: Vapnik, V. (1995). “The Nature of Statistical Learning Theory.”
6. K-Nearest Neighbors (KNN)
- Overview: A simple, non-parametric method used for classification and regression. It predicts outcomes based on the ‘k’ nearest data points.
- Complexity: ⭐⭐
- Inventor: Evelyn Fix (DOB: 1904-1987) and Joseph Hodges (DOB: 1922-2000)
- Key Publication:
- Paper:
- DOI: 10.2307/1403797
- Fix, E., & Hodges, J.L. (1951). “Discriminatory Analysis.”
- Book: Fix, E., & Hodges, J.L. (1951). “Discriminatory Analysis.”
- Paper:
7. Naive Bayes
- Overview: Based on Bayes’ theorem, this technique is widely used for text classification due to its simplicity and effectiveness with large datasets.
- Complexity: ⭐
- Inventor: Thomas Bayes (DOB: 1701-1761)
- Key Publication:
8. Gradient Boosting
- Overview: An ensemble technique that combines weak learners to minimize errors iteratively. Known for its high accuracy in structured data tasks.
- Complexity: ⭐⭐⭐⭐
- Inventor: Jerome Friedman (DOB: 1939)
- Key Publication:
- DOI: 10.2307/2699986
- Paper: Friedman, J.H. (2001). “Greedy Function Approximation: A Gradient Boosting Machine.”
9. Neural Networks
- Overview: Inspired by biological neural networks, these algorithms are used in complex tasks like image recognition and language processing.
- Complexity: ⭐⭐⭐⭐⭐
- Inventor: Frank Rosenblatt (DOB: 1928-1971)
- Key Publication:
10. K-Means Clustering
- Overview: An unsupervised algorithm that partitions data into ‘k’ clusters based on similarity. Commonly used for market segmentation and image compression.
- Complexity: ⭐⭐
- Inventor: Stuart Lloyd (DOB: 1933-2006)
- Key Publication:
- Paper:
- DOI:10.1109/TIT.1982.1056489
- Lloyd, S.P. (1982). “Least Squares Quantization in PCM.”
- Paper:
11. Principal Component Analysis (PCA)
- Overview: A dimensionality reduction technique that transforms data into a set of orthogonal components, enhancing interpretability without sacrificing information.
- Complexity: ⭐⭐
- Inventor: Karl Pearson (DOB: 1857-1936)
- Key Publication:
- Paper:
- DOI:10.1080/14786440109462720
- Pearson, K. (1901). “On Lines and Planes of Closest Fit to Systems of Points.”
- Paper:
12. Reinforcement Learning (RL)
- Overview: An area focused on training models to make sequences of decisions, particularly for game playing and robotics.
- Complexity: ⭐⭐⭐⭐⭐
- Inventor: Richard Sutton (DOB: 1952)
- Key Publication:
- Books in Brief:
- DOI:10.1109/TNN.1998.712192
- Sutton, R., & Barto, A. (1998). “Reinforcement Learning: An Introduction.”
- semanticscholar link
- Book: Sutton, R., & Barto, A. (1998). “Reinforcement Learning: An Introduction.” - online_1
- Book: Sutton, R., & Barto, A. (1998). “Reinforcement Learning: An Introduction.” - online_2
- Books in Brief:
These techniques represent a wide spectrum of machine learning methods used across industries. Whether for predictive modeling, classification, clustering, or decision-making, understanding these core methods is foundational to exploring more advanced machine learning concepts.
Below is other view on ML models
| Model Name | Year of Creation | Inventor(s) | Key Publication | DOI |
|---|---|---|---|---|
| Linear Discriminant Analysis (LDA) | Early 1900s | Ronald Fisher | The Use of Multiple Measurements in Taxonomic Problems.1936 Read on sci-hub.se | 10.1111/j.1469-1809.1936.tb02137.x |
| Support Vector Machine (SVM) | 1960s | Vladimir Vapnik | The Nature of Statistical Learning Theory. Read on sci-hub.se | 10.1007/978-1-4757-3264-1 |
| Kernel SVM | 1990s | Bernhard Schölkopf, Alexander Smola | Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond 1 | 10.1007/978-0-387-31471-7 |
| Naive Bayes | 1763s | Thomas Bayes | An Essay towards solving a Problem in the Doctrine of Chances. 1763 Read on sci-hub.se Read on bayes.wustl.edu | 10.1098/rstl.1763.0053 |
| Naive Bayes | Early 1900s | Pierre-Simon Laplace | Théorie Analytique des Probabilités | - |
| Logistic Regression | Early 1900s | Various Researchers | Statistical Methods for Research Workers. Read on sci-hub.se | 10.1016/b978-044450871-3/50148-0 |
| Decision Tree | 1960s | J. Ross Quinlan | Induction of decision tree.1s986 Read on sci-hub.se | 10.1023/A:1022643204877 |
| Random Forest | 1990s | Leo Breiman | Random Forests | 10.1023/A:1010933404324 |
| Gradient Boosting Machine (GBM) | 1990s | Leo Breiman | Friedman, J. H. (1999). Greedy Function Approximation: A Gradient Boosting Machine. 2001 Read on sci-hub or Read on jerryfriedman.su | 10.1214/aos/1013203451 |
| Gaussian Mixture Model (GMM) | Early 1900s | Karl Pearson | Contributions to the Mathematical Theory of Evolution.1896 Read on sci-fi.se or Read on quantresearch.org | 10.1098/rsta.1896.0007 |
| Hidden Markov Model (HMM) | 1960s | Leonard E. Baum, Lloyd R. Rabiner | Statistical Methods for Speech Recognition.2006 Read on sci-hub.se | 10.1016/b0-08-044854-2/00907-x |