Data Analysis Techniques

Classification

Classification is a Supervised Learning algorithm where the goal is to predict the category or class labelof given data points. It is used when the output variable is categorical .

How Classification Works?

1. Training Phase:

A model learns the relationship b/w the input features and their corresponding class labels from a labeled dataset.

2. Testing Phase:

The trained model is used to classify new, unseen data points into one of the predefined classes.

Types of Classification:

Binary Classification
Multi class classification
Multi label classification

Examples of Classification Tasks:

Email filtering
Medical diagnosis
Image recognition
sentiment Analysis

Classification Algorithms

Logistic Regression
Decision Trees
Support Vector Machines
K-Nearest Neighbours
Navie bayes
Random Forest

Evalution Metrics for Classification

Accuracy - Percentage of correctly labeled classes
Precision - How many +ve predictions are correct
Recall
F1 Score
ROC-AUC

Workflow of Classification

Data Collection: Gather labeled data.
Data Preprocessing: Clean, normalize, and split the data into training and testing sets.
Feature Selection: Identify the most relevant features.
Model Training: Train a classification model using a suitable algorithm.
Model Evaluation: Test the model on unseen data using evaluation metrics.
Prediction: Use the model for classifying new data.

Decision Tree Algorithm

A Decision Tree is a supervised machine learning algorithm used for classification and regression tasks. It models decisions and their possible consequences as a tree-like structure of nodes, where:

• Internal Nodes: Represent a decision based on a feature.
• Branches: Indicate the outcome of a decision.
• Leaf Nodes: Represent the final outcome (class label or prediction).

The tree splits the data into subsets based on the feature that provides the most information gain or least impurity at each step.

Concepts in Decision Trees

1. Root Node:
   The starting point of the tree, representing the entire dataset.

2. Splitting:
   Dividing data at a node into subsets based on a condition (e.g., feature > value).

3. Leaf Node:
   A terminal node that provides the final prediction.

4. Pruning:
   Reducing the size of the tree to avoid overfitting.

5. Impurity Measures (used for splitting):

   • Gini Impurity: Measures the likelihood of incorrect classification if a random sample is classified based on the distribution. $$Gini=1-\sum _{i=1}^n{p}_i^{\mathrm{2}}$$
   • Entropy: Measures the randomness or impurity of a dataset. $$Entropy=-\sum _{i=1}^n{p}_i{\log }_2(p_i)$$
   • Information Gain: The reduction in entropy or impurity achieved by a split. $$\text{Information Gain} = \text{Entropy(parent)} - \text{Weighted Entropy(children)}$$
     

Advantages of Decision Trees

1. Simple to Understand: Mimics human decision-making.
2. Interpretable: Provides clear visualization of the decision process.
3. Non-Parametric: No assumptions about data distribution.
4. Handles Non-linear Relationships: Can model complex patterns.

Disadvantages of Decision Trees

1. Overfitting: Trees can grow too large and fit the training data perfectly, leading to poor generalization.
2. Bias to Dominant Features: Sensitive to unbalanced datasets.
3. Instability: Small changes in data can result in significantly different trees.

How Decision Trees Work

1. Start at the Root Node: Evaluate all features and split the data based on the feature that provides the highest information gain or lowest Gini impurity.
2. Recursive Splitting: Continue splitting each subset at child nodes until:
   • All data points in a node belong to the same class.
   • A stopping criterion (e.g., maximum depth, minimum samples) is met.
3. Prediction: For a new data point, follow the path from the root node to a leaf node to determine the predicted class.

=======================================================================
Python Code

from sklearn.datasets import load_iris

from sklearn.model_selection import train_test_split

from sklearn.tree import DecisionTreeClassifier, plot_tree

import matplotlib.pyplot as plt

# Load dataset

data = load_iris()

X = data.data  # Features

y = data.target  # Target labels

# Split data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Train a decision tree classifier

model = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=42)

model.fit(X_train, y_train)

# Visualize the decision tree

plt.figure(figsize=(12, 8))

plot_tree(model, feature_names=data.feature_names, class_names=data.target_names, filled=True)

plt.show()

# Evaluate the model

accuracy = model.score(X_test, y_test)

print(f"Accuracy: {accuracy:.2f}")

======================================================================

Bayes theorem

Bayes' Theorem is a fundamental concept in probability theory and statistics that describes the probability of an event, based on prior knowledge of conditions that might be related to the event. In machine learning, it is widely used for classification tasks, particularly in Naive Bayes classifiers.

---

Bayes' Theorem Formula

Bayes' Theorem is expressed mathematically as:

$$P(A\mathrm{\mid }B)=\frac{P(B\mathrm{\mid }A)\cdot P(A)}{P(B)}$$

Where:

• $P(A|B)$ is the posterior probability: The probability of event $A$ occurring given that $B$ has occurred.
• $P(B|A)$ is the likelihood: The probability of event $B$ occurring given that $A$ has occurred.
• $P(A)$ is the prior probability: The probability of event $A$ occurring before observing event $B$.
• $P(B)$ is the evidence or normalizing constant: The total probability of event $B$ occurring (the denominator ensures the total probability sums to 1).

---

Explanation of the Components

1. Prior Probability ($P(A)$): This represents what is known about $A$ before any new data is observed. It’s the initial belief or assumption about a class or event.

2. Likelihood ($P(B|A)$): This is the likelihood of observing the evidence $B$ given that $A$ is true. It quantifies how well the evidence supports the hypothesis.

3. Posterior Probability ($P(A|B)$): This is the probability of $A$ occurring after considering the evidence $B$. It's the updated belief after observing the data.

4. Evidence ($P(B)$): This is the total probability of observing $B$ across all possible outcomes. It is used to normalize the result to ensure the probabilities sum to 1.

---

Naive Bayes Classifier in Machine Learning

In machine learning, the Naive Bayes classifier is based on Bayes' Theorem, with the assumption that the features used for prediction are conditionally independent given the class label. This simplifies the calculation, hence the term "naive."

Steps for Naive Bayes Classification:

1. Compute the Prior Probability:
   The probability of each class occurring in the dataset.

   $$P(C_k) = \frac{\text{Number of instances of class } C_k}{\text{Total number of instances}}$$

2. Compute the Likelihood:
   Calculate the likelihood for each feature given the class label. For continuous features, this often assumes a Gaussian distribution.

   $$P(X_i | C_k) = \text{Likelihood of feature } X_i \text{ given class } C_k$$

3. Compute the Posterior Probability:
   For each class $C_k$, compute the posterior probability of a new data point $X = (X_1, X_2, ..., X_n)$:

   $$P(C_k | X) = \frac{P(C_k) \cdot \prod_{i=1}^{n} P(X_i | C_k)}{P(X)}$$

4. Classify the Data Point:
   Choose the class $C_k$ with the highest posterior probability as the predicted class.

==================================================================

Handling Class Imbalance in Support Vector Machines (SVM)

Class imbalance occurs when the number of samples in one class significantly exceeds those in another class. This can negatively impact the performance of Support Vector Machines (SVM), as the decision boundary may be biased towards the majority class, leading to poor performance on the minority class.

---

Challenges with Class Imbalance in SVM

1. Bias Toward Majority Class: The SVM objective function may focus more on maximizing the margin for the majority class, ignoring the minority class.
2. Skewed Decision Boundary: The decision boundary may shift closer to the minority class, reducing its classification performance.
3. Poor Evaluation Metrics: Standard metrics like accuracy may not reflect the true performance of the model on imbalanced datasets.

---

Techniques to Address Class Imbalance in SVM

1. Class Weight Adjustment:
   Assign higher weights to the minority class to balance the penalty during training.

   • In Scikit-learn, this can be done using the class_weight parameter:

     from sklearn.svm import SVC
     
     # Assign 'balanced' to automatically compute weights
     model = SVC(class_weight='balanced')
     

   • Or manually specify weights:

     model = SVC(class_weight={0: 1, 1: 10})  # Higher weight for class 1 (minority)
     

2. Resampling Techniques:

   • Oversampling: Increase the number of minority class samples by duplicating or generating synthetic samples (e.g., using SMOTE).
   • Undersampling: Reduce the number of majority class samples to match the minority class.

   Example using SMOTE:

   from imblearn.over_sampling import SMOTE
   from sklearn.svm import SVC
   from sklearn.model_selection import train_test_split
   
   # Create synthetic samples for the minority class
   smote = SMOTE(random_state=42)
   X_resampled, y_resampled = smote.fit_resample(X, y)
   
   # Train an SVM on the balanced dataset
   model = SVC()
   model.fit(X_resampled, y_resampled)
   

3. Change the Decision Threshold: Adjust the threshold for classifying a sample as a minority class based on the predicted probabilities.

   Example:

   from sklearn.svm import SVC
   from sklearn.metrics import classification_report
   
   # Fit SVM
   model = SVC(probability=True)
   model.fit(X_train, y_train)
   
   # Predict probabilities
   y_prob = model.predict_proba(X_test)[:, 1]
   
   # Adjust threshold
   threshold = 0.3
   y_pred = (y_prob >= threshold).astype(int)
   
   # Evaluate performance
   print(classification_report(y_test, y_pred))
   

4. Kernel Selection: Use kernels that better separate the minority and majority classes, such as the RBF kernel, which can handle non-linear boundaries.

5. Feature Engineering: Transform or create features that emphasize differences between the classes. This can improve separability in the feature space.

6. Use of Alternative Metrics: Evaluate the model using metrics like:

   • Precision, Recall, F1 Score
   • Area Under the Receiver Operating Characteristic Curve (AUC-ROC)
   • Precision-Recall Curve

========================================================================

Random Forest is a popular ensemble learning technique used for both classification and regression tasks. It combines the predictions of multiple decision trees to improve accuracy, reduce overfitting, and enhance generalization.

---

Key Features of Random Forest

1. Ensemble Method: It creates multiple decision trees during training and combines their results for better performance.
2. Bagging: It employs bootstrapping (sampling with replacement) to create different subsets of the training data for each tree.
3. Random Feature Selection: At each split in a tree, only a random subset of features is considered, making the model less prone to overfitting.
4. Majority Voting (Classification): For classification tasks, it uses the majority vote of the trees as the final prediction.
5. Averaging (Regression): For regression tasks, it uses the average of the predictions from all trees.

---

How Random Forest Works

1. Bootstrap Aggregation (Bagging):

   • Randomly sample the dataset with replacement to create multiple subsets of the data (one for each tree).
   • Train each decision tree on a different subset.

2. Feature Randomness:

   • At each split in a decision tree, a random subset of features is considered instead of evaluating all features.
   • This adds diversity to the trees and prevents overfitting.

3. Prediction Aggregation:

   • For classification: Take the majority vote across all decision trees.
   • For regression: Compute the average of the predictions.

---

Advantages of Random Forest

1. Robust to Overfitting: Combines multiple trees, reducing the risk of overfitting compared to individual decision trees.
2. Handles Non-linear Data: Captures complex relationships in data.
3. Works Well with Missing Values: Can handle datasets with missing values by averaging predictions.
4. Scales Well to Large Datasets: Performs efficiently with high-dimensional data.
5. Feature Importance: Provides insights into feature significance, aiding interpretability.

---

Disadvantages of Random Forest

1. Computationally Intensive: Training multiple decision trees can be slow, especially with large datasets.
2. Not as Interpretable: While decision trees are interpretable, combining them into a forest makes the model harder to understand.
3. Bias in Small Data: May struggle with small datasets if not tuned properly.
4. Memory Usage: Requires more memory as it stores multiple trees.

---

Random Forest Algorithm

1. Input: Dataset $D$ with $n$ samples, number of trees $T$, and number of features $m$ to select at each split.
2. For Each Tree:
   • Draw a bootstrap sample from $D$.
   • Build a decision tree on the bootstrap sample.
   • At each split, randomly select $m$ features and choose the best split among them.
3. Aggregate Predictions:
• For classification, use majority voting.
• For regression, compute the average

=======================================================================