Support vectors are data points that maximize the margin around a hyperplane that separates positive and negative instances in dataset.
Basic idea:
Another method for building a classifier where we view the data “spatially”, and predict a new instance’s class based on where it is “located in space”
For simplicity, we’ll assume decision is binary (positive/negative, yes/no, etc.)
Assume numeric attributes (easier to work with as “points in space” and perform arithmetic calculations); however, not restricted to 2D
Hyperplane: plane (or line if 2D space) that separates positive and negative instances in dataset
Support vectors: data points (actual instances from training dataset) that maximize the margin around the hyperplane; they “anchor” the hyperplane
Linear separability
One equation for defining a line is ax + by = c
Find a, b, c such that ax + by ≥ c for red points and ax + by ≤ c for green points In general, many possible solutions for a, b, c
Defining the hyperplane (assuming you knew the support vectors)
Ex: In the dataset below, assume decision attribute is z
Easy to see that support vectors are s1 = (1 0), s2 = (3 1), and s3 = (3 -1)
Augment vectors with 1 more dimension; assign value of 1 for this dimension in each support vector s1 = (1 0 1), s2 = (3 1 1), s3 = (3 -1 1)
Set up equations we need to solve:
φ is a function that maps instance data to a “feature space”
α‘s are parameters which, when “combined” with support vectors and respective decisions, will help us define the hyperplane
α1φ(s1) • φ(s1) + α2φ(s2) • φ(s1) + α3φ(s3) • φ(s1) = -1 s1’s decision
α1φ(s1) • φ(s2) + α2φ(s2) • φ(s2) + α3φ(s3) • φ(s2) = 1 s2’s decision
α1φ(s1) • φ(s3) + α2φ(s2) • φ(s3) + α3φ(s3) • φ(s3) = 1 s3’s decision
Let φ( ) be I (identity vector):
Since data in this example are linearly separable, we’re going to be able to use a linear SVM; so we’re just going to use identify function for φ
α1s1 • s1 + α2s2 • s1 + α3s3 • s1 = -1
α1s1 • s2 + α2s2 • s2 + α3s3 • s2 = 1
α1s1 • s3 + α2s2 • s3 + α3s3 • s3 = 1
Compute dot product in each equation:
2α1 + 4α2 + 4α3 = -1 e.g., s2 • s1 = (3 1 1) • (1 0 1) = (3*1) + (1*0) + (1*1) = 4
4α1 + 11α2 + 9α3 = 1
4α1 + 9α2 + 11α3 = 1
Solve for α1, α2, and α3:
α1 = -3.5, α2 = 0.75, α3 = 0.75
Define the discriminating hyperplane:
w’ = α1s1 + α2s2 + α3s3
= -3.5 (1 0 1) + 0.75 (3 1 1) + 0.75 (3 -1 1)
= (1 0 -2)
For hyperplane in 2D, (a b c) translates to ax + by + c = 0
For hyperplane in 3D, (a b c d) translates to ax + by + cz + d = 0 etc.
So in this case (1 0 -2) translates to 1x + 0y + -2 = 0 or simply x = 2
Doing Linear SVM in Python
Import Libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVC Read Data
df = pd.read_csv('svm_exampleData.csv')Change non numeric value into numeric value
# Just for graphing purposes, change non-decision attr's to have numeric values
df= df.replace({'z': r'positive'}, {'z':1}, regex=True)
df= df.replace({'z': r'negative'}, {'z':-1}, regex=True)X = df.iloc[:, 0:2].values
y = df.iloc[:, 2].values
r,_ = df.shape # Display original data points
plt.scatter(X[:, 0], X[:, 1], c=y, s=r, cmap='winter');
plt.show() Make Linear SVM
# Make a linear SVM
model = SVC(kernel='linear')
model.fit(X, y)
print(model.support_vectors_) # The support vectors
print(model.coef_) # The weights (x and y)
print(model.intercept_) # The intercept # Function to plot SVM boundary lines - cool!
def plot_svc_decision_function(model, ax=None, plot_support=True):
if ax is None:
ax = plt.gca()
xlim = ax.get_xlim()
ylim = ax.get_ylim()
# Create grid to evaluate model
x = np.linspace(xlim[0], xlim[1], 30)
y = np.linspace(ylim[0], ylim[1], 30)
Y, X = np.meshgrid(y, x)
xy = np.vstack([X.ravel(), Y.ravel()]).T
P = model.decision_function(xy).reshape(X.shape)
# Plot decision boundary and margins
ax.contour(X, Y, P, colors='k',
levels=[-1, 0, 1], alpha=0.5,
linestyles=['--', '-', '--'])
# Plot support vectors
if plot_support:
ax.scatter(model.support_vectors_[:, 0],
model.support_vectors_[:, 1],
s=300, linewidth=1, facecolors='none');
ax.set_xlim(xlim)
ax.set_ylim(ylim)
# Call the function to show points and SV boundaries
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='winter')
plot_svc_decision_function(model);
plt.show()
# Make predictions (will be array([-1]) or array([1]))
model.predict([[-3, 4]])
model.predict([[7, 5]])
Thanks for your support!
You can send your request directly at realcode4you@gmail.com if you need any SVMs related help, machine learning related help, data science assignment help, python assignment help or any project related to machine learning and get instant help with an affordable prices.
Comments