Bagging in overcoming variance of a classifier, clustering algorithm or regressor

Bootstrap aggregating

Bootstrap aggregating, also called bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach. 

Bagging: Sampling with replacement

Build classifier on each bootstrap sample
Each sample has probability (1 - (1 - 1/n) ^ n) of being selected in training set.

Sampling data for bagging



Probability that a sample will be selected in m rounds of bagging



Probability that ensemble will make an error with majority voting decision making technique

Tags: Technology,Machine Learning,Classification,Clustering,Regression,

Linear Regression Using Java Code And Weka JAR

File: Regression.java

import weka.core.Instance;
import weka.core.Instances;
import weka.core.converters.ConverterUtils.DataSource;
import weka.classifiers.functions.LinearRegression;

public class Regression{
    public static void main(String args[]) throws Exception{
        //Load Data set
        DataSource source = new DataSource("/home/ashish/Desktop/ws/weka/e4_linear_regression_using_java_code/house.arff");
        Instances dataset = source.getDataSet();
        //set class index to the last attribute
        dataset.setClassIndex(dataset.numAttributes()-1);
        
        //Build model
        LinearRegression model = new LinearRegression();
        model.buildClassifier(dataset);
        //output model
        System.out.println("LR FORMULA : "+model);	
        
        // Now Predicting the cost 
        Instance myHouse = dataset.lastInstance();
        double price = model.classifyInstance(myHouse);
        System.out.println("-------------------------");	
        System.out.println("PRECTING THE PRICE : "+price);	
    }
}

File: house.arff

@RELATION house

@ATTRIBUTE houseSize NUMERIC
@ATTRIBUTE lotSize NUMERIC
@ATTRIBUTE bedrooms NUMERIC
@ATTRIBUTE granite NUMERIC
@ATTRIBUTE bathroom NUMERIC
@ATTRIBUTE sellingPrice NUMERIC

@DATA
3529,9191,6,0,0,205000 
3247,10061,5,1,1,224900 
4032,10150,5,0,1,197900 
2397,14156,4,1,0,189900 
2200,9600,4,0,1,195000 
3536,19994,6,1,1,325000 
2983,9365,5,0,1,230000

File: Execution.log

~/Desktop/ws/weka/e4_linear_regression_using_java_code$ javac -cp ./weka-3.7.0.jar Regression.java
~/Desktop/ws/weka/e4_linear_regression_using_java_code$ 
~/Desktop/ws/weka/e4_linear_regression_using_java_code$ ls -l
total 5232
-rw-rw-r-- 1 ashish ashish     365 Mar  5 09:05 house.arff
drwxrwxr-x 2 ashish ashish    4096 Mar  5 09:12 jar
-rw-rw-r-- 1 ashish ashish    1714 Mar  5 09:24 Regression.class
-rw-rw-r-- 1 ashish ashish     924 Mar  5 09:18 Regression.java
-rw-rw-r-- 1 ashish ashish 5340945 Sep 27  2011 weka-3.7.0.jar
~/Desktop/ws/weka/e4_linear_regression_using_java_code$ java -cp .:./weka-3.7.0.jar Regression
LR FORMULA : 
Linear Regression Model

sellingPrice =

    -26.6882 * houseSize +
      7.0551 * lotSize +
  43166.0767 * bedrooms +
  42292.0901 * bathroom +
 -21661.1208
-------------------------
PRECTING THE PRICE : 222921.57101904938
~/Desktop/ws/weka/e4_linear_regression_using_java_code$ 




(base) ashish@ashish-VirtualBox:~/Desktop/ws/weka/e4_linear_regression_using_java_code$ ls -l
total 13852
-rw-rw-r-- 1 ashish ashish      923 Mar  5 09:25 execution.log
-rw-rw-r-- 1 ashish ashish      365 Mar  5 09:05 house.arff
drwxrwxr-x 2 ashish ashish     4096 Mar  6 15:04 jar
-rw-rw-r-- 1 ashish ashish     1714 Mar  5 09:24 Regression.class
-rw-rw-r-- 1 ashish ashish      924 Mar  5 09:18 Regression.java
-rwxrwxrwx 1 ashish ashish 14163929 Jan 25 16:06 weka-3.8.6.jar
(base) ashish@ashish-VirtualBox:~/Desktop/ws/weka/e4_linear_regression_using_java_code$ java -cp .:./weka-3.8.6.jar Regression
Exception in thread "main" java.lang.NoClassDefFoundError: no/uib/cipr/matrix/Matrix
	at Regression.main(Regression.java:15)
Caused by: java.lang.ClassNotFoundException: no.uib.cipr.matrix.Matrix
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 1 more
(base) ashish@ashish-VirtualBox:~/Desktop/ws/weka/e4_linear_regression_using_java_code$ javac -cp .:./weka-3.8.6.jar Regression.java
(base) ashish@ashish-VirtualBox:~/Desktop/ws/weka/e4_linear_regression_using_java_code$ java -cp .:./weka-3.8.6.jar Regression
Exception in thread "main" java.lang.NoClassDefFoundError: no/uib/cipr/matrix/Matrix
	at Regression.main(Regression.java:15)
Caused by: java.lang.ClassNotFoundException: no.uib.cipr.matrix.Matrix
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 1 more

- - - 

TRYING AGAIN WITH WEKA-3.7.0.JAR:

(base) ashish@ashish-VirtualBox:~/Desktop/ws/weka/e4_linear_regression_using_java_code$ java -cp .:./weka-3.7.0.jar Regression
LR FORMULA : 
Linear Regression Model

sellingPrice =

    -26.6882 * houseSize +
      7.0551 * lotSize +
  43166.0767 * bedrooms +
  42292.0901 * bathroom +
 -21661.1208
-------------------------
PRECTING THE PRICE : 222921.57101904938

Tags: Technology,Machine Learning,Regression