Intro

NNS.boost

NNS.boost is a routine in the NNS R-package for ensemble classification based on the NNS.reg routine. You can read more about the underlying clustering and regression methods here:

Nonparametric Regression Using Clusters http://rdcu.be/tz0J.

xgboost

Abstract:

xgboost is a highly effective and widely used machine learning method. In this paper, we describe a scalable end-to-end tree boosting system called XGBoost, which is used widely by data scientists to achieve state-of-the-art results on many machine learning challenges. We propose a novel sparsity-aware algorithm for sparse data and weighted quantile sketch for approximate tree learning. More importantly, we provide insights on cache access patterns, data compression and sharding to build a scalable tree boosting system. By combining these insights, XGBoost scales beyond billions of examples using far fewer resources than existing systems.

Full paper available here: https://arxiv.org/abs/1603.02754

Install the Latest Version of NNS (>= 0.4.4) and xgboost

library(devtools); install_github('OVVO-Financial/NNS', ref = "NNS-Beta-Version")
require(NNS)
require(plyr)
require(xgboost)
require(methods)
require(pROC)

Example #1: UCI Credit Approval Classification

Basic example and other methods provided here https://rpubs.com/omicsdata/credit.

Step 1: Load Data UCI Credit Screening

dataset = read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/credit-screening/crx.data",sep=",", header=F, na.strings="?")
dataset = dataset[complete.cases(dataset),]
head(dataset)

Step 2: Run NNS.boost and xgboost over 1,000 iterations

accuracy = list()

## convert factor into dummy variable matrix
into_factor = function(x){
     if(class(x) == "factor"){
         n = length(x)
         data.fac = data.frame(x=x, y=1:n)
         output = model.matrix(y~x, data.fac)[,-1]
         
     }
     else{
        output = x
     }
     output
}

### Prepare dataset for xgboost
dataset2 = dataset

dataset2 = colwise(into_factor)(dataset2)
dataset2 = do.call(cbind, dataset2)
dataset2 = as.data.frame(dataset2)


## For parallel processing
cores = detectCores()
cl <- makeCluster(cores[1]-1)
registerDoParallel(cl)


accuracy <- foreach(i = 1:1000,.packages = c("NNS","pROC","xgboost","plyr"))%dopar%{
set.seed(123+i)
n = dim(dataset)[1]
index = sample(n, round(0.7*n))

### NNS.dataset
train = dataset[index,]
test=dataset[-index,]

nns.predictions = NNS.boost(IVs.train = train[,1:15], 
                          DV.train = train[,16],
                          IVs.test= test[,1:15], feature.importance = FALSE,
                          representative.sample = FALSE, 
                          epochs = 100, learner.trials = 100,
                          depth = "max", ncores = 1, subcores = 1)$results

accuracy$nns = auc(roc(as.numeric(test[,16]),round(nns.predictions)))


### xgboost dataset
train.xg = dataset2[index,]
test.xg = dataset2[-index,]
label = as.matrix(train.xg[,38, drop=F])
data = as.matrix(train.xg[,-38, drop=F])
data2 = as.matrix(test.xg[,-38, drop=F])
label2 = as.matrix(test.xg[,38,drop=F])

### Prepare xgboost parameters
xgmat = xgb.DMatrix(data, label=label, missing = -10000)
param = list("object" = "binary:logistic", "best:eta" = 1, "bst:max_depth"=10, "eval_metric" = "logloss", "silent"=1,"nthread"=16,"min_child_weight"=1.45)
nround = 500

### Run xgboost model
bst = xgb.train(param, xgmat, nround)
res1 = predict(bst, data2)

pre1 = ifelse(res1>0.5, 1, 0)

accuracy$xgb = auc(roc(as.numeric(label2), pre1))

return(accuracy)
}

stopCluster(cl)
registerDoSEQ()

Step 3: Compare Accuracy

On average NNS performs just as well or better than xgboost on varous seeds. Instead of using just the standard deviation of results, we are comparing the lower partial moment LPM(), because (like finance) really good estimates to the upside should not be penalized as merely an increase in variance!

nns.accuracy=unlist(lapply(accuracy, `[[`, 1))
xgb.accuracy=unlist(lapply(accuracy, `[[`, 2))

rbind(
  "Mean NNS" = mean(nns.accuracy),
  "Mean XGB" = mean(xgb.accuracy),
  "LPM NNS" = LPM(2, mean(nns.accuracy), nns.accuracy),
  "LPM XGB" = LPM(2, mean(xgb.accuracy), xgb.accuracy))
##                  [,1]
## Mean NNS 0.8608110673
## Mean XGB 0.8558886609
## LPM NNS  0.0002272005
## LPM XGB  0.0002693473
t.test(nns.accuracy,xgb.accuracy)
## 
##  Welch Two Sample t-test
## 
## data:  nns.accuracy and xgb.accuracy
## t = 4.9593, df = 1991.5, p-value = 7.674e-07
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.002975846 0.006868967
## sample estimates:
## mean of x mean of y 
## 0.8608111 0.8558887

Example #2: agaricus Dataset

This time we will utilize the data from within xgboost, the agaricus dataset. Mushrooms described in terms of physical characteristics; classification: poisonous or edible.

### Load [agaricus] dataset
data(agaricus.train, package = 'xgboost')
data(agaricus.test, package = 'xgboost')

### Set up data for xgboost
dtrain = xgb.DMatrix(agaricus.train$data, label = agaricus.train$label)
dtest = xgb.DMatrix(agaricus.test$data, label = agaricus.test$label)


### Create parameters and run xgboost
watchlist = list(train = dtrain, eval = dtest)

param = list(max_depth = 2, eta = 1, silent = 1, nthread = 2, 
              objective = "binary:logistic", eval_metric = "auc")

bst = xgb.train(param, dtrain, nrounds = 2, watchlist)
## [1]  train-auc:0.958228  eval-auc:0.960373 
## [2]  train-auc:0.981413  eval-auc:0.979930
### Set up data for NNS
agaricus.iv.train = as.matrix(agaricus.train$data)
agaricus.dv.train = agaricus.train$label
agaricus.iv.test = as.matrix(agaricus.test$data)
agaricus.dv.test = agaricus.test$label


### Run NNS.boost
a = NNS.boost(IVs.train = agaricus.iv.train, DV.train = agaricus.dv.train, 
            IVs.test = agaricus.iv.test, feature.importance = FALSE, 
            representative.sample = TRUE, epochs = 100, learner.trials = 100,
            depth = "max", ncores = 1, subcores = 1)$results


### Evaluate accuracy  
mean(round(a) == as.numeric(agaricus.dv.test))
## [1] 1
### Evaluate AUC
auc(roc(as.numeric(agaricus.dv.test), round(a)))
## Area under the curve: 1

As we can see, NNS achieves a perfect classification of the substantial (1,611 observations) out-of-sample test set while the xgboost AUC is approximately 98%.

Comments

NNS is a full machine learning methodology, all based on the seamless underlying clustering NNS.part embedded within the regression method NNS.reg.

xgboost is maintained by an army of computer scientists, NNS is maintained by a single economist! Speed overwhelmingly favors xgboost, but with ample resources and collaboration, NNS will be able to compete on that front as well. Additional computational efficiency would permit even more NNS testing and further improvement!

NNS is not a one-trick pony, as it has been demonstrated to excel in time-series forecasting, nonlinear continuous regressions, and provide solutions for econometric applications. See the following examples:

I look forward to further discussions and collaboration with those equally as passionate about these issues, and open to embracing alternative solutions. If you found this presentation interesting or useful, please feel free to reach out via e-mail:

Thanks for your interest!