1 Introduction

The dataset (Boston Housing Price) was taken from the StatLib library which is maintained at Carnegie Mellon University and is freely available for download from the UCI Machine Learning Repository. The dataset consists of 506 observations of 14 attributes. The median value of house price in $1000s, denoted by MEDV, is the outcome or the dependent variable in our model. Below is a brief description of each feature and the outcome in our dataset:1

  1. CRIM – per capita crime rate by town
  2. ZN – proportion of residential land zoned for lots over 25,000 sq.ft
  3. INDUS – proportion of non-retail business acres per town
  4. CHAS – Charles River dummy variable (1 if tract bounds river; else 0)
  5. NOX – nitric oxides concentration (parts per 10 million)
  6. RM – average number of rooms per dwelling
  7. AGE – proportion of owner-occupied units built prior to 1940
  8. DIS – weighted distances to five Boston employment centres
  9. RAD – index of accessibility to radial highways
  10. TAX – full-value property-tax rate per $10,000
  11. PTRATIO – pupil-teacher ratio by town
  12. B – 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
  13. LSTAT – % lower status of the population
  14. MEDV – Median value of owner-occupied homes in $1000’s
#require(devtools); install_github('OVVO-Financial/NNS', ref = "NNS-Beta-Version")

library(NNS)
library(data.table)
library(rgl)
library(mlbench)


data("BostonHousing")
str(BostonHousing)
## 'data.frame':    506 obs. of  14 variables:
##  $ crim   : num  0.00632 0.02731 0.02729 0.03237 0.06905 ...
##  $ zn     : num  18 0 0 0 0 0 12.5 12.5 12.5 12.5 ...
##  $ indus  : num  2.31 7.07 7.07 2.18 2.18 2.18 7.87 7.87 7.87 7.87 ...
##  $ chas   : Factor w/ 2 levels "0","1": 1 1 1 1 1 1 1 1 1 1 ...
##  $ nox    : num  0.538 0.469 0.469 0.458 0.458 0.458 0.524 0.524 0.524 0.524 ...
##  $ rm     : num  6.58 6.42 7.18 7 7.15 ...
##  $ age    : num  65.2 78.9 61.1 45.8 54.2 58.7 66.6 96.1 100 85.9 ...
##  $ dis    : num  4.09 4.97 4.97 6.06 6.06 ...
##  $ rad    : num  1 2 2 3 3 3 5 5 5 5 ...
##  $ tax    : num  296 242 242 222 222 222 311 311 311 311 ...
##  $ ptratio: num  15.3 17.8 17.8 18.7 18.7 18.7 15.2 15.2 15.2 15.2 ...
##  $ b      : num  397 397 393 395 397 ...
##  $ lstat  : num  4.98 9.14 4.03 2.94 5.33 ...
##  $ medv   : num  24 21.6 34.7 33.4 36.2 28.7 22.9 27.1 16.5 18.9 ...

2 NNS.stack

NNS.reg is the nonparametic regression technique most of NNS is built on.

NNS.stack is a wrapper function that tests the number of clusters n.best parameter in NNS.reg, and the correlation / dependence threshold in the NNS.reg dimension reduction regression.

It is suitable for both continuous and categorical independent / dependent variables. No data preprocessing or transformations are required.

library(caret)
set.seed(12345)


#Do data partitioning
inTrain <- createDataPartition(y = BostonHousing$medv, p = 0.70, list = FALSE)

training <- BostonHousing[inTrain,]
testing <- BostonHousing[-inTrain,]


nns_reg_estimates <- NNS.stack(training[, -14], training[, 14], IVs.test = testing[, -14],
                               status = TRUE,
                               obj.fn = expression( sqrt(mean((predicted - actual)^2))),
                               objective = 'min')


# RMSE of Optimum NNS regression
sqrt(mean((nns_reg_estimates$reg-testing[,14])^2))
## [1] 4.138821
# RMSE of Optimum NNS dimension reduction regression
sqrt(mean((nns_reg_estimates$dim.red-testing[,14])^2))
## [1] 3.661433
# RMSE of Combined Methods
sqrt(mean(abs(testing[,14]-nns_reg_estimates$stack)^2))
## [1] 2.955937
plot(nns_reg_estimates$stack, testing[,14])

# Actual Values
lines(testing[,14],testing[,14],col='red')

3 Random Forest

Using the random forest method, we can compare results.

library(randomForest)
set.seed(12345)
fit.rf <- randomForest(formula = medv ~ ., data = training)

pred.rf <- predict(fit.rf, testing)

sqrt(mean((pred.rf - testing$medv)^2))
## [1] 2.371259
plot(nns_reg_estimates$stack, testing[,14], pch = 19)
lines(testing[,14],testing[,14],col='red')
points(pred.rf, testing[,14],col='red')

4 Comments

The point of this single seed example2 is to further highlight the robust flexibility of the NNS multivariate regression. Whether the goal is classification or inter / extrapolation of continuous variables, NNS.reg is capable of the task.

One distinct advantage of NNS over tree based methods is the ability to seamlessly extrapolate beyond the current range of observations.

NNS.reg is not a black-box ML solution, rather, NNS provides a theoretically sound solution to dynamically partitioning regressors, and finding conditional outputs from those partitions.

NNS is not a one-trick pony, as it has been demonstrated to excel in time-series forecasting, nonlinear continuous regressions, and provide solutions for econometric applications. See the following examples:

I look forward to further discussions and collaboration with those equally as passionate about these issues, and open to embracing alternative solutions. If you found this presentation interesting or useful, please feel free to reach out via e-mail:

Thanks for your interest!

  1. https://rpubs.com/chocka314/251613↩︎

  2. Numerous seeds are required for a more robust experiment and efficiency conclusions like presented in the NNS vs xgboost example.↩︎