Introduction

Marcos Lopez de Prado recently shared a presentation titled, The 7 Reasons Most Econometric Investments Fail. It is a wholesale identification on the misuse of traditional statistical methods in quantitative finance.

Please download Marcos’ report here https://ssrn.com/abstract=3373116, or watch a video on the presentation here: https://www.youtube.com/watch?v=BRUlSm4gdQ4.

This quick note is not an argument against any of the points Marcos raises, rather to illustrate how NNS and my research with David Nawrocki and Hrishikesh Vinod has been addressing these issues for the last several years. I will use his examples with each of his pitfalls and show how NNS provides superior insight to traditional methods.

Load Required Packages in R `NNS (>= 11.6)`

#require(devtools); install_github('OVVO-Financial/NNS', ref = "NNS-Beta-Version")
require(NNS)
require(plyr)
require(data.table)

Pitfall #1: Structured Data

NNS does not utilize standard transformations in order to achieve stationarity.
NNS is based on a hierarchical and partitional clustering method that is able to preserve the original covariance matrix.
Unstructured, categorical, and other non-numeric data is seamlessly encoded in NNS for analysis.

`NNS` Clustering

Below is an example of the partitioning method based on partial moment quadrants. Each observation is sequentially labelled per the quadrant it resides in. A useful analogy is that of an observation being a leaf and sequentially labeling the leaf from the trunk of the tree through the successively smaller branches that leaf is located on.

The (lengthy) output shows the order of partitioning, the observations and their associated quadrant label.

set.seed(123);x=rnorm(100);y=rnorm(100)
NNS.part(x,y,Voronoi=TRUE)

## $order
## [1] 2
## 
## $dt
##                 x           y quadrant prior.quadrant
##   1: -0.560475647 -0.71040656      q43             q4
##   2: -0.230177489  0.25688371      q23             q2
##   3:  1.558708314 -0.24669188      q31             q3
##   4:  0.070508391 -0.34754260      q32             q3
##   5:  0.129287735 -0.95161857      q34             q3
##   6:  1.715064987 -0.04502772      q13             q1
##   7:  0.460916206 -0.78490447      q34             q3
##   8: -1.265061235 -1.66794194      q44             q4
##   9: -0.686852852 -0.38022652      q41             q4
##  10: -0.445661970  0.91899661      q21             q2
##  11:  1.224081797 -0.57534696      q31             q3
##  12:  0.359813827  0.60796432      q12             q1
##  13:  0.400771451 -1.61788271      q34             q3
##  14:  0.110682716 -0.05556197      q14             q1
##  15: -0.555841135  0.51940720      q23             q2
##  16:  1.786913137  0.30115336      q13             q1
##  17:  0.497850478  0.10567619      q14             q1
##  18: -1.966617157 -0.64070601      q42             q4
##  19:  0.701355902 -0.84970435      q33             q3
##  20: -0.472791408 -1.02412879      q43             q4
##  21: -1.067823706  0.11764660      q24             q2
##  22: -0.217974915 -0.94747461      q43             q4
##  23: -1.026004448 -0.49055744      q42             q4
##  24: -0.728891229 -0.25609219      q42             q4
##  25: -0.625039268  1.84386201      q22             q2
##  26: -1.686693311 -0.65194990      q42             q4
##  27:  0.837787044  0.23538657      q13             q1
##  28:  0.153373118  0.07796085      q14             q1
##  29: -1.138136937 -0.96185663      q44             q4
##  30:  1.253814921 -0.07130809      q13             q1
##  31:  0.426464221  1.44455086      q12             q1
##  32: -0.295071483  0.45150405      q23             q2
##  33:  0.895125661  0.04123292      q13             q1
##  34:  0.878133488 -0.42249683      q31             q3
##  35:  0.821581082 -2.05324722      q33             q3
##  36:  0.688640254  1.13133721      q11             q1
##  37:  0.553917654 -1.46064007      q34             q3
##  38: -0.061911711  0.73994751      q21             q2
##  39: -0.305962664  1.90910357      q21             q2
##  40: -0.380471001 -1.44389316      q43             q4
##  41: -0.694706979  0.70178434      q22             q2
##  42: -0.207917278 -0.26219749      q41             q4
##  43: -1.265396352 -1.57214416      q44             q4
##  44:  2.168955965 -1.51466765      q33             q3
##  45:  1.207961998 -1.60153617      q33             q3
##  46: -1.123108583 -0.53090652      q42             q4
##  47: -0.402884835 -1.46175558      q43             q4
##  48: -0.466655354  0.68791677      q21             q2
##  49:  0.779965118  2.10010894      q11             q1
##  50: -0.083369066 -1.28703048      q43             q4
##  51:  0.253318514  0.78773885      q12             q1
##  52: -0.028546755  0.76904224      q12             q1
##  53: -0.042870457  0.33220258      q23             q2
##  54:  1.368602284 -1.00837661      q33             q3
##  55: -0.225770986 -0.11945261      q23             q2
##  56:  1.516470604 -0.28039534      q31             q3
##  57: -1.548752804  0.56298953      q24             q2
##  58:  0.584613750 -0.37243876      q32             q3
##  59:  0.123854244  0.97697339      q12             q1
##  60:  0.215941569 -0.37458086      q32             q3
##  61:  0.379639483  1.05271147      q12             q1
##  62: -0.502323453 -1.04917701      q43             q4
##  63: -0.333207384 -1.26015524      q43             q4
##  64: -1.018575383  3.24103993      q22             q2
##  65: -1.071791226 -0.41685759      q42             q4
##  66:  0.303528641  0.29822759      q14             q1
##  67:  0.448209779  0.63656967      q12             q1
##  68:  0.053004227 -0.48378063      q32             q3
##  69:  0.922267468  0.51686204      q11             q1
##  70:  2.050084686  0.36896453      q11             q1
##  71: -0.491031166 -0.21538051      q41             q4
##  72: -2.309168876  0.06529303      q24             q2
##  73:  1.005738524 -0.03406725      q13             q1
##  74: -0.709200763  2.12845190      q22             q2
##  75: -0.688008616 -0.74133610      q43             q4
##  76:  1.025571370 -1.09599627      q33             q3
##  77: -0.284773007  0.03778840      q23             q2
##  78: -1.220717712  0.31048075      q24             q2
##  79:  0.181303480  0.43652348      q12             q1
##  80: -0.138891362 -0.45836533      q41             q4
##  81:  0.005764186 -1.06332613      q34             q3
##  82:  0.385280401  1.26318518      q12             q1
##  83: -0.370660032 -0.34965039      q41             q4
##  84:  0.644376549 -0.86551286      q34             q3
##  85: -0.220486562 -0.23627957      q41             q4
##  86:  0.331781964 -0.19717589      q32             q3
##  87:  1.096839013  1.10992029      q11             q1
##  88:  0.435181491  0.08473729      q14             q1
##  89: -0.325931586  0.75405379      q21             q2
##  90:  1.148807618 -0.49929202      q31             q3
##  91:  0.993503856  0.21444531      q13             q1
##  92:  0.548396960 -0.32468591      q32             q3
##  93:  0.238731735  0.09458353      q14             q1
##  94: -0.627906076 -0.89536336      q43             q4
##  95:  1.360652449 -1.31080153      q33             q3
##  96: -0.600259587  1.99721338      q22             q2
##  97:  2.187332993  0.60070882      q11             q1
##  98:  1.532610626 -1.25127136      q33             q3
##  99: -0.235700359 -0.61116592      q41             q4
## 100: -1.026420900 -1.18548008      q44             q4
##                 x           y quadrant prior.quadrant
## 
## $regression.points
##    quadrant          x          y
## 1:       q1  0.5135824  0.3326156
## 2:       q2 -0.5625470  0.6488116
## 3:       q3  0.6961300 -0.7567517
## 4:       q4 -0.6945103 -0.6947171

`NNS` Covariance

Next we can show the covariance matrix from the partial moment quadrants:

Co-upper partial moment CUPM [upper right]
Co-lower partial moment CLPM [lower left]
Divergent-lower partial moment DLPM [lower right]
Divergent-upper partial moment DUPM [upper left]

By adding of the CLPM and CUPM off-diagonals, and subtracting the DLPM and DUPM off-diagonals, we arrive at the covariance of \(x,y\).

This is critical as it permits completely different objective functions, rather than just manipulating the entire covariance matrix.

Please see the following for more on partial moments’ role as the elements of variance and links to even more equivalences with traditional measures:

Elements of Variance https://www.linkedin.com/pulse/elements-variance-fred-viole/

Here are the individual partial moment matrices, and the aggregate covariance matrix provided by NNS.

# Store x,y into Matrix form
A=cbind(x,y)
PM.matrix(LPM_degree = 1, UPM_degree = 1, target = "mean", variable = A, pop_adj = TRUE)

## $cupm
##           x         y
## x 0.4299250 0.1033601
## y 0.1033601 0.5411626
## 
## $dupm
##           x         y
## x 0.0000000 0.1469182
## y 0.1560924 0.0000000
## 
## $dlpm
##           x         y
## x 0.0000000 0.1560924
## y 0.1469182 0.0000000
## 
## $clpm
##           x         y
## x 0.4033078 0.1559295
## y 0.1559295 0.3939005
## 
## $cov.matrix
##             x           y
## x  0.83323283 -0.04372107
## y -0.04372107  0.93506310

# Traditional Covariance
cov(A)

##             x           y
## x  0.83323283 -0.04372107
## y -0.04372107  0.93506310

Pitfall #2: Correlations / Betas

NNS is able to determine non-linear relationships between variables. See the article for a full demonstration on NNS correlation and dependence:

Nonlinear Correlation and Dependence Using NNS https://ssrn.com/abstract=3010414.

NNS regressions are not dominated by outliers and can accurately determine partial derivatives in the presence of noise. See the following article,

Nonparametric Regression Using Clusters http://rdcu.be/tz0J.

Below we recreate the examples Marcos provides and report the NNS results for correlation and dependence:

# Generate x,y,e 
set.seed(123);x=rnorm(1000);y=rnorm(1000);e=rnorm(1000)
y=100*x+e
NNS.dep(x,y,print.map = TRUE)

## $Correlation
## [1] 0.9877512
## 
## $Dependence
## [1] 0.9877512

y=100*abs(x)+e
NNS.dep(x,y,print.map = TRUE)

## $Correlation
## [1] 0.09562359
## 
## $Dependence
## [1] 0.9086349

Pitfall #3: Variance Adjudication & The Causality Fallacy

NNS regression is not just interpolation, but very good extrapolation as well. See the previous regression link for more. Nonparametric Regression Using Clusters
Regression is not the wrong tool per se, rather what and how you are regressing is of more importance. See the following forecasting presentation which utilizes NNS regressions to achieve excellent forecasting results.

`NNS` Forecasting Presentation

Download the .pdf file here: https://ssrn.com/abstract=3382300

`NNS` Forecasting vs. KERAS LSTM Deep Learning

View and download the .html file here: https://htmlpreview.github.io/?https://github.com/OVVO-Financial/NNS/blob/NNS-Beta-Version/examples/Sunspots_example.html

Pitfall #4: Specification-Interaction Search

NNS can properly consider missing interaction effects as well, since, NNS is indeed a machine learning method! The clustering NNS.part is the basis for NNS.reg . It effectively bridges the supervised and unsupervised leaning branches in the image below.

`NNS` Machine Learning

See the following link for NNS comparisons vs. other machine learning methods on classical problem sets:

Classification Using NNS Clustering Analysis https://ssrn.com/abstract=2864711

`NNS` Stack

See the following link for a brief comparison of the NNS.stack method (again, all based on the NNS clustering and regression methods) vs. other inference methods https://papers.ssrn.com/abstract=3592491

Below is the example Marcos uses for the missing interaction term and correlation between predicted and actual observations using NNS.

set.seed(123);x_1=rnorm(1000);x_2=rnorm(1000);e=rnorm(1000)

y = 100*x_1 + 100*x_2 + 100*(x_1 * x_2) + e

# Generate new test data
set.seed(321);test.x_1=rnorm(1000);test.x_2=rnorm(1000)

# Store estimates
nns.estimates = NNS.stack(cbind(x_1,x_2), y, IVs.test = cbind(test.x_1,test.x_2))$stack

actual.values = 100*test.x_1 + 100*test.x_2 + 100*(test.x_1 * test.x_2)

plot(nns.estimates,actual.values)

cor(nns.estimates,actual.values)

## [1] 0.9913709

Pitfall #5: p-Values

We identify a problem with p-values where the power of the test is completely ignored. NNS proposes a solution based on degree 1 Lower Partial Moment CDFs.

Below is an image from the paper illustrating a decreasing \(\beta\), signifying an increasing power (in blue) and how p-values (in red) jump to significant levels almost immediately!

`NNS` p-value Paper

The full paper demonstrating this effect is available for download here:

Continuous CDFs and ANOVA with NNS https://ssrn.com/abstract=3007373

`NNS` Feature Importance

Below is an example where the NNS dimension reduction regression is able to identify the significant regressors from 45 additional nonsensical noisy regressors. NNS also contains a routine NNS.stack to find the optimal threshold to seperate these regressors.

# Noisy regressor example from: http://www.win-vector.com/blog/2016/05/pcr_part1_xonly/
# build example where even and odd variables are bringing in noisy images
# of two different signals.
mkData <- function(n) {
  for(group in 1:10) {
    # y is the sum of two effects yA and yB
    yA <- rnorm(n)
    yB <- rnorm(n)
    if(group==1) {
      d <- data.frame(y=yA+yB+rnorm(n))
      code <- 'x'
    } else {
      code <- paste0('noise',group-1)
    }
    yS <- list(yA,yB)
    # these variables are correlated with y in group 1,
    # but only to each other (and not y) in other groups
    for(i in 1:5) {
      vi <- yS[[1+(i%%2)]] + rnorm(nrow(d))
      d[[paste(code,formatC(i,width=2,flag=0),sep='.')]] <- ncol(d)*vi
    }
  }
  d
}

set.seed(12345)
dTrain <- mkData(1000)
dTest <- mkData(1000)

# Find optimal threshold and store output 
optimal.threshold = NNS.stack(dTrain[,-1], dTrain[,1], method = 2,
                              dim.red.method = "cor")$NNS.dim.red.threshold
optimal.threshold

## [1] 0.36

# Print synthetic regressor equation using 'optimal.threshold'
print( NNS.reg(dTrain[,-1], dTrain[,1],  dim.red.method = "cor" ,
               threshold = optimal.threshold, plot = TRUE, smooth = TRUE)$equation )

##        Variable Coefficient
##  1:        x.01   0.4064923
##  2:        x.02   0.3623050
##  3:        x.03   0.4342246
##  4:        x.04   0.3943531
##  5:        x.05   0.4009567
##  6:   noise1.01   0.0000000
##  7:   noise1.02   0.0000000
##  8:   noise1.03   0.0000000
##  9:   noise1.04   0.0000000
## 10:   noise1.05   0.0000000
## 11:   noise2.01   0.0000000
## 12:   noise2.02   0.0000000
## 13:   noise2.03   0.0000000
## 14:   noise2.04   0.0000000
## 15:   noise2.05   0.0000000
## 16:   noise3.01   0.0000000
## 17:   noise3.02   0.0000000
## 18:   noise3.03   0.0000000
## 19:   noise3.04   0.0000000
## 20:   noise3.05   0.0000000
## 21:   noise4.01   0.0000000
## 22:   noise4.02   0.0000000
## 23:   noise4.03   0.0000000
## 24:   noise4.04   0.0000000
## 25:   noise4.05   0.0000000
## 26:   noise5.01   0.0000000
## 27:   noise5.02   0.0000000
## 28:   noise5.03   0.0000000
## 29:   noise5.04   0.0000000
## 30:   noise5.05   0.0000000
## 31:   noise6.01   0.0000000
## 32:   noise6.02   0.0000000
## 33:   noise6.03   0.0000000
## 34:   noise6.04   0.0000000
## 35:   noise6.05   0.0000000
## 36:   noise7.01   0.0000000
## 37:   noise7.02   0.0000000
## 38:   noise7.03   0.0000000
## 39:   noise7.04   0.0000000
## 40:   noise7.05   0.0000000
## 41:   noise8.01   0.0000000
## 42:   noise8.02   0.0000000
## 43:   noise8.03   0.0000000
## 44:   noise8.04   0.0000000
## 45:   noise8.05   0.0000000
## 46:   noise9.01   0.0000000
## 47:   noise9.02   0.0000000
## 48:   noise9.03   0.0000000
## 49:   noise9.04   0.0000000
## 50:   noise9.05   0.0000000
## 51: DENOMINATOR   5.0000000
##        Variable Coefficient

Pitfall #6: Training-Set Overfitting

NNS has the ability to perfectly fit any training set. However, the ability to accurately determine signal:noise ratios allows NNS to avoid overfitting. The regression paper Nonparametric Regression Using Clusters link from pitfall #2 thoroughly discusses and proves this avoidance of overfitting.

Pitfall #7: Test-Set Overfitting

The use of EXPECTED partial moments goes a long way in addressing this issue in finance. We published a not the way to accomplish this and underlying rationale borrowed from information theory. Please see the following link for more on the method and link to the paper:

Expected Partial Moments https://www.linkedin.com/pulse/expected-partial-moments-fred-viole/

Summary

NNS is a quite capable methodology able to properly address many of these pressing issues Marcos so diligently raises.

I look forward to further discussions and collaboration with those equally as passionate about these issues, and open to embracing alternative solutions. If you found this presentation interesting or useful, please feel free to reach out via e-mail: ovvo.financial.systems@gmail.com

Thanks for your interest!

The 7 Reasons Most Econometric Investments Fail - NNS Contributions Towards Solutions

Introduction

Load Required Packages in R NNS (>= 11.6)

Pitfall #1: Structured Data

NNS Clustering

NNS Covariance

Pitfall #2: Correlations / Betas

Pitfall #3: Variance Adjudication & The Causality Fallacy

NNS Forecasting Presentation

NNS Forecasting vs. KERAS LSTM Deep Learning