next up previous contents
Next: 5.2.1.9 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.7 Basic Stability Test:   Contents


5.2.1.8 Basic Stability Test: ds1.100pca


Table 5.9: IRLS stability experiments for ds1.100pca. binitmean is disabled and wmargin is 0. The first four columns represent the state of modelmin and modelmax, margin, rrlambda, and cgwindow and cgdecay.
           Loose Epsilon Moderate Epsilon  Tight Epsilon             
mm  mar  rrl  cgw AUC  NaN  DEV  Time AUC  NaN  DEV  Time AUC  NaN  DEV  Time
-  -  -  - 0.906  -  3736  34 0.916  -  3356  66 0.919  -  3275  177
x  -  -  - 0.906  -  3736  36 0.916  -  3356  66 0.919  -  3275  177
-  x  -  - 0.906  -  3553  35 0.914  -  3246  60 0.919  -  3154  173
x  x  -  - 0.906  -  3553  35 0.914  -  3246  62 0.919  -  3154  173
-  -  x  - 0.906  -  3745  34 0.915  -  3402  62 0.919  -  3291  163
x  -  x  - 0.906  -  3745  34 0.915  -  3402  62 0.919  -  3291  164
-  x  x  - 0.906  -  3561  36 0.914  -  3259  61 0.918  -  3166  163
x  x  x  - 0.906  -  3561  32 0.914  -  3259  61 0.918  -  3166  164
-  -  -  x 0.905  -  3754  27 0.919  -  3295  61 0.919  -  3275  149
x  -  -  x 0.905  -  3754  27 0.919  -  3295  59 0.919  -  3275  150
-  x  -  x 0.905  -  3572  28 0.916  -  3213  53 0.919  -  3154  151
x  x  -  x 0.905  -  3572  28 0.916  -  3213  53 0.919  -  3154  152
-  -  x  x 0.905  -  3760  27 0.917  -  3342  55 0.919  -  3291  137
x  -  x  x 0.905  -  3760  28 0.917  -  3342  55 0.919  -  3291  137
-  x  x  x 0.903  -  3586  28 0.914  -  3244  47 0.918  -  3166  137
x  x  x  x 0.903  -  3586  27 0.914  -  3244  47 0.918  -  3166  138

We now consider the same stability experiments as above on two dense datasets, ds1.100pca and ds1.10pca. As discussed in Section 5.1.3, these datasets are PCA projections of ds1 down to 100 and 10 dimensions, respectively. Table 5.9 summarizes the results on ds1.100pca. As in the sparse experiments, all values in the Loose Epsilon group are within ten percent of one another. The modelmin and modelmax parameters are familiarly ineffective. The margin parameter seems to increase speed and decrease deviance in some experiments, but those same experiments show a decreased AUC . Regularization through rrlambda may have some effect, but that effect is even less than that of margin. The cgwindow and cgdecay parameters clearly reduce the time required by the computations, and have a small and probably insignificant effect on the scores. No overfitting is apparent anywhere in Table 5.9. We have run additional experiments with cgeps as low as 1e-9 and lreps as low as 1e-7. While the time was approximately doubled, the minimum average deviance per fold and AUC scores were unchanged.


next up previous contents
Next: 5.2.1.9 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.7 Basic Stability Test:   Contents
Copyright 2004 Paul Komarek, komarek@cmu.edu