next up previous contents
Next: 5.2.1.10 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.8 Basic Stability Test:   Contents


5.2.1.9 Basic Stability Test: ds1.10pca


Table 5.10: IRLS stability experiments for ds1.10pca. binitmean is disabled and wmargin is 0. The first four columns represent the state of modelmin and modelmax, margin, rrlambda, and cgwindow and cgdecay.
           Loose Epsilon Moderate Epsilon  Tight Epsilon             
mm  mar  rrl  cgw AUC  NaN  DEV  Time AUC  NaN  DEV  Time AUC  NaN  DEV  Time
-  -  -  - 0.829  -  6255  4 0.842  -  5064  6 0.846  -  4985  11
x  -  -  - 0.829  -  6255  4 0.842  -  5064  6 0.846  -  4985  11
-  x  -  - 0.831  -  5679  4 0.841  -  4892  7 0.846  -  4828  10
x  x  -  - 0.831  -  5679  4 0.841  -  4892  6 0.846  -  4828  12
-  -  x  - 0.829  -  6256  4 0.842  -  5066  7 0.846  -  4985  12
x  -  x  - 0.829  -  6256  4 0.842  -  5066  7 0.846  -  4985  11
-  x  x  - 0.831  -  5679  4 0.841  -  4894  7 0.846  -  4828  12
x  x  x  - 0.831  -  5679  4 0.841  -  4894  7 0.846  -  4828  11
-  -  -  x 0.826  -  6333  3 0.841  -  5079  6 0.846  -  4985  12
x  -  -  x 0.826  -  6333  4 0.841  -  5079  7 0.846  -  4985  12
-  x  -  x 0.828  -  5570  4 0.841  -  4905  7 0.846  -  4837  27
x  x  -  x 0.828  -  5570  4 0.841  -  4905  6 0.846  -  4837  29
-  -  x  x 0.826  -  6333  2 0.841  -  5081  6 0.846  -  4985  18
x  -  x  x 0.826  -  6333  3 0.841  -  5081  6 0.846  -  4985  16
-  x  x  x 0.828  -  5571  4 0.841  -  4907  6 0.845  -  4837  31
x  x  x  x 0.828  -  5571  4 0.841  -  4907  6 0.845  -  4837  33

Table 5.10 summarizes results for dataset ds1.10pca. New in this table is that margin in combination with rrlambda decrease the deviance by more than ten percent. However no improvement in AUC or speed is seen from this combination. There is a somewhat startling jump in time on some of the experiments with cgwindow and cgdecay enabled. Comparison of these experiments to their counterparts lacking cgwindow and cgdecay reveals that cgwindow and cgdecay may have terminated CG iterations too quickly. In one dramatic example the cgwindow and cgdecay experiment ran thirty IRLS iterations, wherein the first two IRLS iterations showed CG stage termination after six CG iterations. The counterpart experiment ran CG for twelve iterations each for the same two IRLS iterations. In this example the cgwindow and cgdecay experiment achieved lower overall deviance for the fold, but not significantly and there was no clear sign of overfitting in either case. The AUC score remained essentially the same, but the speed decreased significantly. Note that the decrease in speed is especially pronounced when margin is in effect.

These experiments on ds1.10pca suggest our cgwindow and cgdecay parameters may be too tight. None of the tested stability parameters has significant effect on AUC score, and none increased speed. If the experiments in which margin is enabled are ignored, the negative effects of cgwindow and cgdecay are less disturbing.


next up previous contents
Next: 5.2.1.10 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.8 Basic Stability Test:   Contents
Copyright 2004 Paul Komarek, komarek@cmu.edu