next up previous contents
Next: 5.2.1.12 Stability Test: binitmean Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.10 Basic Stability Test:   Contents


5.2.1.11 Stability Test: wmargin


Table 5.11: IRLS stability experiments for ds1 with wmargin=1e-13. binitmean is disabled.
        Moderate Epsilon
mm mar rrl cgw AUC NaN DEV Time
- - - - 0.890 x 479 585
x - - - 0.893 - 307 1153
- x - - 0.894 - 592 659
x x - - 0.893 - 595 647
- - x - 0.948 - 2087 127
x - x - 0.948 - 2087 128
- x x - 0.948 - 1990 128
x x x - 0.948 - 1990 129
- - - x 0.934 x 1367 164
x - - x 0.933 - 1376 183
- x - x 0.927 - 1204 151
x x - x 0.927 - 1204 150
- - x x 0.948 - 2087 95
x - x x 0.948 - 2087 94
- x x x 0.948 - 1990 94
x x x x 0.948 - 1990 95

Turning our attention to the wmargin parameter, we examine Table 5.11. These experiments enable the wmargin parameter for moderate epsilon experiments on the ds1 dataset. We chose to use the ds1 dataset and the Moderate Epsilon group for these tests because this combination showed adequate sensitivity to parameter changes in Section 5.2.1.3. The wmargin parameter was added to our software before rrlambda was implemented in an effort to eliminate NaN values. When this table is compared with the Moderate Epsilon group of Table 5.5, we observe the elimination of NaN values in all non-rrlambda experiments for which either modelmin and modelmax, or margin, are enabled. This same subset of experiments have similar scores. We note that wmargin has an undesirable effect on speed, especially in the second row of the table. In previous experiments, the occurrence of NaN values caused immediate termination of IRLS iterations. With wmargin eliminating the NaN values, training continues longer, deviance is reduced, and times increase. Since the holdout performance, judged by AUC , decreased we can assume the wmargin helps IRLS overfit the training data. Our conclusion is that using wmargin is inferior to using combinations of rrlambda, cgwindow and cgdecay.


Table 5.12: IRLS stability experiments for ds1.100pca with wmargin=1e-13. binitmean is disabled.
        Moderate Epsilon
mm mar rrl cgw AUC NaN DEV Time
- - - - 0.916 - 3356 65
x - - - 0.916 - 3356 65
- x - - 0.914 - 3246 60
x x - - 0.914 - 3246 59
- - x - 0.915 - 3402 59
x - x - 0.915 - 3402 60
- x x - 0.914 - 3259 59
x x x - 0.914 - 3259 59
- - - x 0.919 - 3295 60
x - - x 0.919 - 3295 60
- x - x 0.916 - 3213 53
x x - x 0.916 - 3213 52
- - x x 0.917 - 3342 54
x - x x 0.917 - 3342 54
- x x x 0.914 - 3244 47
x x x x 0.914 - 3244 47

In Table 5.12 we repeat the wmargin experiments for the dense dataset ds1.100pca. Since wmargin was intended to remove some NaN values, and that ds1.100pca experiments had no NaN values, it is not surprising that this table is nearly identical to the Moderate Epsilon group of Table 5.9. As a result, these experiments leave our previous assessment of wmargin in tact.


next up previous contents
Next: 5.2.1.12 Stability Test: binitmean Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.10 Basic Stability Test:   Contents
Copyright 2004 Paul Komarek, komarek@cmu.edu