next up previous contents
Next: 5.2.1.7 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.5 Basic Stability Test:   Contents


5.2.1.6 Basic Stability Test: ds2


Table 5.8: IRLS stability experiments for ds2. binitmean is disabled and wmargin is 0. The first four columns represent the state of modelmin and modelmax, margin, rrlambda, and cgwindow and cgdecay.
           Loose Epsilon Moderate Epsilon  Tight Epsilon             
mm  mar  rrl  cgw AUC  NaN  DEV  Time AUC  NaN  DEV  Time AUC  NaN  DEV  Time
-  -  -  - 0.478  -  8898  290 0.684  -  4378  731 0.669  x  41  21477
x  -  -  - 0.478  -  8898  293 0.684  -  4378  665 0.669  x  41  21486
-  x  -  - 0.477  -  8541  293 0.688  -  3963  641 0.649  -  45  56435
x  x  -  - 0.477  -  8541  294 0.688  -  3963  643 0.649  -  45  57015
-  -  x  - 0.478  -  8903  292 0.689  -  4377  681 0.720  -  963  10804
x  -  x  - 0.478  -  8903  296 0.689  -  4377  684 0.720  -  963  10799
-  x  x  - 0.477  -  8546  294 0.688  -  3966  658 0.722  -  879  10800
x  x  x  - 0.477  -  8546  295 0.688  -  3966  657 0.722  -  879  10805
-  -  -  x 0.478  -  8898  292 0.684  -  4378  738 0.681  x  254  3200
x  -  -  x 0.478  -  8898  292 0.684  -  4378  738 0.682  x  253  3250
-  x  -  x 0.477  -  8541  292 0.688  -  3963  694 0.683  x  218  3647
x  x  -  x 0.477  -  8541  293 0.688  -  3963  647 0.686  x  206  4502
-  -  x  x 0.478  -  8903  292 0.689  -  4377  742 0.720  -  961  6034
x  -  x  x 0.478  -  8903  295 0.689  -  4377  747 0.720  -  961  5707
-  x  x  x 0.477  -  8546  295 0.688  -  3966  716 0.722  -  877  6312
x  x  x  x 0.477  -  8546  296 0.688  -  3966  657 0.722  -  877  5852

The final sparse dataset is ds2, and stability experiments for this dataset may be found in Table 5.8. Again modelmin and modelmax make little difference, and the same can be said for margin. The large time difference between the first two pairs of rows is due to NaN values terminating IRLS iterations. The application of rrlambda does help AUC scores by apparently reducing overfitting in experiments with tight epsilons, as usual. There is one new twist with this dataset. When rrlambda is combined with cgwindow and cgdecay for experiments with tight epsilons we observe a dramatic speed improvement. It is possible that choosing a different rrlambda value than 10 may achieve the same speed improvement, and we will address this issue later.


next up previous contents
Next: 5.2.1.7 Basic Stability Test: Up: 5.2.1 Indirect (IRLS) Stability Previous: 5.2.1.5 Basic Stability Test:   Contents
Copyright 2004 Paul Komarek, komarek@cmu.edu