Incremental Learning with Learn++ & Learn++.NC

Robi Polikar

---

For details on how to use Learn++.NC on an incremental learning problem with new classes, please see our recent paper:

Muhlbaier M., Topalis A., Polikar R., “Learn++.NC: Combining Ensemble of Classifiers Combined with Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes,” IEEE Transactions on Neural Networks, vol. 20, no. 1, pp. 152 – 168, 2009.

---

The information on this page is intended for the reviewers of the paper entitled

---

This page includes the following:

1. A powerpoint presentation that provides some preliminary background on ensemble systems, the Learn++ concept, and some additional results on two artificial datasets. The datasets were designed to provide additional insight to how the Learn++.NC algorithm works, and not to be particularly challenging from a pattern recognition point of view. The synthetic and real datasets that were used to present the results in the paper are, however, challenging datasets. Several animations are also included in the slideshow to demonstrate some of the concepts described in the paper.
   
   You may find this presentation here: Learn⁺⁺NC.pps
   
   This file is password protected. The password to open the file is lpp.nc. If Powerpoint asks for an additional password to save/modify the file, please click on “Read Only” to access the file.
2. A movie for the triple spiral database (discussed in the paper) that shows the decision boundaries generated by Learn++ and Learn++.NC as each new classifier is added.
   
   You can access the movie file here: Learn⁺⁺.NC movie
   
   The typical layout of the frames in the movie is as follows, which is described below the figure.

The problem is learning the decision boundaries of a double interweaved spiral from the first dataset, and then learn a third spiral place in between from a second dataset. The black spirals indicate the training data (200 points per spiral), where as the painted regions indicate the decision regions for each class.

The upper left figure shows the ensemble decision of Learn++ after 10 classifiers (in the above example), whereas the upper right figure shows the decision of the 10th individual classifier in the ensemble. The lower figures show the ensemble decision of Learn++.NC (left) and the 10th individual Learn++.NC classifier. Note that both classifiers have learned the boundaries using 10 classifiers (or fewer –see the movie) .

A second database that includes a third spiral (also 200 points) is then introduced, and both algorithm are asked to continue generating additional classifiers. The following frame shows the decision regions after the 11th classifier.



Now, the individual classifiers, which are trained with data that include all three classes learn the boundaries (upper and lower right). Looking at the ensemble decisions, however, we see that Learn++.NC have immediately learned the third class (lower left), whereas Learn++ has not. This is because, the vote provided by the 11th Learn++ classifier (the only one so far that has seen the third class) is out-voted by the previous 10 that have not seen the third class. The dynamically-weighted consult-and-vote used by Learn++ specifically addresses this problem.

The figure below shows the boundaries after 20 classifiers, where Learn++ starts learning the third class:



And after 30 classifiers, at which time both algorithms have learned all three classes equally well.

---

The material provided here is based upon work supported by the National Science Foundation under Grant No ECS-0239090, “CAREER: An Ensemble of Classifiers Based Approach for Incremental Learning,” by Grant No ECCS 0926159 Incremental Learning from Unbalanced Data in Nonstationary Environments , and by Grant No. ECCS 1310496 Learning from Initially Labeled Nonstationary Streaming Data. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.