Hello, everyone, I’m Weikai Yang from Tsinghua University.
Today I’d like to introduce our work: “Diagnosing Ensemble Few-Shot Classifiers.”
Few-shot classification aims to train a classifier
with very few labeled samples,
and that’s what we call “shots.”
A promising way to do that is,
the learners will process these images and measure the similarities between images.
And the prediction is made based on the similarities between the sample and the shots.
For example, this sample x will be classified as y3star since it looks very similar to x3star.
From this process, we can see that the quality of learners and shots are the key factors
to the success of few-shot classification.
Including some poor-performing learners or low representative shots
will hurt the performance of the ensemble model.
Therefore, the key challenge is how to
select high-quality learners and shots that produce good classification results.
For learners, first, they should be well-performing.
In addition, they should be diverse because
combining multiple identical learners do not help improve the performance.
But they also need to be cooperative so that
they do not introduce too much contradiction and harm the performance.
For shots, we would need to remove those shots that cannot well represent their class
and add necessary new ones.
However, manually tuning the selection is a tedious trial-and-error process,
so we develop our visual analysis tool to facilitate this process.
First, both tasks aim to find a small representative subset from a large collection,
so we formulate them as sparse subset selection to automatically
recommend learners and shots.
However, the recommendations are not always perfect.
For example, a learner that wrongly predicts some samples with high confidence
may be recommended because it is mistaken as a well-performing learner for those samples.
Therefore, we develop a visualization to explain the recommendation results and
facilitate the interactive tuning of the selection.
Here comes the system overview.
Given a set of learners which can be extracted from the public available pretrained models,
for example, downloaded from pytorch or tensorflow,
and given a set of unlabeled samples and shots, the sparse subset selection module
automatically recommends a subset of learners and a small set of representative shots.
And they are used to build an ensemble model.
Next, the matrix visualization illustrates the performance of the learners
and helps adjust the selection.
Users can also examine the quality of the shots in the scatterplot
and replace the low-representative shots with the high-representative ones
After these adjustments, the recommendation can also be updated
and users can start a new round for analysis.
Next, I’d like to introduce the sparse subset selection module.
If we have defined the representation cost between each element ui and each element vj.
sparse subset selection can find a small subset of U that can represent V best.
Given a subset A, the representation cost will be the sum of the minimum cost
that represents each vj using the elements in subset A.
To penalize a large subset, we add a sparsity term, which is defined as the size of the subset.
Using the binary variable zij to indicates whether ui is used to represent vj,
we can rewrite this formulation.
Then we relax them to a continuous optimization problem and
solve it using the ADMM framework.
We then extend it to recommend learners and shots, respectively.
For more details about this part please refer to our paper.
We then conducted a quantitative evaluation to see whether the recommended learners
and shots can improve the performance of few-shot classification.
The results show that using the recommended learners or shots alone
can boost the performance,
and combining them can further improve the performance.
Next, we introduce our visualization that facilitate the understanding and
interactive improvement of the selection.
In sample view, we use tsne to project the samples to facilitate the
exploration and the adjustments on shots.
Since it is easy to implement, here we focus more on the learner view.
The learner view aims to help users examine each base learner and make adjustments.
The first issue is that we do not have ground-truth to evaluate the performance of each base learner.
So we compare their predictions with the ensemble model.
We design a matrix view.
Here, each row represents a learner,
and the first column encodes the overall difference on the prediction,
and the remaining columns encode the prediction differences on different classes.
In each cell, we use the venn diagram to represent the samples.
The first set represents the samples that are predicted as the class by the learner,
and the second set represents the samples that are predicted by the ensemble model.
The second issue is the scalability.
When there are too many learners or classes,
we group similar ones together,
and users can freely explore them and adjust the clusters.
Here, I would like to show a case study on hand-written digits classification.
I will introduce two steps in it, one starts from analyzing the learner,
and the other one starts from analyzing the shot
Four learners are recommended by the algorithm.
Among them, BL-tiered-6 has the largest prediction difference from the ensemble model.
We first analyze this learner and found that three regions
have a large drop in prediction confidence.
We selected these samples in the region and found that
There is no shot of 3 and only one shot of 5.
So we decided to recommend more shots in this region and add shots of 3 and 5.
Our system can also recommend shots globally.
Two low quality shots are recommended to be removed.
Their poor converge indicates that they are not representative.
Three samples are recommended as shots to be labeled.
By adjusting the learners, removing 2 shots and adding 13 new ones,
the accuracy is improved from 0.497 to 0.707.
To conclude our work, we developed a visual analysis tool for visually
diagnosing an ensemble few-shot classifier.
Our method includes two subset selections for recommending learners and shots
And we developed the learner view and sample view to facilitate interactive improvement.
In the future, we plan to generalize the learner view to other tasks such as object detection,
and improve the efficiency of sparse subset selection to handle millions of samples.
For more details and the materials such as code, video, and slides,
please refer to our repo.
Thank you and I am happy to take any question.