Survey

Knowledge

Response-based Knowledge

Feature-based Knowledge

Relation-based Knowledge

Relation between layers/samples

Distillation Schemes

Offline Distillation