Filtering by subfolder option in parse_folder script

[crohkohl]

This pull request is the result of the discussion in #212.

It is aimed at handling the following import case when parsing data folders.

Simple example: there are images from two categories and for every image we create augmented versions by mirroring resulting in a folder structure comparable to:

├── images/
│   ├── cat/
│   │   ├── img1/
│   │   |     └── 1_original.jpg
│   │   |     └── 1_mirrorX.jpg
│   │   ├── img2/
│   │   |     └── 2_original.jpg
│   │   |     └── 2_mirrorX.jpg

Currently, the structure is just flattened and then splitted into train and validation datasets. So it could happen that this split is chosen:

Training:

│   ├── cat/
│   │  └── img1/1_original.jpg
│   │  └── img2/2_mirrorX.jpg

Validation:

│   ├── cat/
│   │  └── img1/1_mirrorX.jpg
│   │  └── img2/2_original.jpg

That is not the desired result because it mixes data that originated from the same source image in training and validation. What you would want is:

Training:

│   ├── cat/
│   │  └── img1/1_original.jpg
│   │  └── img1/1_mirrorX.jpg

Validation:

│   ├── cat/
│   │  └── img2/2_original.jpg
│   │  └── img2/2_mirrorX.jpg

A new argument called --split_by_subfolder has been added to

https://github.com/crohkohl/DIGITS/blob/split_by_subfolder/tools/parse_folder.py#L524

which leads to that behaviour. The data is first grouped by the deepest sub-folder name in a dictionary - then divided into train / val / test and finally all group items are added to the image lists.

[lukeyeager]

Neat! I tested it and it seems to work.

1. This should be exposed as an option through the UI so people can actually use it.
   • Add a field to digits/dataset/images/classification/forms.py
   • Display it in the template at digits/templates/datasets/images/classification/new.html
2. Add some tests, please!
   • At least in tools/test_parse_folder.py
   • For bonus points, add a test to TestCreation in digits/dataset/images/classification/test_views.py. That may require modifying digits/dataset/images/classification/test_imageset_creator.py to create classes with subfolders.

[crohkohl]

Okay, I will look into that.

[lukeyeager]

If it's helpful, you can look at what I've done in #226 as reference for how to do (1) and (2).

[lukeyeager]

@crohkohl, you still there? This is great stuff and I would like to see it merged. Do you want for me to take over?