ENH: Automatic feature selection for DDIC

[klemengit]

@ThijsIllimited, you have nicely implemented the DDIC algorithm and it works great!
I have been using it a lot, however, it would be very nice if we would also have the automatic feature selection implemented somewhere in pyidi (used in the article).

What do you think, would you be up for it? 🙂

[ThijsIllimited]

Yes, I am interested in doing this. I was thinking of adding this as functionality to the selection.py file, but I understand you started a new selection tool already?

Maybe we can make a second tab where you set the filter type and the selection algorithm.
Let me know if you have a different idea.

[klemengit]

The selection tool remains the same, I started work on the result viewer (pyqtgraph based video + displacement viewer).
It would make the most sense to have a new tab in the selection tool; first tab for manual selection (selecting subset, selecting polygon of subsetes etc.) and the second tab for automatic feature selection.

But I would start by just implementing the code (without the interface) for the automatic feature selection.
I created a new branch selection_upgrade where the functionality upgrade should happen (to keep the master clean in case some updates must be released before the selection functionality is finished). We can create forks and make pull-requests to this branch. On the new branch I also added a selection folder and moved the selection.py in that folder. In the folder we would have the separated UI code and the actual implementations.

In addition to the automatic feature selection, I would also like to upgrade the current functionality of the selection tool. So I'll work on that a bit, and you can prepare the code for automatic feature selection? Then we can figure out how to best do the interface.

[ThijsIllimited]

I started something for the automatic feature selection, I will continue soon. I put everything I need in a separate folder for now: Automatic_selection

The pipeline I have in mind is:

1. Mask the image. For example inside a polygon from the Selector GUI
2. Filter the image. For example Shi-Tomasi
3. Pick the points. I have three picking algorithms implemented.

I like the idea of the automatic selection functionalities to also be available outside of the GUI.

Let me know if I'm not being clear or if you have a different vision for this

[klemengit]

Hi, yes, that's exactly what I had in mind 🙂
I'm not sure that you have seen the recent pushes we did (also a new release).
The current implementation (see #52 ) works like this: first the points are selected (by drawing a polygon, line, using a brush ...). Then these points are filtered using one of the algorithms. Currently the Shi-Tomasi and just a simple direction filter are implemented.

But the filtering methods are integrated in the GUI code, so I agree to have them separate.

Here's how the current implementation looks like.

[ThijsIllimited]

I saw, this is going to be very convenient!

The difference I am suggesting is to filter the whole image instead of just the points on a line or a grid. The tracking points are selected after, for example at local maxima. This would enable picking points on intricate structures (like spider webs).

Definitely not suggesting to remove what you have now. I think regular grids will be something people want. Let's do both.

How about I implement above as standalone first, and then we can talk about how to combine it in the selection tool?

[ThijsIllimited]

I added Showcase_point_picker.ipynb as a tutorial of my codes. Only the masking is not included yet. I'm interested to hear what you think about it

[klemengit]

I like this! I like the modularity and I think we can include it in the selection tool nicely.

Regarding the workflow, I think that what you say makes sense. And in some sense this workflow is already implemented, but a bit more implicitly. So:

1. Mask the image: first you select all the candidate points (using, e.g., the Grid, which should probably be renamed to "Polygon" or something). If the "Distance between subsets" is set to negative 1/2 of the subset size, all the locations inside that polygon (mask) are selected.
2. Filter the image: Then, the filtering tab is selected, where the filter is applied.
3. Pick the points: After the filter is computed, the points can be selected by changing the threshold values.

My reasoning was that sometimes taking all the points into account is a bit much. So a user would create a spaced grid. Then, filter the selected points to not take any that don't have sufficient features.

I'm thinking that the tool could be changed in the following way to accommodate both ideas:

• The Selection/Filter tabs remain.
• The Selection methods (basically masking tools) also remain: grid (Polygon), Manual, Along the line, Brush.
• A checkbox is added. If checked, all the points inside the mask are selected. If not checked, the "Distance between subsets" slider appears to reduce the number of selected points.
• When going to the "Filter" tab, a filter method can be selected. The points, selected in the previous step, are filtered. Depending on the selected filter, the filtering options are shown.
• Changing the filtering options (threshold etc.) the final points are obtained.

What do you think? If you have a more elegant solution, don't hesitate to share :)

In any case, I'll further develop the GUI to make a kind of a framework for efficiently adding new selection and filtering methods and make maintenance easier (currently it's a bit of a mess).

[klemengit]

I was thinking about the naming of the steps and the "picking the points" in the last step doesn't feel right to me :) The "picking" step seems like the filtering step. I suggest the following, tell me what you think about it:

1. Mask the image: select the regions and candidate points (can be all points in the mask or spaced).
2. Evaluate the candidates: use one of the algorithms to compute the score for each of the selected regions/points (e.g. Shi-Tomasi).
3. Filter the candidates: select the algorithm to filter through the candidate points based on the evaluation scores of the points.

Basically, the filtering step before didn't feel like a filtering step. It basically computes the scores for each point. These steps seem more natural and logical to me. Let me know your thoughts.

[ThijsIllimited]

About the methods:
My preference is to quantify expected tracking performance everywhere with a filter to generate a score image. When using scipys filters I have found this to be fast. Especially, as High Speed Cameras currently are 1MP only. This is also conform with methods like SIFT, ORB and OpenCV's goodFeaturesToTrack. The problem I have with only checking regularly spaced candidates is that you might miss good locations between sampling points. Another consideration: state of the art feature selectors like SIFT and ORB allow for subpixel keypoint selection. If we ever want to include such functionalities, I believe it fits better in a context of generating a score image with a filter, than checking candidates in isolation.
I think including the regularly spaced features is a good idea. What do you think about an approach that masks score_image such that only the desired candidates are included in the picking algorithm?

About the terminology:
I see that using "filtering" is going to be confusing. What do you think about calling this step:

• Convolve
• Quantify expected performance
• Generate score image

Then selecting the points from the candiates/score_image we can call

• Detection (Like in ORB)
• Extract keypoints (like in SIFT)
• Selection

[jankoslavic]

Thank you @ThijsIllimited and @klemengit . We should try to make the terminology obvious and simple. Where we already have terms defined from other fields, we should use them and change them only if it really makes sense in our field.

As I see it, @ThijsIllimited was talking about filtering when @klemengit was talking about evaluation. When @ThijsIllimited was talking about selection, @klemengit was talking about filtering:). Maybe I am wrong, but we have three steps:

1. apply mask -> get candidate subsets
2. evaluate [quality of candidate subsets] -> get some sort of quantitative measure of quality
3. select [best candidates]

How does this sound to you?

[klemengit]

@ThijsIllimited I definitely agree, using regularly spaced intervals only would be problematic, but that's not a limitation even in the current implementation. One can always set the spacing in such a way that all points in the mask are selected.

Similar to what you describe, my idea was (building on your last comment), to have the following steps (actually they coincide with what @jankoslavic suggested pretty well):

1. Mask: define the mask with different tools. The tools are: inside polygon, selection along the line, manual point picking.
2. Evaluate: compute the scores for the locations inside the combined mask. I like the term "evaluate" but "generate score image" is also a very good fit.
3. Filter: Depending on the method, the user would change the parameters (some sort of threshold for score, min distance between the points etc.). The term "select" also fits, but I actually think that the term "filter" is still better. Because in this step, the actual filtering of the subsets/points/pixels happens. The filter uses some data (score values) to filter the appropriate location.

I actually already started a new implementation based on this workflow.

My vision is:

• in the "mask" step, the GUI would not show any points, it would just define the masks (maybe it would display a transparent overlay to indicate the masked areas).
• In the "evaluate" step, the user would select the method and define the subset size, gradient direction etc. (depending on the selected method).
• In the "filter" step, the user would again select the method and adjust the method-specific parameters. The candidate points are actually shown only now, when the filter is adjusted.

I will try to make it modular and able to accommodate different evaluation and filtering method.

So, what do you think about the workflow and the naming? 🙂

[jankoslavic]

Sounds fine to me! Thanks

[ThijsIllimited]

This all looks good to me. I vote "evaluate" for step 2 and "select" for 3.

In step 2, are you planning to to use the generic_filters from scipy like I did? I expect issues when filtering masked images? Or will you just make a loop over a list of pixels making up the mask?