How does LabGym identify different individuals of the same species?

[prashers]

Hello, I am interested in trying out LabGym for detecting quails in a flight pen that were recorded from a top-down view (see image below). I have installed the software successfully and produced some images to annotate using Roboflow.

It is extremely important for my work to know the identities of each individual in my videos (e.g., I need to know whether birdA pecked birdB or whether the peck actually came from birdC). I am wondering how LabGym accomplishes this.

When annotating images in Roboflow, do I need to have a unique class for each individual so that the detector learns to identify each individual bird? Or does the individual identification happen at a later step?

Thanks!

[yujiahu415]

Hi,
I noticed that there was a color marker for each bird, right? If the color is visually easy to identify all the time in the video, then you can try annotating each bird as a separate class, like "green bird", "red bird"...In this way, you don't need to worry about ID swap. Otherwise you just use one class "bird" to label all the individuals. During analysis, the IDs (will be shown as numbers) will be automatically assigned for each individual at the first frame, and LabGym will then track the IDs over time. In this case, ID swap might happen. To prevent that, annotate as many frames in which two or more birds collide as possible. If the Detector sees most colliding scenarios during training, ID swap can be significantly avoided. If you want to 100% eliminate ID swap, train a dedicated Detector for each video. When training such kind of Detectors, use the original frame size or its approximation (if the original is not an even number) as inferencing frame size, and use the maximum iteration number (5000).

[dr-moo]

I would also like to chime in again, @prashers! What is your setting? Are we talking broilers (my first assumption when I hear birds)? Research box? Any enrichment? Walkway with single individual? Several birds, and if that, how many? Commercial setup with many birds at the same time? And how is the camera placed - angled, top-down? Height? All these details matter and could steer the number of initial frames for annotation.

For example, when we had some projects with broilers and used segmentation for detection, we tried the commercial setup with several thousand birds in a frame and an enriched pen with 15-20 individuals. For the research pen, we had roughly 260 frames with 5000+ birds manually annotated, and for the commercial farm, we had 15 high-resolution frames with 7000 birds annotated across the frames. It all depends on your initial research question and scene variability. But the first iteration of algorithm development is ALWAYS time-consuming, and it is worth it.

When selecting frames for annotation, it could also be helpful to think in terms of dynamic scenes and pick variable sequences with as much change as possible but still close to the desired scene/event type you want to analyse, thus somewhat minimising the manual labour.

[yujiahu415]

Great inputs! @dr-moo
I guess the number of birds in @prashers 's scenarios would not be too many. But the identity of each bird is important. It's a good point that when selecting frames, you need to consider the balance between making the training dataset as diverse as possible and making it still close to the scene in analysis. A more extreme case is that you just annotate the frames in the videos to analyze to minimize the detection difficulty--after all you're not testing the generalizability of the Detectors but just want to make it as accurate as possible to enable analysis.

[prashers]

Thank you both for the input/suggestions!

@dr-moo , I have 6 birds in an indoor flight pen recorded from a top-down view (see picture in my first post in this thread). I really appreciate you giving me a reference point for how many frames it might take for me to get reliable results!

Updates:
Simply making sure I resized the frames to match the inferencing frame size (960) improved the tracking dramatically! It is still mixing up bird identities quite a bit (especially between birds wearing the white, black and blue markers), but it seems promising otherwise. I will be trying the other suggestions you mentioned to see if things improve. I will definitely be annotating more images - 45 was just for the initial test.

I would like the analysis to run a bit faster than it did for a frame size of 960, so I trained another detector with an inferencing frame size of 640 and 5000 iterations (the total_loss got down to 0.1523). I tried to analyze a video resized to 640 and it was running as normal, until after 15000 frames processed it returned this error:

C:\Users\sanja\miniconda3\envs\labgym4\lib\site-packages\LabGym\analyzebehaviorsdetector.py:857: RuntimeWarning: invalid value encountered in divide np.sum(np.logical_and(masks[:, None], masks), axis=(2, 3)) C:\Users\sanja\miniconda3\envs\labgym4\lib\site-packages\LabGym\analyzebehaviorsdetector.py:857: RuntimeWarning: invalid value encountered in divide np.sum(np.logical_and(masks[:, None], masks), axis=(2, 3)) Traceback (most recent call last): File "C:\Users\sanja\miniconda3\envs\labgym4\lib\site-packages\LabGym\gui\analysis\analyze_behaviors.py", line 1372, in analyze_behaviors AAD.acquire_information( File "C:\Users\sanja\miniconda3\envs\labgym4\lib\site-packages\LabGym\analyzebehaviorsdetector.py", line 1420, in acquire_information self.detect_track_individuals( File "C:\Users\sanja\miniconda3\envs\labgym4\lib\site-packages\LabGym\analyzebehaviorsdetector.py", line 927, in detect_track_individuals sorted(cnts, key=cv2.contourArea, reverse=True)[0] IndexError: list index out of range

[yujiahu415]

This error is simply because a detection failure, in which the contour cannot be closed and its area cannot be calculated, causing an opencv error. You can simply increase the frame size or train a Detector with more annotated images.

[yujiahu415]

The ID mix is possibly because the white, black, and blue markers are visually more difficult to distinguish. I guess the Detector had higher accuracy in distinguishing IDs with green and red markers?

[dr-moo]

Sorry, @prashers, i missed the picture initially, but i understand what you mean now. As @yujiahu415 points out, some markers will always be more difficult to distinguish. It does not mean it is impossible to solve, but robust tracking will require additional steps. You could always (and it is usually the best idea) include more annotated frames, where you make sure you have as many occluding birds as possible, as well as birds with different markers) to improve the overall detector accuracy. But this is only good until certain point, and as animal scientist, you might need to have a clear cut off point, where time investment for annotation becomes almost as extensive as just asking a student to look at the video manually (or if you do it yourself).

Another strategy is to accept the potential ID switches and work with log files, where you could stitch tracelets together by filtering out false IDs in between frames. This is still manual work but not as extensive as fully manual observations.

And path number three, if the continuous tracking is an absolute priority - Yolo (any version from 4 and up) in combination with SORT/DeepSORT work very well in setups like yours, but then you don't have user-friendly interface and comfort of LabGym.

[yujiahu415]

Thank you again, @dr-moo, for the insightful comments!!
@prashers, some experience / tips:

1. As @dr-moo mentioned, you only need to select the frames in which colliding occurs for annotation. In other word, you don't need to annotate any frames in which all birds are well separated.
2. If the green and red markers work well for distinguishing the two birds, keep them as separate classes, like "green bird" and "red bird". For another markers that didn't work well, you can have only one class, like "other bird", and annotate more frames. In my hand, for a video of 5 similar-looking, interacting flies (640X640, 10-minute duration, 25 fps), if I annotate ~100 frames in which collision / occlusion occurs, with only one class, "fly", and trained a Detector for this video at 640 inferencing frame size with 5,000 iterations, the identity switches are typically less than 5. If I annotate ~300 frames in which collision / occlusion occurs, the identity switch can be 0.

[prashers]

Thanks for the additional suggestions.

I took a step back to think about whether the time investment would be worth it for my project. I realized I am not sure if LabGym would actually give me the data I am hoping for. I am hoping to get information on the frequency of social interactions between particular birds to understand how their relationships change over time. For example, I want to know how many times blue pecked green in a trial (not just the number of times blue pecked any bird in general), or how many times yellow groomed red (not just the number of times yellow groomed any other bird).

Would LabGym give me information about the actor and receiver of each social interaction, or is it only able to tell me how many times the focal bird exhibited each behavior I define?

[yujiahu415]

LabGym currently can only tell you at a time point which bird "is pecking" (any other bird) and which bird "is being pecked" (by any other bird). So if there are multiple "pecking" and "being pecked" pairs occur at the same time point, you need additional posthoc analysis (e.g., calculate the distance between all possible "pecking" bird and "being pecked" bird and sort the shortest distance) to link the two birds that have the relationship.

[prashers]

I understand. After some more thought, I think it might be too difficult/time consuming to get LabGym working for my purposes. I really appreciate all of your advice! I am sure it could prove to be useful for myself or my colleagues in the future.

Out of curiosity, is there anything in development to associate each detected social behavior with a particular actor and receiver (especially when there may be multiple behaviors happening simultaneously)? I think that would be a great feature for people studying social behavior like myself!

[yujiahu415]

No problem! And thank you for your effort! We will implement new features to address the "multiple pairs of particular actor and receiver" problem in future versions. As I mentioned in previous responses, this is doable because LabGym can already detect "actor" and "receiver", we only need to introduce an additional step of sorting out the distance between all possible actors and receivers to match the particular pairs.