DXDetctor

Description

This program defines the function DebrisDetect(img_path, Params) which implements the debris detection algorithm as described in the following paper:

Detecting Debris in UAV Imagery of Disasters
Matt Hegarty, Khuoung Nguyen, Robin Murphy

There are two parts to this algorithm: an edge detector and a corner detector. If the edge detector detects anomalies in the image, then the function returns its results. If the edge detector doesn't find anything, the corner detector will take over and look for anomalies and return its results.

Input

The function DebrisDetect(img_path, Params) takes in img_path and Params.

img_path is the path to the image to be analyzed, or an OpenCV image object (Numpy matrix). Python OpenCV uses a Numpy matrix to store images.

Params contains the parameters from the parameters.ini file that are pertinent to the debris detection algorithm. These are as follows:

• LineGaussianIter
• LineDilationIter
• LineBilatBlurColor
• LineBilatBlurSpace
• LineCannyEdgeLowerBound
• LineCannyEdgeThreshold
• CornerGaussianIter
• CornerErosionIter
• CornerBilateralColor
• CornerBilateralSpace
• CornerMaxDistance
• CornerNumPoints

Output

This function will output the following in this order:

• The name of the original file
• A heatmap of the image with the anomalies drawn on it
• The time taken to analyze the image
• The stats of the analysis, which is a percentage of anomalous pixels

Imports

The function imports the following python libraries

• os
• sys
• cv2
• numpy
• scipy
• spectral
• distance from scipy.spatial
• math
• copy
• custom timer class

Global Variables

The following code block is all the global variables in the function

scale_value = 1
window_property = cv2.WINDOW_KEEPRATIO
window_init_width = 1600
window_init_height = 900

• scale_value is to ensure the image is not resized or scaled.
• window_property was used during testing when images were displayed using OpenCV's named windows to preserve image ratios.
• window_init_width was used during testing to adjust image width.
• window_init_height was used during testing to adjust image height.

Before Detectors are Called - Setup

Before the images are ran through the two detectors, there is pre-processing.

color = (0,180,0)

This color value is used to draw the lines representing edges that are found by the edge detector as well as connecting corners found by the corner detector into a polygon. This RGB value results in a neutral color value after applied on the color map in analyze.py. It results in a light blue so that anomalies found by this algorithm don't overpower the anomalies found by the RXDetector.

If you want to adjust the colors of anomalies produced by this algorithm, this value must be changed to a different RGB value. But take into consideration that it will be drawn on top of a color map.

The function then starts the timer and reads in the image provided by the argument img_path using OpenCV's imread function.

The image is then multiplied by 0 to zero out the image and is stored in the variable heatmap. This is what anomalies are drawn on and returned by the function so that analyze.py can combine the results of the two algorithms. Zeroing out is to ensure that only anomalies are returned by the function to the call in analyze.py.

The name of the image is then extracted from the argument img_path and stored in the variable result_name.

Then if needed, the image is scaled back to its original dimensions. At the moment, nothing changes the variable scale_value so this does nothing and can be ignored. That is, there is no alteration of the image after reading it in. This is performed more to cover bases in case of future works that would alter the variable scale_value.

And finally for pre-processing, the height and width is extracted from the image that was read in. These values are stored in height and width respectively. With these two values extracted, the variable avg_dim is calculated which is defined as two-thirds of the average dimensions of the image.

avg_dim = (((width+height)//2)//3)*2

This formula was obtained from the research paper and is used to determine if straight lines found are long enough to trigger edge detection.

Edge Detection

Edge detection starts off with applying an Gaussian Blur to the image. The ksize argument of the function

void cv::GaussianBlur (InputArray src,
			OutputArray dst,
			Size ksize,
			double sigmaX,
			double sigmaY = 0,
			int borderType = BORDER_DEFAULT 
			) 

is (5,5). The only parameter the user can change here is the Gaussian kernel standard deviation, under the parameter LineGaussianIter in the application and parameters file. This corresponds to sigmaX in the function above. This changes both sigmaX and sigmaY to the same value provided. By default it is 0. dst and borderType is omitted from the function call. The altered image is saved as the variable proc_img.

After the image has been applied a Gaussian Blur, a Kernel Dilation happens next. proc_img is passed into the function

void cv::dilate ( InputArray src,
		OutputArray dst,
		InputArray kernel,
		Point anchor = Point(-1,-1),
		int iterations = 1,
		int borderType = BORDER_CONSTANT,
		const Scalar & borderValue = morphologyDefaultBorderValue() 
		) 	

What we provide is the src, kernel, and iterations arguments. src corresponds to proc_img, kernel is again (5,5), and iterations is able to be changed by the user through LineDilationIter. The default value if not changed by the user is 1.

A Bilateral Blur is then performed on the same image after Gaussian and Dilation.

void cv::bilateralFilter ( InputArray src,
			OutputArray dst,
			int d,
			double sigmaColor,
			double sigmaSpace,
			int borderType = BORDER_DEFAULT 
			) 	

dst is again proc_img, d is 9 which corresponds to the diameter of each pixel neighborhood that is used during filtering, and sigmaColor and sigmaSpace are parameters that users can alter (LineBilatBlurColor and LineBilatBlurSpace). Our default provided values are 75 for both. The other arguments are not used in our function call.

After the image finishes going through the preprocessing steps, edge detection can begin.

void cv::Canny ( InputArray image,
	 	OutputArray edges,
	 	double threshold1,
	 	double threshold2,
	 	int apertureSize = 3,
	 	bool L2gradient = false 
 	 	) 	

Canny is called first and we provide it proc_img for image and two arguments: threshold1 and threshold2. These are both able to be changed by the user (LineCannyEdgeLowerBound and LineCannyEdgeThreshold) and the default values we provide are 100 and 140 respectively. The output of this function is stored in a variable dst.

cdst is no longer in use.

The final step for edge detection is Hough Lines.

void cv::HoughLines ( InputArray image,
		OutputArray lines,
		double rho,
		double theta,
		int threshold,
		double srn = 0,
		double stn = 0,
		double min_theta = 0,
		double max_theta = CV_PI 
		) 	

image is dst, rho is 0.7, theta is float(np.pi / 180.0), and threshold is half of avg_dim. The user cannot change any parameters for this function call. We decided to use half of avg_dim as the normal amount never produced any output for this detector. The output is an array of lines that is stored in the variable lines.

The find step is to check if lines is empty. If it is not, then we iterate through each line and draw each one on heatmap, which is the zeroed out image. The timer is stopped, scored are calculated, and the function returns.

If lines is empty, we move onto corner detection.

Corner Detection

A Gaussian Blur is applied again to the image read in originally. The only difference is the user can alter the parameter CornerGaussianIter for the corner detection portion of the algorithm.

Now instead of a Dilation, we perform an Erosion so that the image is sharper for corner detection.

void cv::erode ( InputArray src,
		OutputArray dst,
		InputArray kernel,
		Point anchor = Point(-1,-1),
		int iterations = 1,
		int borderType = BORDER_CONSTANT,
		const Scalar & borderValue = morphologyDefaultBorderValue() 
		) 	

This is identical to Dilation in the previous half of the algorithm. The user can change the CornerErosionIter parameter to alter iterations. The only difference is what the function does.

Another Bilateral Blur happens. The difference here is that it is stronger. So d is 16 here instead of 9 for the edge detection. The user can change CornerBilateralColor and CornerBilateralSpace for sigmaColor and sigmaSpace respectively. The default value for these are 200 and 500 respectively. You can notice the higher values for a stronger bilateral blur.

Now corner detection can begin. First the image is gray-scaled with cvtColor and the COLOR_BGR2GRAY color space. Corner detection is done through Shi-Tomasi, which in OpenCV is goodFeaturesToTrack.

void cv::goodFeaturesToTrack ( InputArray image,
				OutputArray corners,
				int maxCorners,
				double qualityLevel,
				double minDistance,
				InputArray mask = noArray(),
				int blockSize = 3,
				bool useHarrisDetector = false,
				double k = 0.04 
				) 	

image is the grey-scaled image gray. maxCorners the the maximum number of corners to detect and its value is 50. qualityLevel is 0.02. minDistance is 10. These values are not able to be changed by the user. The top 50 corners are then stored in corners.

For more processing, the original img is converted into the HSV color space. There are two colors that the algorithm looks to filter if a corner has all of its surrounding pixels these similar to these two colors: green and brown. This represents the grass and dirt in the surrounding areas of UAV imagery. NOTE: these values are optimized for the image data sets that were provided to us. In the future, this should be able to be changed dynamically to adapt to the image being processed. As is, the hard coded HSV values of the green and brown we use are as below.

lower_green = np.array([33, 87, 78])
upper_green = np.array([53, 107, 158])
lower_brown = np.array([167, 13, 186])
upper_brown = np.array([179, 33, 255])

There is an upper value and lower value for both colors so that there is a range a pixel can fall in between.

Now the algorithm goes through every corner in corners. To access a corner's surrounding pixels, there is bounds checking done. Then the HSV values of the corner is taken and compared to the HSV values of the green and brown above. If all of its surrounding pixels fall in between the values for green and brown, then it is discarded.

Then if the pixel has all of its surrounding pixels have low saturation and high values, then it is also discarded. For this we defined low saturation as <=20 and high value as >=220. NOTE: For HSV, Hue range is [0,179], Saturation range is [0,255] and Value range is [0,255]

The next step is removing corners that aren't close to any other corners. We utilized Euclidean distance between corners and removed any that has a Euclidean distance of greater than CornerMaxDistance. The user can alter this parameter and its default value is 75 distance, which works for our general data of 4k images.

For corners that are kept, they are stored in the list kept_corners.

The next step is to connect corners that are near each other to form polygons. First, connected_pairs is created, which is an list of list of tuples. At initialization, every kept corner is a tuple consisting of its (x,y) coordinates in the image in its own list. You will see below how this is utilized.

Now the algorithm will begin connecting corners. It goes through every pair of corners in kept_corners and if its distance is less than CornerMaxDistance, they are connected. For connection, it goes through connected_pairs in order and checks if the pair is already in the current list. If it is, then it breaks since there is already a connection between the pair. If one is in and the other isn't, it appends to the list the missing corner. If they're not in the list at all, it will continue to the next list in connected_pairs. This ensures that each list in connected_pairs represents a polygon of connected corners. Each list initially was a corner by itself but at the end of execution, some of the lists will represent all the connections possible for the corners in that list. NOTE: although two lists in connected_pairs should be the same if they start off with corners that belong in a polygon, there will be only one full list in connected_pairs for each full polygon since the algorithm breaks computation when it appends the appropriate corners. This saves computational time as at the end, the biggest lists from connected_pairs are taken as the polygons.

To meet the criteria of anomalous, a polygon after connection will need to contain more than CornerNumPoints corners. This can be altered by the user and is by default 3. So a polygon will need to have at least 4 corners to trigger detection. connected_pairs is gone through and any lists with a size greater than three meets the criteria. If there is at least one, then the algorithm detects an anomaly. It will then draw lines between each corner in the polygon onto heatmap, the zeroed out image. Scores are then calculated and the function returns.

If there are no lists in connected_pairs that more than CornerNumPoints corners, the function returns non-anomalous.