I’m doing image annotation recently, and the following picture will appear, and the borders need to be removed.
1. Idea
- Manually mark the range P of the picture frame, and use the marking tool to draw a point A on the picture frame.
- Get the coordinates and color of point A. Within the range P, replace the color of each point x that is similar to the color of point A with the average color of several points within the range above and below (or left and right) of point x. But for the edge part, there may be problems as shown in the figure below:
- Perform additional processing on the edge part Q of the border. The color of the border may be quite different from the color of point A, so other methods need to be used for processing. I use the outlier detection algorithm because some residual points in Q that were not removed in step 2 will be quite different from the original image. Combining the frequency information can filter out these points and get the following effect:
2. Code implementation
- Guide package and necessary parameters and models:
from PIL import Image import numpy as np from sklearn.ensemble import IsolationForest from collections import Counter # Used to determine rgb similarity, it can be smaller threshold=15 threshold_border=15 # The sampling radius can be set smaller and can be set according to the thickness of the border. redius=20 redius_border=20 # Parameters that need to be adjusted to process the border step_add=37 step_max=2000 step=0 all_outliers=[] top_outlier_num=10 #Create Isolation Forest model model = IsolationForest(contamination=0.05) # Set the proportion of outliers
- Helper function
#Determine whether the colors are similar
def my_similar(a,b):
if sum([abs(x-y) for (x,y) in zip(list(a),list(b))])/3<threhold:
return True
return False
# Determine whether the colors are similar
def my_similar_border(a,b_s):
for b in b_s:
if sum([abs(x-y) for (x,y) in zip(list(a),list(b)[0])])/3<threhold_border:
return True
return False
# Get the outlier value, used to process the edge Q of the border
def get_outlier(tmp,model):
normal_point=[]
unnormal_point=[]
# Pass data to the model for training
model.fit(tmp)
# Get the prediction label of each data point, -1 represents an abnormal point, 1 represents a normal point
labels = model.predict(tmp)
# Print labels for each data point
for i, label in enumerate(labels):
if label == -1:
unnormal_point.append(tmp[i])
if label==1:
normal_point.append(tmp[i])
return normal_point,unnormal_point
# Get neighbor nodes in the horizontal or disposition direction for color replacement
def get_right_neighborhood(target_color,neighborhood_x,neighborhood_y):
diff_x=0
diff_y=0
for neighborhood in neighborhood_x:
diff_x + =abs(sum([x-y for x,y in zip(list(neighborhood),list(target_color))])/3)
for neighborhood in neighborhood_y:
diff_y + =abs(sum([x-y for x,y in zip(list(neighborhood),list(target_color))])/3)
if diff_x>diff_y:
return neighborhood_x,1
return neighborhood_y,2
- Main function and necessary preparation data
############################################## ########################################
############################Start manually specifying area P and point A############ ####################
###############################Corresponds to "Ideas" Part 1########### ##########################
################################################ #####################################
# Use manual or labeling tools such as labelimg,
# Determine area "P" in part 1 of "Idea"
# Manually select the approximate position of the frame, which needs to be wrapped
x_1=1427
y_1=723
x_2=2061
y_2=1363
# Use manual or labeling tools such as labelimg,
# Determine point "A" in part 1 of "Idea"
x = 1495 #x coordinate of the target pixel
y = 1294 #y coordinate of the target pixel
# Main function
def replace_color_around_point(image_path, x, y, radius=redius):
# Number of sampling steps
step=0
# Open image
image = Image.open(image_path)
pixels = image.load()
# Get the width and height of the image
width, height = image.size
################################################ #####################################
############################Start processing the internal nodes of the border############## ##################
################################Corresponds to Part 2 of "Ideas"########### ##########################
################################################ #####################################
# Get the color of the target pixel
target_color = pixels[x, y]
# Loop through each pixel in the image
for i in range(x_1,x_2):
for j in range(y_1,y_2):
# If the pixel color is the same as the target color, replace it with the average color of the surrounding area
if my_similar(pixels[i, j],target_color):
# Calculate the color average of the surrounding horizontal and vertical areas
neighborhood_x = []
neighborhood_y=[]
for m in range(i - redius, i + radius + 1):
if 0 <= m < width:
neighborhood_x.append(pixels[m, j])
for n in range(j - redius, j + radius + 1):
if 0 <= n < height:
neighborhood_y.append(pixels[i, n])
neighborhood,direction=get_right_neighborhood(target_color,neighborhood_x,neighborhood_y)
neighborhood=[n for n in neighborhood if not my_similar(n,target_color)]
neighborhood = np.array(neighborhood)
average_color=tuple(np.mean(neighborhood, axis=0, dtype=int))
average_color_part_1 = tuple(np.mean(neighborhood[0:int(len(neighborhood)/2)], axis=0, dtype=int))
average_color_part_2 = tuple(np.mean(neighborhood[int(len(neighborhood)/2) + 1:], axis=0, dtype=int))
# Replace pixel color
pixels[i, j] = average_color
# If it is horizontal direction
if direction==1:
for m in range(i - int(redius_border/3), i):
if 0 <= m < width:
pixels[m,j]=average_color_part_1
for m in range(i, i + int(redius_border/3) + 1):
if 0 <= m < width:
pixels[m,j]=average_color_part_2
# If it is vertical direction
if direction==2:
for n in range(j - int(redius_border/3), j):
if 0 <= n < height:
pixels[i,n]=average_color_part_1
for n in range(j ,j + int(redius_border/3) + 1):
if 0 <= n < height:
pixels[i,n]=average_color_part_2
step + =1
if step%step_add==0 and step<step_max:
normal_point,unnormal_point=get_outlier(neighborhood,model)
unnormal_point=[tuple(x) for x in unnormal_point]
all_outliers.extend(unnormal_point)
################################################ #####################################
############################Start processing the edge nodes of the border############## ##################
###############################corresponds to "Ideas" Part 2########### ##########################
################################################ #####################################
# Use Counter to count the number of occurrences of triples
all_outlier_counts = Counter(all_outliers)
# Get the ten triplets that appear most often and use them to remove borders
top_outliers = all_outlier_counts.most_common(top_outlier_num)
# Loop through each pixel in the image
for i in range(x_1,x_2):
for j in range(y_1,y_2):
# If the color of this point is similar to that of the abnormal point, replace it
if my_similar_border(pixels[i, j] ,top_outliers):
# Calculate the color average of the surrounding horizontal and vertical areas
neighborhood_x = []
neighborhood_y=[]
for m in range(i - int(radius/3), i + int(radius/3) + 1):
if 0 <= m < width:
neighborhood_x.append(pixels[m, j])
for n in range(j - int(radius/3), j + int(radius/3) + 1):
if 0 <= n < height:
neighborhood_y.append(pixels[i, n])
neighborhood,direction=get_right_neighborhood(target_color,neighborhood_x,neighborhood_y)
neighborhood=[n for n in neighborhood if not n in top_outliers]
neighborhood = np.array(neighborhood)
average_color=tuple(np.mean(neighborhood, axis=0, dtype=int))
# Replace pixel color
pixels[i, j] = average_color
# Save the modified image
image.save(r"../data/bbb.jpg")
- transfer
image_path = r"../data/aaa.jpg" # Replace with your input image path replace_color_around_point(image_path, x, y)
3.Existing problems
- When doing color replacement, only the average is used (average_color related content in the code), maybe other linear interpolation algorithms can be used.
- The parameters need to be carefully adjusted, otherwise the image in the frame may have the following “burr phenomenon”, and the “frame” cannot be completely removed:
After adjusting the threshold parameter to a smaller value, the glitches disappear. Other parameters can also be adjusted.
