Video and picture restoration technology is a challenging AI vision task that involves filling in missing or damaged areas in a video or picture sequence while maintaining spatial and temporal coherence. This technology is widely used in video completion, object removal, video recovery and other fields. In recent years, two prominent solutions have emerged in video restoration: flow-based propagation and spatiotemporal Transformers. Although both solutions are good, they also have some limitations, such as spatial misalignment, limited time horizon and exorbitant cost.
To put it bluntly, you use AI technology to remove watermarks or repair an unclear video, but the result cannot guarantee coherence. People can tell at a glance that the video or picture is still missing. At the same time, too high computing power The cost is also unbearable for ordinary people.
This time, we solve the video watermarking task through the ProPainter framework, which introduces a new method called dual-domain propagation and an efficient mask-guided video Transformer. These components jointly enhance the performance of video restoration while maintaining computational efficiency and lower costs, allowing ordinary people to complete complex watermark removal tasks. As the saying goes: clear water brings out hibiscus, and natural carvings are removed.
Install and configure ProPainter
The old rule is to clone the project first:
git clone https://github.com/sczhou/ProPainter.git
This project is based on the CUDA framework, please ensure that the CUDA version of the local environment is greater than 9.2.
Execute the command to view the local CUDA version:
nvcc --version
Output:
PS C:\Users\zcxey> nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Tue_Mar__8_18:36:24_Pacific_Standard_Time_2022 Cuda compilation tools, release 11.6, V11.6.124 Build cuda_11.6.r11.6/compiler.31057947_0
As of the publication of this article, the author’s version is 11.6. Regarding the configuration of CUDA and cudnn on this machine, please move to: The sound is good, the appearance is good, and the artificial intelligence AI speech model is equipped with dynamic pictures (Python3.10) based on PaddleGAN. Due to space limitations, we will not go into details here.
Then enter the project:
cd ProPainter
Install dependencies:
pip3 install -r requirements.txt
Then download ProPainter’s pre-trained model: https://github.com/sczhou/ProPainter/releases/tag/v0.1.0
Put it into the weights directory of the project. The directory structure after the model is placed is as follows:
weights |- ProPainter.pth |- recurrent_flow_completion.pth |- raft-things.pth |- i3d_rgb_imagenet.pt (for evaluating VFID metric) |- README.md
At this point, ProPainter is configured.
Object removal
ProPainter has thoughtfully put some examples into the project. We run the command directly in the root directory of the project:
python3 inference_propainter.py
Program output:
E:\work\ProPainter>python inference_propainter.py Pretrained flow completion model has loaded... Pretrained ProPainter has loaded... Network [InpaintGenerator] was created. Total number of parameters: 39.4 million. To see the architecture, do print(network). Processing: bmx-trees [80 frames]... 100%| 16/16 [00:10<00:00, 1.52it /s] All results are saved in results\bmx-trees
ProPainter will automatically demonstrate the object removal function of an 80-frame video and output it in the results folder of the project:
As you can see, the script removed the child riding a bicycle and the bicycle from the screen.
The specific operation is to put the object mask and the original picture to be removed into the inputs folder of the project, and then the pre-trained model will complete the removal and completion actions based on the mask.
Generate mask
In order to prevent abuse by criminals, the project author removed the watermark example. Now we will demonstrate how to remove the watermark. First, I have a video or picture with a watermark:
It can be seen that the watermark is very huge, covering part of the sofa, table and bed in the original picture. So in the first step we need to generate a mask of the watermark so that the program can easily identify the outline of the watermark.
First install the Open-cv library:
pip3 install opencv-python
Then write code to extract the logo and generate a mask:
import cv2
import numpy as np
room = cv2.imread('D:/Downloads/room.png' )
logo = cv2.imread('D:/Downloads/logo.png' )
#--- Resizing the logo to the shape of room image ---
logo = cv2.resize(logo, (room.shape[1], room.shape[0]))
#--- Apply Otsu threshold to blue channel of the logo image ---
ret, logo_mask = cv2.threshold(logo[:,:,0], 0, 255, cv2.THRESH_BINARY|cv2.THRESH_OTSU)
cv2.imshow('logo_mask', logo_mask)
cv2.waitKey()
cv2.imwrite('D:/Downloads/logo_mask.png', logo_mask)
running result:
Of course, if you don’t want to do it through code, you can also do it through Photoshop, directly through Photoshop’s content selection-“Reverse selection-“Fill black-“and then reverse selection again-“Fill white, to complete:
The final effect is the same as that of Open-cv.
Remove watermark
In this way, we get the original picture and the watermark mask, create the test directory in the inputs directory of the project, then create the img and mask directories, and put the original picture and watermark mask into the directories respectively:
├─inputs │ ├─test │ │ ├─img │ │ └─mask
Note that since this project is based on video, it must have at least two frames. If there is only one frame, an error will be reported.
Run command:
python3 inference_propainter.py --video inputs/test/img --mask inputs/test/mask
The program returns:
E:\work\ProPainter>python inference_propainter.py --video inputs/test/img --mask inputs/test/mask Pretrained flow completion model has loaded... Pretrained ProPainter has loaded... Network [InpaintGenerator] was created. Total number of parameters: 39.4 million. To see the architecture, do print(network). Processing: img [2 frames]... 100%| 1/1 [00:54<00:00, 54.30s/it] IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1227, 697) to (1232, 704) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). [swscaler @ 0000025d0a1b5900] Warning: data is not aligned! This can lead to a speed loss IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1227, 697) to (1232, 704) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility). [swscaler @ 000001b30eb858c0] Warning: data is not aligned! This can lead to a speed loss All results are saved in results\img
As you can see, the program outputs the processed two-frame video results to the results/img directory of the project. The results after removing the watermark:
The removal effect is amazing.
Of course, we only processed two frames of the video. A video of about 10 minutes usually requires a lot of GPU memory. By inputting the following parameters, you can effectively solve the local “explosion of video memory” error:
Reduce the number of local lengths by reducing --neighbor_length (default 10). Reduce the number of global reference frames by increasing --ref_stride (default 10). Resize the processed video by setting --resize_ratio (default is 1.0). Set a smaller video size by specifying --width and --height. Set --fp16 to use fp16 (half precision) during inference. By reducing the number of frames of the sub-video - subvideo_length (default is 80), the GPU memory cost and video length are effectively separated.
Conclusion
ProPainter is undoubtedly a great project, but it should be noted that removing watermarks may involve copyright infringement or breach of contract terms. Whether it is illegal depends on the laws and regulations of your country or region and the provisions of the relevant contract.
In many cases, watermarks are a form of copyright protection used to identify ownership or authorization of a work. If you remove the watermark without authorization, you may infringe the copyright rights of the original author, which may violate copyright law.
In addition, if you agree to the relevant terms of use and privacy policy when using a service or software, those terms and policies will usually provide that you may not remove or modify any watermark or copyright information. Breach of these contractual terms may result in legal liability.
Therefore, it is recommended that where watermarks are involved, you should comply with applicable laws, regulations and contractual terms, and respect the copyright and intellectual property rights of the original work.