Please don’t question the fact that I’ve been writing articles because my computer has been formatted and my needs have changed. This must be more in line with the times!
To lay the foundation for my GRCNN crawling, I used to run it on Ubuntu: [Robotic arm visual grabbing from theory to practice], yes, now I am running it on WIN11, there will be corresponding demonstration videos later Oh
1. Download miniconda
Official website address: https://docs.conda.io/projects/miniconda/en/latest/
Click Miniconda3 Windows 64-bit to download
If you want to experience the full functionality, you can download the full version: https://www.anaconda.com/download
2. Install miniconda
Run as administrator
Click next
Click I agree
Click next
Select the appropriate installation path and click Next
Click to select all. The second item must be checked. Here is to add environment variables to facilitate Vscode to find it later. Click to install.
Click Finish
Select application in the menu, search for miniconda, and open the miniconda terminal
# Check which virtual environments there are conda env list # Check which packages a certain virtual environment has conda list
It is worth noting. If you use the conda environment to configure subsequent environments, you need to pay attention to the corresponding relationship between the python version and the versions of Pytorch, Tensorflow, etc.! The following installation and configuration are based on the system environment, not the conda environment.
3. miniconda source change
Replace conda with the domestic Tsinghua image source in the windows environment
or
step1 Enter the following command under Anaconda Prompt to generate the .condarc file
conda config --set show_channel_urls yes
step2 Find the .condarc file. Generally, the file is in the directory C:\Users\username.
step3 Open .condarc with Notepad and modify the content as follows:
channels: -defaults show_channel_urls: true default_channels: - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 custom_channels: conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud deepmodeling: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/
step4 Run conda clean -i to clear the index cache and ensure that the index provided by the mirror station is used.
conda clean -i
step5 Enter the following command to display the conda configuration information. The source change is successful! !
conda config --show
4. Install pytorch
Enter the following command.
nvidia-smi
As shown in the infographic below, you can see that the driver version is 528.02; the highest supported CUDA version is version 12.0. After getting the highest supported CUDA version of the graphics card, we can install the environment based on this information.
You need to choose the appropriate version according to your own development environment, please refer to: https://github.com/pytorch/vision
The following is the corresponding torchvision versions and supported Python
versions.
torch |
torchvision |
Python |
|---|---|---|
main / nightly |
main / nightly |
>=3.8, <=3.11 |
2.1 |
0.16 |
>=3.8, <=3.11 |
2.0 |
0.15 |
>=3.8, <=3.11 |
1.13 |
0.14 |
>=3.7.2, <=3.10 |
older versions
torch |
torchvision |
Python |
|---|---|---|
1.12 |
0.13 |
>=3.7 , <=3.10 |
1.11 |
0.12 |
>=3.7, <=3.10 |
1.10 |
0.11 |
>=3.6, <=3.9 |
1.9 |
0.10 |
>=3.6, <=3.9 |
1.8 |
0.9 |
>=3.6, <=3.9 |
1.7 |
0.8 |
>=3.6, <=3.9 |
1.6 |
0.7 |
>=3.6, <= 3.8 |
1.5 |
0.6 |
>=3.5, <=3.8 |
1.4 |
0.5 |
==2.7, >=3.5, <=3.8 |
1.3 |
0.4.2 / 0.4.3 |
==2.7, >=3.5, <=3.7 |
1.2 |
0.4.1 |
==2.7, >=3.5, <=3.7 |
1.1 |
0.3 |
==2.7, >=3.5, <=3.7 |
<=1.0 |
0.2 |
==2.7, > =3.5, <=3.7 |
Considering that yolov8 is needed for later development, we created a python3.8.10 virtual environment `torch`= `1.12`, `torchvision` =`0.13`
# Create a new environment conda create -n mytorch python==3.8.10 #Activate environment conda activate mytorch # Delete environment conda remove -n mytorch --all #Exit the current environment conda deactivate
Enter y
Enter the mytorch environment
# Activate environment conda activate mytorch
According to the one-click installation provided by the official website
#3. Install cuda. Note that 30 series requires cudatoolkit11 or above. #CUDA 10.2 conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=10.2 -c pytorch #CUDA 11.3 conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch #CUDA 11.6 conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.6 -c pytorch -c conda-forge # CPU Only conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cpuonly -c pytorch
5. Test whether the installation is successful
After activating the environment in the terminal, enter python and enter the following commands:
import torch import torchvision # This command displays the pytorch version print(torch.__version__) # If cuda is installed, true will be displayed torch.cuda.is_available()
return
Sometimes you can use pip to temporarily change the image source.
When using the pip command to install packages in China, sometimes the installation speed is too slow due to foreign servers. If you use domestic mirror sources to install packages, the speed will be very fast. The following are domestic mirror sources:
Tsinghua: https://pypi.tuna.tsinghua.edu.cn/simple
Alibaba Cloud: http://mirrors.aliyun.com/pypi/simple/
Douban: http://pypi.douban.com/simple/
Just add the -i parameter after pip and the above mirror source. The example is as follows:
pip install requests -i http://mirrors.aliyun.com/pypi/simple/
6. Question:
If anaconda cannot be used, you can consider adding environment variables
illustrate
After installing Anaconda normally on the Win11 system, in the cmd command line window:
Set environment variables
1. This computer-"Properties-"Advanced system settings-"Environment variables
2. Find Path in the system variables and add the following two variables to Path
3. Test
At this point, OK! ! !
7. Summary
No matter how the environment is updated, as long as you grasp its essence, everything will take care of itself.
import os
import time
import matplotlib.pyplot as plt
import numpy as np
import torch
from UR_Robot_real import UR_Robot_real
from inference.post_process import post_process_output
from utils.data.camera_data import CameraData
from utils.dataset_processing.grasp import detect_grasps
from utils.visualisation.plot import plot_grasp
import cv2
from PIL import Image
import torchvision.transforms as transforms
importyaml
import pyrealsense2 as rs
from ultralytics.yolo.utils.torch_utils import select_device, time_sync
# from utils.general import (
# check_img_size, non_max_suppression, apply_classifier, scale_coords,
# xyxy2xywh, strip_optimizer, set_logging)
from ultralytics.yolo.utils.checks import check_imgsz
from ultralytics.yolo.utils.ops import non_max_suppression, scale_coords
from ultralytics.yolo.utils import LOGGER
from ultralytics import YOLO
from ultralytics.yolo.data.dataloaders.v5loader import LoadStreams, LoadImages, letterbox
# from models.experimental import attempt_load,
from ultralytics.nn.tasks import attempt_load_weights, attempt_load_one_weight
from ultralytics import YOLO
tool_xyz = np.asarray([-0.1, 0.25, 0.34])
tool_orientation = np.asarray([-np.pi,0,0])
hole_xyz = np.asarray([-0.105, -0.305, 0.345])
hole_orientation = np.asarray([np.pi/2,np.pi,0])
camera_target_position = np.asarray([[0],[0],[0],[1]])
target_position = np.asarray([-0.105, -0.305, 0.345])
image_wide_size = 640
image_high_size = 480
#PyTorch
#YoloV5-PyTorch
BH_yaml_path = 'weights/yolov8n.yaml'
with open(BH_yaml_path, 'r', encoding='utf-8') as f:
yolov8 = yaml.load(f.read(), Loader=yaml.SafeLoader)
color_dict = {<!-- -->} # Create an empty dictionary to store the correspondence between numbers and colors
# Generate colors for each category
for class_id in range(yolov8['nc']):
color = [np.random.randint(0, 255) for _ in range(3)] # Generate a random color
color_dict[class_id] = color # Add numbers and colors to the dictionary
class PlaneGraspClass:
def __init__(self, saved_model_path=None,use_cuda=True,visualize=False,include_rgb=True,include_depth=True,output_size=300):
if saved_model_path==None:
saved_model_path = 'logs\230802_1421_training_jacquard\epoch_30_iou_1.00'
# saved_model_path = 'trained-models/jacquard-rgbd-grconvnet3-drop0-ch32/epoch_48_iou_0.93'
self.model = torch.load(saved_model_path)
# Path to YOLOV5 model configuration file (YAML format) yolov5_yaml_path
self.model1 = YOLO("weights/best1.pt")
# model = YoloV8(yolov8_yaml_path='ultralytics/models/v8/yolov8.yaml')
print("[INFO] YoloV8 model loading completed")
self.device = "cuda:0" if use_cuda else "cpu"
self.visualize = visualize
self.cam_data = CameraData(include_rgb=include_rgb,include_depth=include_depth,output_size=output_size)
# Load camera pose and depth scale (from running calibration)
self.ur_robot = UR_Robot_real(tcp_host_ip="192.168.56.10", tcp_port=30003, workspace_limits=None, is_use_robotiq85=True,
is_use_camera=True)
self.cam_pose = self.ur_robot.cam_pose
self.cam_depth_scale = self.ur_robot.cam_depth_scale
self.intrinsic = self.ur_robot.cam_intrinsics
if self.visualize:
self.fig = plt.figure(figsize=(6, 6))
else:
self.fig = None
def get_aligned_images(self):
# frames = pipeline.wait_for_frames() # Wait to get image frames
# aligned_frames = align.process(frames) # Get aligned frames
# aligned_depth_frame = aligned_frames.get_depth_frame() # Get the depth frame in the aligned frame
# color_frame = aligned_frames.get_color_frame() # Get the color frame in the aligned frame
aligned_depth_frame, color_frame= self.ur_robot.get_camera_data1() # meter
############### Obtaining camera parameters #######################
intr = color_frame.profile.as_video_stream_profile().intrinsics # Get camera intrinsic parameters
depth_intrin = aligned_depth_frame.profile.as_video_stream_profile(
).intrinsics # Get depth parameters (used to convert pixel coordinate system to camera coordinate system)
'''camera_parameters = {'fx': intr.fx, 'fy': intr.fy,
'ppx': intr.ppx, 'ppy': intr.ppy,
'height': intr.height, 'width': intr.width,
'depth_scale': profile.get_device().first_depth_sensor().get_depth_scale()
}'''
# Save internal parameters to local
# with open('./intrinsics.json', 'w') as fp:
# json.dump(camera_parameters, fp)
################################################ #####
depth_image = np.asanyarray(aligned_depth_frame.get_data()) # Depth image (default 16 bits)
depth_image_8bit = cv2.convertScaleAbs(depth_image, alpha=0.03) # Depth image (8 bits)
depth_image_3d = np.dstack(
(depth_image_8bit, depth_image_8bit, depth_image_8bit)) # 3-channel depth map
color_image = np.asanyarray(color_frame.get_data()) # RGB image
# Return camera internal parameters, depth parameters, color images, depth images, and depth frames in all frames
return intr, depth_intrin, color_image, depth_image, aligned_depth_frame
def camera_xyz_list_function(self):
# Wait for a coherent pair of frames: depth and color
intr, depth_intrin, color_image, depth_image, aligned_depth_frame = self.get_aligned_images() # Get aligned images and camera internal parameters
time.sleep(4)
intr, depth_intrin, color_image, depth_image, aligned_depth_frame = self.get_aligned_images() # Get aligned images and camera internal parameters
# Convert images to numpy arrays
# Apply colormap on depth image (image must be converted to 8-bit per pixel first)
depth_colormap = cv2.applyColorMap(cv2.convertScaleAbs(
depth_image, alpha=0.03), cv2.COLORMAP_JET)
# Stack both images horizontally
images = np.hstack((color_image, depth_colormap))
# Show images
camera_xyz_list = []
t_start = time.time() # Start timing
#YoloV8 Object Detection
results = self.model1.predict(color_image)
xyxy_list = results[0].boxes.xyxy
conf_list = results[0].boxes.conf
class_id_list = results[0].boxes.cls
canvas = np.copy(color_image)
t_end = time.time() # End timing
for i in range(len(xyxy_list)):
if conf_list[i] > 0.6:
x_min = int(xyxy_list[i][0])
y_min = int(xyxy_list[i][1])
x_max = int(xyxy_list[i][2])
y_max = int(xyxy_list[i][3])
if x_min < image_wide_size and y_min < image_high_size and x_max < image_wide_size and y_max < image_high_size:
dis_min = aligned_depth_frame.get_distance(x_min, y_min)
dis_max = aligned_depth_frame.get_distance(x_max, y_max)
dis = (dis_min + dis_max) / 2
ux = int((x_min + x_max) / 2)
uy = int((y_min + y_max) / 2)
camera_xyz = rs.rs2_deproject_pixel_to_point(
depth_intrin, (ux, uy), dis) # Calculate the xyz of the camera coordinate system
camera_xyz = np.round(np.array(camera_xyz), 3) # Convert to 3 decimal places
camera_xyz = camera_xyz.tolist()
camera_xyz_list.append(camera_xyz)
num_cls = int(class_id_list[i].item())
label = '%s ' % (yolov8['class_name'][num_cls])
color = color_dict[num_cls]
cv2.putText(canvas, label + str(round(conf_list[i].item(), 2)),
(int(xyxy_list[i][0]), int(xyxy_list[i][1])), cv2.FONT_HERSHEY_SIMPLEX, 1,
(255, 255, 255),
2)
cv2.rectangle(canvas, (int(xyxy_list[i][0]), int(xyxy_list[i][1])),
(int(xyxy_list[i][2]), int(xyxy_list[i][3])), color, 2)
cv2.circle(canvas, (ux, uy), 4, (255, 255, 255), 5) # Mark the center point
cv2.putText(canvas, str(camera_xyz), (ux + 20, uy + 10), 0, 1,
[225, 255, 255], thickness=2, lineType=cv2.LINE_AA) # Mark the coordinates
fps = int(1.0 / (t_end - t_start))
cv2.putText(canvas, text="FPS: {}".format(fps), org=(50, 50),
fontFace=cv2.FONT_HERSHEY_SIMPLEX, fontScale=1, thickness=2,
lineType=cv2.LINE_AA, color=(0, 0, 0))
cv2.namedWindow('detection', flags=cv2.WINDOW_NORMAL |
cv2.WINDOW_KEEPRATIO | cv2.WINDOW_GUI_EXPANDED)
cv2.imshow('detection', canvas)
return camera_xyz_list, t_end, t_start, canvas
def generate(self):
target1_position= [0,0,0]
destination1=np.append(hole_xyz,hole_orientation)
self.ur_robot.move_j_p(destination1)
xyz_list, t_end, t_start, canvas = self.camera_xyz_list_function()
xyz_list = np.asarray(xyz_list)
index=xyz_list.shape[0]
print(xyz_list)
# target1_position[0] = hole_xyz[0] + xyz_list[0,0]-0.043
# target1_position[1] = hole_xyz[1]- xyz_list[0,2] + 0.28
# target1_position[2] = hole_xyz[2] + xyz_list[0,1] + 0.075
# target1_position= np.asarray(target1_position)
# target1_position = np.append(target1_position,hole_orientation)
# print(target1_position)
# self.ur_robot.move_j_p(target1_position)
for i in range(index):
# time.sleep(1.5)
# # Add fps display
self.ur_robot.move_j([-(0 / 360.0) * 2 * np.pi, -(90 / 360.0) * 2 * np.pi,
(0 / 360.0) * 2 * np.pi, -(90 / 360.0) * 2 * np.pi,
-(0 / 360.0) * 2 * np.pi, 0.0])# return home
# time.sleep(1.5)
## if you want to use RGBD from camera,use me
# Get RGB-D image from camera
grasp_home = np.append(tool_xyz,tool_orientation)
self.ur_robot.move_j_p(grasp_home)
time.sleep(1.5)
rgb, depth= self.ur_robot.get_camera_data() # meter
depth = depth *self.cam_depth_scale
depth[depth >1]=0 # distance > 1.2m,remove it
## if you don't have realsense camera, use me
# num =2 # change me num=[1:6],and you can see the result in '/result' file
# rgb_path = f"./cmp{num}.png"
# depth_path = f"./hmp{num}.png"
# rgb = np.array(Image.open(rgb_path))
# depth = np.array(Image.open(depth_path)).astype(np.float32)
# depth = depth * self.cam_depth_scale
# depth[depth > 1.2] = 0 # distance > 1.2m,remove it
depth= np.expand_dims(depth, axis=2)
x, depth_img, rgb_img = self.cam_data.get_data(rgb=rgb, depth=depth)
time.sleep(1.5)
x, depth_img, rgb_img = self.cam_data.get_data(rgb=rgb, depth=depth)
rgb = cv2.cvtColor(rgb,cv2.COLOR_BGR2RGB)
with torch.no_grad():
xc = x.to(self.device)
pred = self.model.predict(xc)
q_img, ang_img, width_img = post_process_output(pred['pos'], pred['cos'], pred['sin'], pred['width'])
grasps = detect_grasps(q_img, ang_img, width_img)
# np.save(self.grasp_pose, grasp_pose)
if self.visualize:
plot_grasp(fig=self.fig, rgb_img=self.cam_data.get_rgb(rgb, False), grasps=grasps, save=True)
if len(grasps) ==0:
print("Detect 0 grasp pose!")
if self.visualize:
plot_grasp(fig=self.fig, rgb_img=self.cam_data.get_rgb(rgb, False), grasps=grasps, save=True)
return False
## For real UR robot
# Get grasp position from model output
pos_z = depth[grasps[0].center[0] + self.cam_data.top_left[0], grasps[0].center[1] + self.cam_data.top_left[1]]
pos_x = np.multiply(grasps[0].center[1] + self.cam_data.top_left[1] - self.intrinsic[0][2],
pos_z / self.intrinsic[0][0])
pos_y = np.multiply(grasps[0].center[0] + self.cam_data.top_left[0] - self.intrinsic[1][2],
pos_z / self.intrinsic[1][1])
if pos_z == 0:
return False
target = np.asarray([pos_x, pos_y, pos_z])
target.shape = (3, 1)
# Convert camera to robot coordinates
camera2tool = self.cam_pose
target_position = self.ur_robot.target1_position(camera2tool, tool_orientation, tool_xyz, target)
# print(target_position)
# Convert camera to robot angle
angle = np.asarray([0, 0, grasps[0].angle])
angle.shape = (3, 1)
target_angle = self.ur_robot.target1_angle(camera2tool, tool_orientation, angle)
angle.shape = (1,3)
# print(target_angle)
# target_angle = np.dot(camera2robot[0:3, 0:3], angle)
# compute gripper width
width = grasps[0].length # mm
if width < 25: # detect error
width=70
elif width <40:
width =45
elif width > 85:
width=85
# Concatenate grasp pose with grasp angle
grasp_pose = np.append(target_position, target_angle[2])
print('grasp_pose: ', grasp_pose)
print('grasp_width: ',grasps[0].length)
destination=np.append([grasp_pose[0],grasp_pose[1],grasp_pose[2]],tool_orientation)
print(destination)
# self.ur_robot.move_j_p(destination)
# hole targrt destination
target1_position[0] = hole_xyz[0] + xyz_list[i,0]-0.037
target1_position[1] = hole_xyz[1]- xyz_list[i,2] + 0.18
target1_position[2] = hole_xyz[2] + xyz_list[i,1] + 0.066
target1_position= np.asarray(target1_position)
target1_position = np.append(target1_position,hole_orientation)
print(target1_position)
# self.ur_robot.move_j_p(target1_position)
# target_position[0] = hole_xyz[0] + xyz_list[i,0]
# target_position[1] = hole_xyz[1]- xyz_list[i,2] + 0.25
# target_position[2] = hole_xyz[2] + xyz_list[i,1]
# target_position = np.append(target_position,hole_orientation)
# self.ur_robot.move_j_p(target_position)
success = self.ur_robot.plane_grasp_hole([grasp_pose[0],grasp_pose[1],grasp_pose[2]],target1_position, yaw=grasp_pose[3], open_size=width/100)
if success==True:
print("success:",i + 1)
elif success==False:
print("unsuccess")
break
print("Grasp and full success:",i + 1)
self.ur_robot.move_j([-(0 / 360.0) * 2 * np.pi, -(90 / 360.0) * 2 * np.pi,
(0 / 360.0) * 2 * np.pi, -(90 / 360.0) * 2 * np.pi,
-(0 / 360.0) * 2 * np.pi, 0.0])# return home
## For having not real robot
# if self.visualize:
# plot_grasp(fig=self.fig, rgb_img=self.cam_data.get_rgb(rgb, False), grasps=grasps, save=True)
# return True
if __name__ == '__main__':
g = PlaneGraspClass(
# saved_model_path='/home/robot/UR_grasping_net/robotic-grasping-explosives/logs/230801_0934_training_jacquard/epoch_34_iou_1.00',
# saved_model_path='/home/robot/UR_grasping_net/robotic-grasping-explosives/logs/230801_2225_training_jacquard/epoch_44_iou_1.00',
saved_model_path = 'logs/230802_0918_training_jacquard/epoch_31_iou_1.00',
# saved_model_path='/home/robot/UR_grasping_net/robotic-grasping-explosives/logs/230802_1421_training_jacquard/epoch_20_iou_1.00',
visualize=True,
include_rgb=True
)
g.generate()