1. Introduction
The processing and understanding of 3D data has become increasingly important in recent years with rapid developments in areas such as autonomous driving, robotics, and augmented reality. For this, effective annotation of 3D data is a prerequisite. In this post, we’ll explore how to build a 3D Bounding Box Annotation Tool (3D-BAT) from scratch using JavaScript.
2. Tool overview
3D-BAT not only supports annotation of 3D point cloud data, but also annotation on 2D images. By providing a clear and intuitive user interface and flexible annotation options, the tool aims to provide researchers and developers with an efficient annotation environment.
3. Before You Begin: Getting Ready
First, you need to make sure the following necessary libraries and tools are installed in your environment:
- Three.js: for 3D visualization.
- Tensorflow.js: Can be used for prediction of deep learning models.
- Node.js: Run back-end code and provide API services.
It can be installed with the following command:
npm install three tensorflow node
4. Project structure
Here is our basic project structure:
/3D-BAT |-- /public | |-- /js | | |-- main.js | | |-- annotator.js |-- /views | |-- index.html |-- server.js
in:
main.js: is our main application logic code.annotator.js: Contains the core functionality of the annotation tool.index.html: is our main page.server.js: is our backend API service.
5. Set up basic 3D interface
In main.js , we first need to set up a basic 3D scene. For this, we need to setup the camera, renderer and scene. This can be done easily with Three.js.
// Initialize the scene
const scene = new THREE. Scene();
// Initialize the camera
const camera = new THREE. PerspectiveCamera(75, window. innerWidth / window. innerHeight, 0.1, 1000);
camera.position.z = 5;
// Initialize the renderer
const renderer = new THREE. WebGLRenderer({<!-- --> antialias: true });
renderer.setSize(window.innerWidth, window.innerHeight);
document.body.appendChild(renderer.domElement);
// render loop
function animate() {<!-- -->
requestAnimationFrame(animate);
renderer. render(scene, camera);
}
animate();
At this point, we have a basic 3D interface. But our tools are much more than that. In the next section, we will add functionality for point cloud data loading and display.
6. Load point cloud data
Point cloud data is usually stored in a variety of formats, the most common of which are PLY and PCD. To simplify the description, we use the PLY format here.
First, you need to add the PLY loader to Three.js. Make sure you have obtained the relevant library of PLYLoader.
const PLYLoader = require('three-ply-loader');
THREE.PLYLoader = PLYLoader(THREE);
In main.js, we add a function to load the PLY file:
const loader = new THREE. PLYLoader();
function loadPointCloud(path) {<!-- -->
loader.load(path, function(geometry) {<!-- -->
geometry.computeVertexNormals();
const material = new THREE. PointsMaterial({<!-- --> size: 0.01, vertexColors: true });
const cloud = new THREE. Points(geometry, material);
scene. add(cloud);
});
}
loadPointCloud('/path_to_your_ply_file.ply');
This code first loads the point cloud file, then creates a Three.js Points object for it and adds it to the scene.
7. Annotation of 2D images
In addition to 3D data, our tool also needs to support 2D image annotation. This requires adding an image element to the HTML page and using JavaScript to listen for mouse events.
In index.html:
<img id="image-to-annotate" src="/path_to_image.jpg" />
In annotator.js:
const imgElement = document. getElementById('image-to-annotate');
imgElement.addEventListener('click', function(event) {<!-- -->
const x = event.pageX - imgElement.offsetLeft;
const y = event.pageY - imgElement.offsetTop;
// Perform other operations based on (x, y) coordinates, such as adding labels, drawing boxes, etc.
});
8. Backend API design
In order to save and load annotations, we need a backend service. In server.js, we can easily accomplish this task using the Express.js framework.
First, make sure you have express installed:
npm install express
Then in server.js:
const express = require('express');
const app = express();
app.use(express.json());
app.use(express.static('public'));
app.get('/annotations', function(req, res) {<!-- -->
// load comments from database or file and send back to frontend
res.send({<!-- --> annotations: [] });
});
app.post('/annotations', function(req, res) {<!-- -->
// Save the comments sent by the front end to the database or file
res.send({<!-- --> success: true });
});
app. listen(3000, function() {<!-- -->
console.log('3D-BAT server running on port 3000');
});
9. Conclusion and further expansion
So far, we’ve covered how to create a basic 3D bounding box annotation tool using JavaScript. However, this is just the beginning. Practical tools require many other functions, such as user authentication, multi-user collaboration, complex annotation forms, etc.
We hope this article has given you a basis for building 3D annotation tools. With the increasing importance of 3D data in many application fields, the demand for such tools will continue to increase.
10. Optimize 3D rendering performance
For large-scale point cloud data, direct rendering may cause performance degradation. In order to improve rendering efficiency, we can adopt the following strategies:
-
LOD (Level Of Detail): Saves rendering time by reducing the detail of points far from the camera. Three.js provides LOD components, we can use it to achieve.
-
Blocked Rendering: Divide the point cloud data into chunks and render only the chunks within the camera’s line of sight.
11. User Interface and Interaction
A good annotation tool not only requires efficient annotation functions, but also an intuitive user interface.
In index.html, add the basic toolbar:
<div class="toolbar">
<button id="draw-box">Draw Box</button>
<button id="delete-box">Delete Box</button>
<!-- ... other tool buttons ... -->
</div>
In main.js, add an event listener for the button:
document.getElementById('draw-box').addEventListener('click', function() {<!-- -->
// Enable drawing 3D bounding box mode
});
document.getElementById('delete-box').addEventListener('click', function() {<!-- -->
// delete the selected 3D bounding box
});
12. Data management and storage
For each annotated point cloud or image data, we need to store related annotation information. This may include the location, size, angle and associated label of the bounding box.
This information can be easily stored and retrieved using the JSON format. For example:
{<!-- -->
"annotations": [
{<!-- -->
"type": "box",
"position": {<!-- --> "x": 10, "y": 5, "z": 2 },
"size": {<!-- --> "width": 2, "height": 2, "depth": 2 },
"rotation": {<!-- --> "x": 0, "y": 0, "z": 0 },
"label": "car"
},
// ... other comments ...
]
}
In server.js, we can use file system or database to store these data. For example, using the Node.js fs module:
const fs = require('fs');
app.post('/annotations', function(req, res) {<!-- -->
const annotations = req. body;
fs.writeFileSync('annotations.json', JSON.stringify(annotations));
res.send({<!-- --> success: true });
});
13. Future work and expansion
Although we have implemented a basic 3D bounding box annotation tool, there are many possible extensions and improvements:
- Add more annotation forms: such as 2D outline, 3D model matching, etc.
- Reinforcement Learning Support: Automatically predict bounding boxes using pre-trained models.
- Multi-user collaboration: Allow multiple users to annotate the same dataset at the same time, and provide version control.
14. Summary
This article shows you how to build a 3D bounding box annotation tool. From initializing a basic 3D interface to loading and displaying point cloud data, to 2D image annotation and backend API design, we’ve covered the basics of the tool.
While this is just the beginning, hopefully this will give you a base to build such a tool from scratch.
To explore this project further and get more features and details, Download the full project for details.