JAVA New Practice 4: Use JAVA to write a map tile download program and merge the tiles into large images in different levels

This article is full of work and uses code directly to show you how to write an autonomous and controllable map tile download program in Java, and merge the tiles into a large image hierarchically so that you can deploy your own tile map in geoserver.

The environment covered in this article is as follows:

Operating system: windows 11

Java JDK: OpenJDK21

Build tool: Gradle 8.4

Development tools: VsCode – Visual Studio Code 1.84.1

Key components:

1. org.apache.httpcomponents.client5:httpclient5:5.1.3 Network request components, download files, etc.

2. opencv 4.8 image processing component

Third-party resources: Tiantu

1. Project code structure

The core code consists of an entry program class, two tool classes, and a model class.

2. Configure project build.gradle

Set according to the following configuration to support org.apache.httpcomponents.client5:httpclient5 and opencv. For details on how to introduce opencv, please refer to my other article “JAVA New Practice 3: A Preliminary Study of Opencv + Java Applications”.

The complete code of build.gradle is as follows:

plugins {
id 'java'
id 'org.springframework.boot' version '3.1.5'
id 'io.spring.dependency-management' version '1.1.3'
}

group = 'com.jojava'
version = '0.0.1-SNAPSHOT'

java {
sourceCompatibility = '21'
}

configurations {
compileOnly {
extendsFrom annotationProcessor
}
}

repositories {
//Ali
maven { url 'https://maven.aliyun.com/nexus/content/groups/public/' }
mavenCentral()
}

dependencies {

implementation 'commons-io:commons-io:2. + '
implementation 'org.apache.httpcomponents.client5:httpclient5:5.1.3'
implementation(files("lib/opencv/opencv-480.jar"))

compileOnly 'org.projectlombok:lombok'
developmentOnly 'org.springframework.boot:spring-boot-devtools'
annotationProcessor 'org.projectlombok:lombok'
testImplementation 'org.springframework.boot:spring-boot-starter-test'
\t
}

3. Write a tile task model class

Create the package “com.jojava.joMapTile.model” and create the “TaskBlockDivide” class under the package. This model class defines the task block entity structure. The code is as follows:

package com.jojava.joMapTile.model;

import java.util.ArrayList;

import lombok.Data;

@Data
public class TaskBlockDivide {

    private Long countX;
    private Long countY;
    private ArrayList<Long[]> divideX;
    private ArrayList<Long[]> divide;

}

4. Write a tile xy coordinate generation tool class

Create the package “com.jojava.joMapTile.utils” and create the “TaskUtils” class under the package. This tool class is used to generate tile xy coordinates and tile serial numbers. The code is as follows:

package com.jojava.joMapTile.utils;

import java.util.ArrayList;

import com.jojava.joMapTile.model.TaskBlockDivide;

/**
 * Generate xy coordinates and numbers of tile task blocks
 *
 * @author lyd
 * @date 2023-11-08
 */

public class TaskUtils {

    public static TaskBlockDivide blockDivide(long xStart, long xEnd, long yStart, long yEnd, double d) {
        TaskBlockDivide divide = new TaskBlockDivide();
        long countX = xEnd - xStart + 1;
        long countY = yEnd - yStart + 1;
        int eachX = (int) Math.ceil(countX / d);
        int eachY = (int) Math.ceil(countY / d);
        if (countX <= d) {
            eachX = (int) Math.ceil(countX / 2);
            eachX = eachX == 0 ? 1 : eachX;
        }
        if (countY <= d) {
            eachY = (int) Math.ceil(countY / 2);
            eachY = eachY == 0 ? 1 : eachY;
        }
        ArrayList<Long[]> divideX = new ArrayList<>();
        ArrayList<Long[]> divideY = new ArrayList<>();
        if (countX / eachX <= 1) {
            Long arr[] = { 0L, countX };
            divideX.add(arr);
        } else {
            long cnt = (int) Math.floor(countX / eachX);
            long e = countX % eachX;
            for (var i = 0L; i < cnt; i + + ) {
                Long arr[] = { i * eachX, (i + 1) * eachX - 1 };
                divideX.add(arr);
            }
            if (cnt * eachX < cnt * eachX + e) {
                Long arrEnd[] = { cnt * eachX, cnt * eachX + e - 1 };
                divideX.add(arrEnd);
            }
        }
        if (countY / eachY <= 1) {
            Long arr[] = { 0L, countY };
            divideY.add(arr);
        } else {
            long cnt = (int) Math.floor(countY / eachY);
            long e = countY % eachY;
            for (var i = 0L; i < cnt; i + + ) {
                Long arr[] = { i * eachY, (i + 1) * eachY - 1 };
                divideY.add(arr);
            }
            if (cnt * eachY < cnt * eachY + e) {
                Long arrEnd[] = { cnt * eachY, cnt * eachY + e - 1 };
                divideY.add(arrEnd);
            }
        }
        divide.setCountX(countX);
        divide.setCountY(countY);
        divide.setDivideX(divideX);
        divide.setDivideY(divideY);
        return divide;
    }

}

5. Write a tool class for tile merging into large images

Create the “TileMergeUtils” class under the package “com.jojava.joMapTile.utils”. This tool class is used to merge tile images at each level into a single large image at the level. The code is as follows:

package com.jojava.joMapTile.utils;

import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
import org.opencv.core.CvType;
import org.opencv.core.Mat;
import org.opencv.core.Range;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

import lombok.Getter;

/**
 * Merge tiles into large image tiff
 *
 * @author lyd
 * @date 2023-11-08
 */

public class TileMergeUtils {

    private Mat des = null;

    private int tileWidth = 256; // Tile size width
    private int tileHeight = 256; // Tile size height

    @Getter
    private long allPixel = 0L;
    @Getter
    private long runPixel = 0L;

    public void init(int width, int height) {
        /*
         * CV_8uc1 single color channel 8 bit</br>
         * CV_8uc2 2 color channels 16 bits</br>
         * CV_8uc3 3 color channels 24 bits</br>
         * CV_8uc4 4 color channels 32 bit</br>
         */
        // CV_8UC4 is an RGBA format that supports transparent PNG
        this.des = Mat.zeros(height, width, CvType.CV_8UC4);
        // Calculate the total number of pixels
        this.allPixel = (long) width * height;
    }

    public void mergeToMat(String pathAndName, long x, long y) {
        //Read the picture
        var tileMat = Imgcodecs.imread(pathAndName, Imgcodecs.IMREAD_UNCHANGED);
        try {
            //Convert image to RGBA format
            Imgproc.cvtColor(tileMat, tileMat, Imgproc.COLOR_BGR2BGRA);
            // Determine coordinate position
            var rectForDes = this.des
                    .colRange(new Range((int) x, (int) x + tileWidth))
                    .rowRange(new Range((int) y, (int) y + tileHeight));
            // Fill to the merged image
            tileMat.copyTo(rectForDes);
        } catch (Exception ignored) {

        }
        // Calculate the number of merged pixels when completed
        this.runPixel + = (long) tileWidth * tileHeight;
    }

    public void output(String path, String name) throws IOException {
        String suffix = "tiff";
        String out = path + name + "." + suffix;
        FileUtils.createParentDirectories(new File(out));
        Imgcodecs.imwrite(out, this.des);
    }

    public void destroy() {
        this.des.release();
        this.des = null;
    }
}

5. Write project startup and running program

Edit the project’s default JoMapTileApplication. The tile download and merge call codes in this case are all written here. You can take them directly when you use them, or you can encapsulate them yourself. Currently, only the tile downloads of Tiantu Map are implemented here, and the others are injected into Baidu Maps. The principle of downloading tiles such as , Tencent Maps and so on is similar, and will be gradually added in the future.

The current startup and tile download program code is as follows:

package com.jojava.joMapTile;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.HttpEntity;
import org.apache.hc.core5.http.HttpStatus;

import com.jojava.joMapTile.utils.TaskUtils;
import com.jojava.joMapTile.utils.TileMergeUtils;

public class JoMapTileApplication {

//Introduce opencv_java480 component
static {
// windows
System.load(System.getProperty("user.dir") + "/lib/opencv/opencv_java480.dll");

//mac
// System.load(System.getProperty("user.dir") +
// "/lib/opencv/opencv_java480.dylib");

// linux
// System.load(System.getProperty("user.dir") +
// "/lib/opencv/opencv_java480.so");
}

// Map brand
public static String mapBrand = "tianditu"; // Default is Tianditu, currently mainly supported

// map key
public static String mapKey = "4913045c9f99f6b423d4027d5fb9658b"; // Some maps require a key to use, such as: Tian Map, please apply for it yourself
// http://lbs.tianditu.gov.cn/server/MapService.html

// map type
public static String mapType = "img_w"; // Each map brand has its own different map type

//Local home directory where tiles are downloaded and saved
public static String basePath = "E:/tianditu";

public static String[] servers = { "t0", "t1", "t2", "t3", "t4", "t5", "t6", "t7" };

public static int minZoom = 1; // Download start level
public static int maxZoom = 18; // Download end level
// Tiananmen 116.320303,39.964566 116.465182,39.871597
public static double startLat = 39.964566;//Start latitude (from north to south)
public static double endLat = 39.871597;//End latitude (from north to south)
public static double startLon = 116.320303;//Start longitude (from west to east)
public static double endLon = 116.465182;//End longitude (from west to east)

// coordinates
// public static double startLat = 44.092424; // Starting latitude (from north to south)
// public static double endLat = 43.940986; // End latitude (from north to south)
// public static double startLon = 126.045746;//Start longitude (from west to east)
// public static double endLon = 126.516601; // End longitude (from west to east)

public static void main(String[] args) {
// SpringApplication.run(TiandituApplication.class, args);

switch (mapBrand) {

case "tianditu":
TianDiTu();
break;

case "baidu":
BaiduMap();
break;

case "gaode":
GaodeMap();
break;

case "tencent":
TencentMap();
break;

default:
TianDiTu();

}

}

// Tiantu - currently only supports this one
public static void TianDiTu() {

// Pay attention to the limit on Tiantu API access times
String tk = mapKey;

String type = mapType;

// Image - Mercator
String img_w = "http://{server}.tianditu.gov.cn/" + type
+ "/wmts?SERVICE=WMTS & amp;REQUEST=GetTile & amp;VERSION=1.0.0 & amp;LAYER=img & amp;STYLE=default & amp;TILEMATRIXSET=w & amp;FORMAT=tiles & amp; TILEMATRIX={z} & amp;TILEROW={y} & amp;TILECOL={x} & amp;tk={tk}";

// String img_w =
// "http://{server}.tianditu.gov.cn/img_w/wmts?SERVICE=WMTS & amp;REQUEST=GetTile & amp;VERSION=1.0.0 & amp;LAYER=img & amp;STYLE= default & amp;TILEMATRIXSET=w & amp;FORMAT=tiles & amp;TILEMATRIX={z} & amp;TILEROW={y} & amp;TILECOL={x} & amp;tk={tk}";

// Plane - latitude and longitude projection
// Image annotation - Mercator
// public static String img_w =
// "http://{server}.tianditu.gov.cn/cia_w/wmts?SERVICE=WMTS & amp;REQUEST=GetTile & amp;VERSION=1.0.0 & amp;LAYER=img & amp;STYLE= default & amp;TILEMATRIXSET=w & amp;FORMAT=tiles & amp;TILEMATRIX={z} & amp;TILEROW={y} & amp;TILECOL={x} & amp;tk={tk}";

String[] urlArr = { img_w };//The layer to be downloaded

// Tile range at all levels
Map<String, Object> tilesMap = new HashMap<String, Object>();

//Start multithreading
ExecutorService exe = Executors.newFixedThreadPool(6);
//The first layer of equal longitude and latitude is 1x2, the number of latitudes is 2^0, and the number of longitudes is 2^1
//The first layer of Mercator projection is 2x2, the number of latitudes is 2^1, and the number of longitudes is 2^1
for (int i = 0; i < urlArr.length; i + + ) {
String url = urlArr[i].replace("{tk}", tk);
System.out.println(url);
String layerName = url.split("tianditu.gov.cn/")[1].split("/wmts?")[0];
for (int z = minZoom; z <= maxZoom; z + + ) {
double deg = 360.0 / Math.pow(2, z) / 256;//How many degrees does one pixel represent?
int startX = (int) ((startLon + 180) / deg / 256);
int endX = (int) ((endLon + 180) / deg / 256);
int startY = (((int) Math.pow(2, z) * 256 / 2)
- (int) ((Math.log(Math.tan((90 + startLat) * Math.PI / 360)) / (Math.PI / 180))
/ (360 / Math.pow(2, z) / 256) + 0.5))
/256;
int endY = (((int) Math.pow(2, z) * 256 / 2)
- (int) ((Math.log(Math.tan((90 + endLat) * Math.PI / 360)) / (Math.PI / 180))
/ (360 / Math.pow(2, z) / 256) + 0.5))
/256;

//Save the calculated tile number range
tilesMap.put(String.valueOf(z), String.valueOf(startX) + "," + String.valueOf(endX) + ","
+ String.valueOf(startY) + "," + String.valueOf(endY));
// Loop to download tiles
for (int x = startX; x <= endX; x + + ) {
for (int y = startY; y <= endY; y + + ) {
final String newUrl = url.replace("{server}", servers[(int) (Math.random() * servers.length)])
.replace("{z}", z + "").replace("{x}", x + "").replace("{y}", y + \ "");
System.out.println(newUrl);
final String filePath = basePath + "/" + layerName + "/" + z + "/" + x + "/" + y + ".png";
exe.execute(new Runnable() {
@Override
public void run() {
File file = new File(filePath);
if (!file.exists()) {
if (!file.getParentFile().exists()) {
file.getParentFile().mkdirs();
}
boolean loop = true;
int count = 0;
while (loop & amp; & amp; count < 5) {// Retry if download error occurs, up to 5 times
count + + ;
try {
InputStream in = getFileInputStream(newUrl);
OutputStream out = new BufferedOutputStream(new FileOutputStream(file));
byte[] b = new byte[8192];
int len = 0;
while ((len = in.read(b)) > -1) {
out.write(b, 0, len);
out.flush();
}
out.close();
in.close();
loop = false;
} catch (Exception e) {
loop = true;
}
}
if (loop) {
System.out.println("Download failed:" + newUrl);
}
}
}
});
}
}

// Merge tiles at this level
System.out.println("The last group of this layer" + endX);
mergeTileImage(layerName, z, startX, endX, startY, endY);
}
}
exe.shutdown();
while (true) {
try {
Thread.sleep(1000L);//The main thread sleeps for 1 second, waiting for the thread pool to finish running, while avoiding endless loops causing CPU waste
} catch (InterruptedException e) {
}
if (exe.isTerminated()) {//All threads in the thread pool end running
break;
}
}
}

\t// Baidu map
public static void BaiduMap() {
// Omit for now
}

// Tencent Map
public static void TencentMap() {
// Omit for now
}

// Gaode map
public static void GaodeMap() {
// Omit for now
}

// Get the file download stream
public static InputStream getFileInputStream(String url) throws Exception {
InputStream is = null;

CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet request = new HttpGet(url);
request.setHeader("User-Agent", "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt)");
CloseableHttpResponse response = httpclient.execute(request);
response.setHeader("Content-Type", "application/octet-stream");
int statusCode = response.getCode();
if (statusCode == HttpStatus.SC_OK) {
HttpEntity entity = response.getEntity();
is = entity.getContent();
}
return is;
}

// Merge images and specify the tile range
public static void mergeTileImage(String layerName, int zoom, long xStart, long xEnd, long yStart,
long yEnd) {

TileMergeUtils mat = new TileMergeUtils();

int tileWidth = 256; // Tile size width
int tileHeight = 256; // Tile size height

String savePath = basePath + "/" + layerName;
if (zoom == 0) {
return;
}
try {
System.out.print("Merging the" + zoom + "level map");

int z = zoom;
long mergeImageWidth = tileWidth * (xEnd - xStart + 1);
long mergeImageHeight = tileHeight * (yEnd - yStart + 1);
if (mergeImageWidth >= Integer.MAX_VALUE || mergeImageHeight >= Integer.MAX_VALUE) {
System.out.print("The merged image width:" + mergeImageWidth + ", height: " + mergeImageHeight + ", the width or height is greater than the maximum value of int"
+ Integer.MAX_VALUE + ", will not be merged. ");
return;
}
var WH = mergeImageWidth * mergeImageHeight;
System.out.print("The merged image width: " + mergeImageWidth + ", height: " + mergeImageHeight + ", pixel size: "
+WH);
if (WH > (long) Integer.MAX_VALUE) {
System.out
.print("The" + zoom + "The pixel size after merging the level map is greater than the maximum value of int" + Integer.MAX_VALUE + ". The merging time may be slightly longer. It is recommended that low-configuration computers do not perform oversized merging\ ");
}
//Start the thread
mat.init((int) mergeImageWidth, (int) mergeImageHeight);
var cpuCoreCount = Runtime.getRuntime().availableProcessors();
var d = Math.floor(Math.sqrt(cpuCoreCount));
var divide = TaskUtils.blockDivide(xStart, xEnd, yStart, yEnd, d);
var divideX = divide.getDivideX();
var divideY = divide.getDivideY();
// Loop through the layers and merge them
for (var i = 0; i < divideX.size(); i + + ) {
for (var j = 0; j < divideY.size(); j + + ) {

// Calculate point pixel position
long topLeftX = xStart, topLeftY = yStart;
long xStartPosit = xStart + divideX.get(i)[0];
long xEndPosit = xStart + divideX.get(i)[1];
long yStartPosit = yStart + divideY.get(j)[0];
long yEndPosit = yStart + divideY.get(j)[1];
//
System.out.println("xEndPosit: " + xEndPosit);
for (var x = xStartPosit; x < xEndPosit; x + + ) {
if (Thread.currentThread().isInterrupted()) {
break;
}
for (var y = yStartPosit; y <= yEndPosit; y + + ) {
if (Thread.currentThread().isInterrupted()) {
break;
}
var positionX = tileWidth * (x - topLeftX);
var positionY = tileWidth * (y - topLeftY);
var filePathAndName = savePath + "/" + z + "/" + x + "/" + y + ".png";
System.out.println(filePathAndName);
mat.mergeToMat(filePathAndName, positionX, positionY);
}
}

}
}

System.out.print("Writing to hard disk...");
//The saving location of the merged image
var outPath = savePath + "/merge" + "/";
var outName = "z" + z;
// Output the merged image
mat.output(outPath, outName);
mat.destroy();
System.out.print("The first" + zoom + "Level map merge completed");
} catch (Exception e) {
e.printStackTrace();
}
}
}

6. Start the running program

Click “Run” in the startup program to start running, you can automatically download tiles, and perform merging after each level of tiles is downloaded.

7. Code warehouse

The project code of this article has been open sourced to gitee, please download it.

joMapTile: A small tool for downloading map tiles and merging large images based on java (gitee.com)https:// gitee.com/duihao/jomaptile

8. Conclusion

I am working hard to continue to share various practical experiences related to JAVA with everyone. All technologies use the latest and mature technical architecture as much as possible. I look forward to helping you.

If you feel that this article is helpful and inspiring to you, thank you for your advice and encouragement to the author. Thank you. Every like, comment, and collection you give me is a great encouragement. I also ask my friends to give me some advice.