Use Docker to deploy the open source distributed task scheduling system DolphinScheduler

Use Docker to deploy the open source distributed task scheduling system DolphinScheduler

A few days ago, I discovered a giant artificial intelligence learning website. It is easy to understand and humorous. I couldn’t help but share it with everyone. Click to jump to the website.

Article directory

• Use Docker to deploy the open source distributed task scheduling system DolphinScheduler
• • Preface
  • 1. Install and deploy DolphinScheduler
  • • 1.1 Start the service
  • 2. Log in to the DolphinScheduler interface
  • 3. Install intranet penetration tools
  • 4. Configure Dolphin Scheduler public network address
  • 5. Fixed DolphinScheduler public network address

Foreword

This tutorial will share with you the installation and deployment of DolphinScheduler and how to achieve remote access to the public network. Combined with the intranet penetration tool, the public network can access the DolphinScheduler intranet and perform remote work, helping developers to schedule and manage remote tasks and improve work efficiency. .

DolphinScheduler is an open source distributed task scheduling system that can help developers schedule and manage tasks more conveniently. DolphinScheduler supports common task types, including Shell, Hadoop, Spark, Hive, etc., and it also provides visual task orchestration and monitoring, making task management easier. The development team of DolphinScheduler originated from Alipay’s data platform team. After years of practice and accumulation, it has become a mature open source project.

1. Install and deploy DolphinScheduler

Prerequisites:

Docker 1.13.1 or above and Docker Compose 1.28.0 or above need to be installed.

1.1 Start service

Use standalone-server mirror

Starting a DolphinScheduler standalone-server container using the standalone-server image should be the fastest way to experience DolphinScheduler. In this way, you can experience most of the functions of DolphinScheduler as quickly as possible and understand the main concepts and content.

$ DOLPHINSCHEDULER_VERSION=3.2.0
$ docker run --name dolphinscheduler-standalone-server -p 12345:12345 -p 25333:25333 -d apache/dolphinscheduler-standalone-server:"${DOLPHINSCHEDULER_VERSION}"

2. Log in to the DolphinScheduler interface

After starting the service, enter http://localhost:12345/dolphinscheduler/ui in the browser to access DolphinScheduler.

Default username: admin Password: dolphinscheduler123

After successful login, enter the DolphinScheduler management interface.

3. Install intranet penetration tool

After completing the deployment of DolphinScheduler, you can install cpolar. To install cpolar on an ubuntu system, you can install it using a simple one-click installation script. As long as you enter the following command on the command line interface of Ubuntu, you can automatically execute the installation program (it should be noted that some Ubuntu versions may not have the curl tool installed, so it is best to execute the command “sudo aptinstall curl” to install the curl tool first).

curl -L https://www.cpolar.com/static/downloads/install-release-cpolar.sh | sudo bash

After the Cpolar installation is completed, you can enter the command “sudo systemctl start cpolar” to start cpolar.

 sudo systemctl start cpolar

Of course, we can also enter “cpolar version” instead of using the systemctl level command.

cpolar version

Query the version number of local cpolar. As long as the version information can be displayed, the cpolar installation is complete.

In order to ensure that cpolar can run on the Ubuntu system for a long time and ensure the stable existence of the data tunnel, it is best to add cpolar to the Ubuntu boot auto-start list. As long as you enter the command “sudo systemctl status cpolar” in the Ubuntu command line interface, you can add cpolar to the self-starting list.

sudo systemctl status cpolar

In order to ensure the data security of each user and create a separate data tunnel for each customer, cpolar uses user password and token code for user verification, so we need to register the user before using cpolar. The registration process is very simple. Just click “User Registration” in the upper right corner of the cpolar homepage and fill in the necessary information on the registration page to complete the registration.

After completing the cpolar user registration, we can use each user’s unique token code to activate the cpolar client. As long as you log in to the cpolar official website, you can find the user’s unique token code on the “Verification” page (or the “Connect Your Account” window).

Copy and paste this token code into the Ubuntu command line interface, and the cpolar client will write the token code into the local cpolar.yml file (token code activation only needs to be done once) as an identification of the user data tunnel. information. The specific command format is “cpolar authtoken user’s unique token code”.

cpolar authtoken token code

After cpolar is successfully installed, access port 9200 on an external browser, enter localhost:9200 in the browser address bar, log in with your cpolar account, and after logging in, you will see the cpolar web configuration interface. Next, go to the web Just configure the management interface.

4. Configure Dolphin Scheduler public network address

Click Tunnel Management – Create Tunnel on the left dashboard to create a Dolphin Scheduler public network address tunnel!

• Tunnel name: You can customize the name. Be careful not to duplicate the existing tunnel name.
• Protocol: Select http
• Local address: 12345
• Domain name type: Choose a random domain name for free
• Region: Select China Top

Click Create

After the tunnel is successfully created, click Status on the left – Online Tunnel List to view the generated public network access address. There are two access methods, one is http and https

Note, copy the cpolar https public address above and add the path after the website address. For example:

http://22da3a53.r8.cpolar.top/dolphinscheduler/ui/login

Access the browser of any device and you can successfully see the DolphinScheduler login interface. In this way, a public network address that can be accessed remotely is created. Using the public network address of cpolar, you do not need to purchase a cloud server yourself, and you can publish it to Public network for remote access.

5. Fixed DolphinScheduler public network address

Since the tunnel created using cpolar above uses a random public network address, it will change randomly within 24 hours, which is not conducive to long-term remote access. Therefore, we can configure a second-level subdomain name for it. This address is a fixed address and will not change randomly [ps: cpolar.cn has been filed]

Note that you need to upgrade the cpolar package to a basic package or above, and the bandwidth corresponding to each package is different. [cpolar.cn has been registered]

Log in to the cpolar official website, click Reserve on the left, select to reserve the second-level subdomain name, set a second-level subdomain name, click Reserve, and copy the reserved second-level subdomain name after the reservation is successful.

After the reservation is successful, copy the name of the second-level subdomain name that was successfully reserved.

Return to the cpolar web UI management interface, click Tunnel Management – Tunnel List on the left dashboard, find the tunnel you want to configure, and click Edit on the right

Modify the tunnel information and configure the successfully reserved second-level subdomain name into the tunnel.

• Domain name type: Select a second-level subdomain name
• Sub Domain: Fill in the successfully reserved second-level subdomain name

Click Update (note, click once to update, no need to submit again)

After the update is completed, open the online tunnel list. At this time, you can see that the public network address has changed and the address name has become a domain name with a fixed second-level subdomain name.

Finally, we use a fixed public network address plus the path behind the website address to access, for example:

http://dolphinscheduler.vip.cpolar.cndolphinscheduler/ui/login

You can see that the access is successful. In this way, a fixed and permanent public network address has been set. You can access the DolphinScheduler intranet and work remotely anytime and anywhere, helping developers to schedule and manage remote tasks and improve work efficiency!