Kirin system installs HDP [Solved]

Kylin system installs HDP

  • Kylin system install HDP
    • 1. Software version introduction
    • 2. File replacement
    • 3 Error resolution
      • 3.1 Solve KeyError: ‘HDP-3.1’ (all machines)
      • 3.2 Install smartsense-hst (all machines)
      • 3.3 Solve Non-ASCII character ‘\xe5’ in file (all machines)
      • 3.4 Solve ImportError: No module named yum (all machines)
        • ==**Uninstall**==
        • python2
        • yum
        • python3 and dnf
        • Install
        • Update three content
        • It may also report an error site-packages
        • The other following dependent packages are downloaded by themselves according to the error report
      • 3.5 Solve redhat-lsb error (all machines)
      • 3.6 Solve No package hadoop_${stack_version}
      • 3.7 Solve the problem that hive cannot connect to the database
      • 3.8 Error ambari-server setup
    • 4. Rely on error reporting offline download

Kylin system install HDP

1. Software version introduction

Operating system: Kylin-Server-10-SP2-x86_64
HDP version: 3.1.5.0-152

2. File replacement

kylin.zip in this installation file

This plan checks them one by one and adds them manually. Due to the urgency, they are directly replaced.
If you have enough time and like to study, you can compare it with the original one.

A better comparison software: Beyond_Compare-4-Win-Release.exe

os_check.py
os_utils(agent).py
os_utils(server).py
yum_manager.py
Make a backup
Modify oc_check.py to disguise the Kirin system as centos8
/var/lib/ambari-agent/tmp/ambari_commons/os_check.py
/usr/lib/ambari-server/lib/ambari_commons/os_check.py
/usr/lib/ambari-agent/lib/ambari_commons/os_check.py

/usr/lib/ambari-server/lib/ambari_commons/repo_manager/yum_manager.py

Taking os_check.py as an example, it basically disguises kylin as centos
—-ps: Here it started to pretend to be 7, and other operations later also reported errors, so it was changed to 8
—- Kirin v10 is similar to centos8

188 lines

80 lines

91 lines

3 Error resolution

3.1 Solve KeyError: ‘HDP-3.1’ (all machines)

After the repair is completed, it can be distributed to other machines
scp -r /etc/yum.repos.d/hdp.repo root@vm1213:/etc/yum.repos.d/
scp -r /etc/yum.repos.d/hdp.repo root@vm1214:/etc/yum.repos.d/

vim /var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/params.py

Lowercase names recognized in code (all machines)
The uppercase name used in the previous installation, here is a direct copy of a lowercase name
cp /etc/yum.repos.d/HDP.repo /etc/yum.repos.d/hdp.repo

The content also needs to be checked and adjusted. Just modify the lowercase directly

According to the code, you can see that the square brackets will be operated. When modifying here, [] is filled with the value in the corresponding code.
[HDP-3.1]
[HDP-UTILS-1.1.0.22]
[HDP-3.1-GPL]

vim hdp.repo

[HDP-3.1]
name=HDP-3.1
baseurl=http://vm1212/HDP/centos7/3.1.5.0-152
gpgcheck=1
gpgkey=http://vm1212/HDP/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://vm1212/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://vm1212/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-3.1-GPL]
name=HDP-GPL Version - HDP-GPL-3.1.5.0
baseurl=http://vm1212/HDP-GPL/centos7/3.1.5.0-152
gpgcheck=1
gpgkey=http://vm1212/HDP-GPL/centos7/3.1.5.0-152/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

3.2 Install smartsense-hst (all machines)

yum -y install smartsense-hst

3.3 Solve Non-ASCII character \xe5’ in file (all machines)

/var/lib/ambari-agent/cache/stack-hooks/before-INSTALL/scripts/params.py

error encoding problem
Adding #coding=utf-8 in the first line of the file does not work. The number of lines where the error is reported is a Chinese comment. Delete the Chinese comment directly (specifically which file it is, modify it according to the error prompt)

3.4 Solve ImportError: No module named yum (all machines)

ImportError: No module named yum
ImportError: dnf.cli

>>> import yum
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named yum

Uninstall

This problem is mainly caused by the version problem, and some attempts have been made to no avail. Uninstall and reinstall directly
This operation uninstalled python2 python3 yum, please operate with caution

Uninstall python

# Uninstall
rpm -qa|grep python|xargs rpm -ev --allmatches --nodeps
# delete all leftover files
whereis python |xargs rm -frv
# Verify the deletion, return no result to indicate that the uninstallation is clean
where is python

Uninstall yum

# Uninstall
rpm -qa|grep yum|xargs rpm -ev --allmatches --nodeps
# delete all leftover files
whereis yum |xargs rm -frv
# Verify the deletion, return no result to indicate that the uninstallation is clean
where is yum

python2

If you have uninstalled python and yum, you need to download and reinstall
Create a new folder path to store the downloaded dependency packages

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-help-2.7.18-1.se.01.ky10. noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-2.7.18-1.se.01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-setuptools-44.1.1-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-tools-2.7.18-1.se.01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-debug-2.7.18-1.se.01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-rpm-4.15.1-17.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-tkinter-2.7.18-1.se.01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-devel-2.7.18-1.se.01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-libxml2-2.9.10-11.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-rpm-4.15.1-17.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-ipaddress-1.0.23-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python2-libdnf-0.48.0-1.p01.ky10.x86_64.rpm

yum

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-langpacks-0.4.5-10.oe1.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-langpacks-help-0.4.5-10.oe1.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-metadata-parser-1.1.4-24.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/yum-metadata-parser-help-1.1.4-24.ky10.x86_64.rpm

python3 and dnf

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-automatic-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-help-4.2.23-3.p02.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-plugins-core-4.0.17-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/dnf-plugins-core-help-4.0.17-1.ky10.noarch.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-3.7.9-6.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-gpgme-1.14.0-1.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-rpm-4.15.1-17.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-libdnf-0.48.0-1.p01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-libcomps-0.1.10-2.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-hawkey-0.48.0-1.p01.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/python3-dnf-4.2.23-3.p02.ky10.noarch.rpm

Install

rpm -Uvh –replacepkgs * –nodeps –force

Update three content

yum install python2
yum install python3
yum install dnf

May also report an error site-packages

Here another way is used. The site-packages file of python2 installed on the centos7 system is packaged
Replace existing python2 package
Path: /usr/lib/python2.7/site-packages all files
Replace all server /usr/lib/python2.7/site-packages (make server backup)

site-packages.tar.gz in this installation file

Other following dependent packages are downloaded by themselves according to the error report

openssl, nmap, libssh2, util-linux-devel

wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/openssl-1.1.1f-4.p01.ky10.x86_64.rpm
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/openssl-libs-1.1.1f-4.p01.ky10.x86_64.rpm
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/nmap-7.80-1.ky10.x86_64.rpm

wget
https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-1.9.0-6.ky10.x86_64.rpm
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-help-1.9.0-6.ky10.noarch.rpm
wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/libssh2-devel-1.9.0-6.ky10.x86_64.rpm

wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-10.p01.ky10. x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-10.p02.ky10.x86_64.rpm
wget http://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/util-linux-devel-2.35.2-8.p01.ky10.x86_64.rpm

3.5 Solve redhat-lsb error (all machines)

Redhat-lsb dependency error, the dependency of Kylin warehouse does not support downloading centos8 to force installation to solve
Install

wget https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/kylin-lsb-5.0-1.p01.ky10.x86_64.rpm

Ensure dependencies are available
Verification: lsb_release

Forcibly install the dependencies at the end of el8.x86_64.rpm
http://mirrors.aliyun.com/centos/8.5.2111/AppStream/x86_64/os/Packages/?spm=a2c6h.25603864.0.0.50f21a7abxdrEu/

This file contains the Kirin version of redhat-lsb but belongs to the aarch64 structure and does not use

Check whether it is available after the installation is successful

3.6 Solve No package hadoop_${stack_version}

resource_management.core.exceptions.Fail: No package found for -yarn(expected name: hadoop_3_1-yarn)

Directly write the version of stack_version

yum_manager.py uses scripts from this document

vim /usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py

3.7 Solve the problem that hive cannot connect to the database

If the hive command cannot be connected to the database, the following error will be reported

Check username and password configuration

Mysql’s hive library permission should be given to the corresponding machine
Mainly execute GRANT

CREATE USER 'hive'@'vm1213' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'vm1213';
CREATE USER 'hive'@'vm1214' IDENTIFIED BY 'bigdata';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'vm1214';

3.8 Error ambari-server setup

Mainly disguised as centos

If you still get an error, replace ambari.properties with ambari.properties in this article
vim /etc/ambari-server/conf/ambari.properties

4. Rely on error reporting offline download

Modify the context as needed:

If the dependency cannot be found when starting the installation, download it according to your own situation
Kylin Warehouse: https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/x86_64/Packages/
Kirin official website
https://update.cs2c.com.cn/NS/V10/V10SP1/os/adv/lic/base/aarch64/Packages/
https://update.cs2c.com.cn/NS/V10/V10SP2/os/adv/lic/base/aarch64/Packages/
Kirin
https://archive.kylinos.cn/yum/v10/kylin-openstack-q/aarch64/aarch64/