zookeeper (zookeeper) is an open source framework widely used in distributed services to provide coordination services Apache
Zookeeper is understood from a design pattern perspective: it is a distributed service management framework designed based on the observer pattern.
Responsible for storing and managing data that everyone cares about
,Then
Accept observer registration
, once the status of these data changes,
Zookeeper will
Responsible notification is already in
Those observers registered on Zookeeper
React accordingly.
zookeeper cluster features
- The zookeeper cluster consists of a leader (Leader) and multiple followers (Follower).
- As long as more than half of the nodes in the cluster survive, the Zookeeper cluster can serve normally. Therefore, Zookeeper is suitable for installing an odd number of servers.
- Global data consistency: Each server saves a copy of the same data. No matter which server the client connects to, the data is consistent.
- Update requests are executed sequentially, and update requests from the same Client are executed sequentially in the order they are sent.
- Data update is atomic, a data update either succeeds or fails.
- Real-time, within a certain time range, the Client can read the latest data.
Local mode installation
Zookeeper is developed in Java. To run zookeeper, you need to install jdk now.
Install jdk
You can download the installation-free version of jdk, unzip it and configure the environment variables.
The environment variable configuration file under linux is in
/etc/profile
Then add the following configuration under the file
export JAVA_HOME=/usr/java/jdk-21/jdk-21.0.1 // My own jdk directory
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
Then reload the configuration file, the command is
source /etc/profile
View Java version
It means that Jdk has been successfully installed.
Install zookeeper
Enter the zookeeper official website, find the download page, and download the -bin.tar.gz version. If you do not have the version you want, you can find it in the archive link on the download page.
Apache ZooKeeper
1. Copy
apache-zookeeper-3.5.7-bin.tar.gz
Install the package to
Linux
Under the system
2. Unzip to the specified directory
tar -zxvf apache-zookeeper-3.5.7- bin.tar.gz -C /mydata/zookeeper
3. Modify the name
mv apache-zookeeper-3.5.7 -bin/ zookeeper-3.5.7
Configuration modification
1. Modify zoo_sample.cfg in the path /mydata/zookeeper/zookeeper-3.5.7/conf to zoo.cfg;
mv zoo_sample.cfg zoo.cfg
2. Open the zoo.cfg file and modify the path of the dataDir configuration to the following content (the default tmp path is a temporary path. After a period of time, Linux will automatically delete the files inside, so it is not suitable for use in a formal environment)
dataDir=/mydata/zookeeper/zookeeper-3.5.7/data
3. Create the data folder in the directory /mydata/zookeeper/zookeeper-3.5.7/
mkdir data
Operate zookeeper
1. Start Zookeeper
Use ./zkServer.sh start in the bin directory to start zookeeper
2,
Check if the process is started
3. Check status
4. Start the client
./zkCli.sh
5. Exit the client:
[zk: localhost:2181(CONNECTED) 0] quit
6. Stop Zookeeper
.bin/zkServer.sh stop
Interpretation of configuration parameters
The meaning of the parameters in the configuration file zoo.cfg in Zookeeper is interpreted as follows:
1) tickTime = 2000: communication heartbeat time, Zookeeperserver Heartbeat time with the client, in milliseconds
2
)
initLimit = 10
:
LF
Initial communication time limit
The maximum number of heartbeats that the Leader and Follower can tolerate during the initial connection (the number of tickTimes)
3)syncLimit = 5：LFSynchronous communication time limit strong>
If the communication time between the Leader and Follower exceeds syncLimit * tickTime, the Leader considers the Follower to be dead and deletes the Follower from the server list.
4)dataDir:Save the data in Zookeeper
Note: The default tmp directory is easily deleted regularly by the Linux system, so the default tmp directory is generally not used.
5
)
clientPort = 2181
: Client connection port, usually not modified.
zookeeper cluster construction
1) Cluster planning
First prepare three Linux virtual machines. You can install one and then use the cloning function provided by vmware to clone. You need to shut down the virtual machine before cloning.
After cloning is complete
Modify cloner
IP
,The following uses hadoop100 (hadoop100 is the host name I set myself) as an example.
(1) Modify the static IP of the cloned virtual machine
Change to
TYPE=”Ethernet”
PROXY_METHOD=”none”
BROWSER_ONLY=”no”
BOOTPROTO=”static”
DEFROUTE=”yes”
IPV4_FAILURE_FATAL=”no”
IPV6INIT=”yes”
IPV6_AUTOCONF=”yes”
IPV6_DEFROUTE=”yes”
IPV6_FAILURE_FATAL=”no”
IPV6_ADDR_GEN_MODE=”stable-privacy”
NAME=”ens33″
UUID=”d7919ed1-15af-49fe-9b4e-aad1456eed3d”
DEVICE=”ens33″
ONBOOT=”yes”IPADDR=192.168.127.100
#wangGuan
GATEWAY=192.168.127.2
#DNS
DNS1=192.168.127.2
(2) View the virtual network editor of the Linux virtual machine, Edit->Virtual Network Editor->VMnet8
(3) View the IP address of the Windows system adapter VMware Network Adapter VMnet8
(4) Ensure that the IP address in the ifcfg-ens33 file of the Linux system, the virtual network editor address and the VM8 network IP address of the Windows system are in the same network segment.
2) Modify the host name of the clone machine. The following uses hadoop100 as an example
(1) Modify the host name
vim /etc/hostname
hadoop102
(2) Configure Linux clone machine host name mapping hosts file, open /etc/host
[root@hadoop100 ~]# vim /etc/hosts
Add the following content
192.168.127.100hadoop100
192.168.127.101hadoop101
192.168.127.102 hadoop102
192.168.127.103 hadoop103
192.168.127.104 hadoop104
192.168.127.105 hadoop105
192.168.127.106 hadoop106
192.168.127.107 hadoop107
192.168.127.108 hadoop108
192.168.127.132 qingmangmall
3) Restart the clone machine hadoop100
[root@hadoop100 ~]# reboot
4) Modify the host mapping file of windows ( hosts file)
- (a) Enter the C:\Windows\System32\drivers\etc path
- (b) Open the hosts file and add the following content, then save
192.168.127.100hadoop100
192.168.127.101hadoop101
192.168.127.102 hadoop102
192.168.127.103 hadoop103
192.168.127.104 hadoop104
192.168.127.105 hadoop105
192.168.127.106 hadoop106
192.168.127.107 hadoop107
192.168.127.108 hadoop108
192.168.127.132 qingmangmall
The cloned virtual machine is now ready for use.
If three virtual machines have been used before, how to quickly share one of the files to other machines?
3) Cluster distribution
Distribute zookeeper to other services
Prerequisite: in
hadoop102
,
hadoop103
,
hadoop104
The /mydata directory has been created
And these two directories have been modified to user:user (the user who wants to operate the directory)
[user@hadoop102 ~]$ sudo chown user:user -R /mydata
in hadoop102
Up, will
hadoop102
middle
/mydata/zookeeper
Directory copied to
hadoop103
superior.
[user@
hadoop102
~]$ scp -r /mydata/zookeeper user@hadoop103:/mydata
Other servers follow this example
At this point, all three servers have zookeeper services.
To enable a zookeeper cluster, you need to create it in the /mydata/zookeeper/zookeeper-3.5.7/data directory
one
myid
document.
Add to the file the
server
The corresponding number (note: there should be no blank lines above and below, and no spaces on the left and right). Here I use the last three digits of the Linux machine IP as the ID number.
100
NOTE: Add
myid
file must be in
Linux
Created in
notepad++
It’s probably garbled inside
(
3
)Copy the configured
zookeeper
to other machines
scp -r /mydata/zookeeper/zookeeper-3.5.7/data
user@hadoop103:/mydata/zookeeper/zookeeper-3.5.7
Configurationzoo.cfgFile
(
1
)Rename
/opt/module/zookeeper-3.5.7/conf
in this directory
zoo_sample.cfg
for
zoo.cfg
[user@hadoop102 conf]$ mv zoo_sample.cfg zoo.cfg
(
2
)Open
zoo.cfg
document
[user@hadoop102 conf]$ vim zoo.cfg
#
Modify data storage path configuration
dataDir=/opt/module/zookeeper-3.5.7/zkData
#
Add the following configuration
######################cluster#########################
server.100=hadoop100:2888:3888
server.103=hadoop103:2888:3888
server.132=qingmangmall:2888:3888
(
3
) Interpretation of configuration parameters
server.A=B:C:D
.
A
It is a number indicating which server this is;
Configure a file in cluster mode
myid
, this file is in
dataDir
directory, there is a data in this file
that is
A
value,
Zookeeper
Read this file at startup and get the data inside and
zoo.cfg
The configuration information inside is
Compare it to determine which one it is
server
.
B
is the address of this server;
C
It’s this server
Follower
with the cluster
Leader
The port through which the server exchanges information;
D
In case of cluster
Leader
The server is down and a port is needed to re-elect and select a new one.
Leader
, and this port is the port used by servers to communicate with each other when performing elections.
(
4
)Synchronize
zoo.cfg
Configuration file
Cluster operations
(
1
) start separately
Zookeeper
[user@hadoop100 zookeeper-3.5.7]$ bin/zkServer.sh start
[user@hadoop103 zookeeper-3.5.7]$ bin/zkServer.sh start
[user@qingmangmall zookeeper-3.5.7]$ bin/zkServer.sh start
(
2
) View status
[user@hadoop100 zookeeper-3.5.7]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode:
follower[user@hadoop103 zookeeper-3.5.7]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode:
leader[user@qingmangmall zookeeper-3.5.7]# bin/zkServer.sh status
JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Mode:
follower
Appendix
1) Create a user and set the group to which the file belongs
Users can be created using the following operations
[root@hadoop100 ~]# useradd user
[root@hadoop100 ~]# passwd user
Configure the user user to have root
Permissions for easy addition later
sudo
implement
root
permissions command
[root@hadoop100 ~]# vim
/etc/sudoers
Revise
/etc/sudoers
file, in
%wheel
This line
Add a line below as shown below:
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
## Allows people in group wheel to run all commands
%
wheel
ALL=(ALL) ALLuser
ALL=(ALL) NOPASSWD:ALL
Notice:
Do not put the user line directly into root
line below because all users belong to
wheel
Group, you first
The user is configured to have the password-free function, but when the program executes to the %wheel line, the function is overwritten and returned to the required function.
password. So user should be placed under the %wheel line.
In the root directory
Create a folder under and modify the owner and group
mkdir /mydata
Modify the owner and group of the mydata folder to be
user
User
[root@hadoop100 ~]# chown user:user /mydata
Check the owner and group of the /mydata folder. You can find that the owner has been changed to the user you set.
2)scp(secure copy) safe< /strong>Copy
(
1
)
scp
definition
scp can copy data between servers. (
from server1 to server2
)
(
2
)Basic syntax
scp -r $pdir/$fname $user@$host:$pdir/$fname
Order
recursion
File path to copy/
Name
destination user
@
Host
:
destination path
/
Name
3) rsync Remote Sync Tool
rsync
Mainly used for backup and mirroring. It has the advantages of being fast, avoiding copying the same content, and supporting symbolic links.
rsync
and
scp
the difference:
use
rsync
Doing a copy of a file is better than
scp
The speed is fast,
rsync
Only make changes to the difference files
new.
scp
Just copy all the files.
(
1
)Basic syntax
rsync -av $pdir/$fname $user@$host:$pdir/$fname
Order
option parameters
File path to copy
/
Name
destination user
@
Host
:
destination path
/
Name
Option parameter description
Option Function
-a archive copy
-v displays the copy process