Elasticsearch Basic Security plus secure HTTPS traffic

Basic security plus secure HTTPS traffic

  • In a production environment, certain Elasticsearch features, such as tokens and API keys, are disabled unless you enable TLS at the HTTP layer. This extra layer of security ensures that all communication to and from the cluster is secure.

  • When you run the elasticsearch-certutil tool in mode http, the tool will ask a few questions about how you want the certificate to be generated. While there are many options, the following options generate certificates suitable for most environments.

  • The first question the elasticsearch-certutil tool prompts you with is whether you want to generate a certificate signing request (CSR). Answer n Do you want to sign your own certificate, or y want to sign a central CA’s certificate.

– Sign your own certificate

  • If you want to use your self-generated CA certificate. Answer n when asked if you want to generate a CSR. Then specify the location of the CA certificate that is signed and generated by the tool.p12.

– Use a central CA to sign the certificate

  • If you work in an environment with a central security team, they may generate certificates for you. The infrastructure within your organization may already be configured to trust existing CAs, so it may be easier for clients to connect to Elasticsearch if you use a CSR and send that request to the team that controls your CA. To use a central CA, answer y to the first question.

Prerequisites

  • Complete all steps in Setting up basic security for the Elastic Stack. Please refer to the following documents:
    Elasticsearch Setup Basic Security

Encrypt HTTP client communication for Elasticsearch

  • On each node in the cluster, stop Elasticsearch and Kibana (if they are running).
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl stop elasticsearch9201
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl stop elasticsearch9202
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl stop elasticsearch9203

  • On any single node, run the Elasticsearch HTTP certificate tool from the directory where you installed Elasticsearch to generate a certificate signing request (CSR).

  • This command produces a .zip file containing certificates and keys for Elasticsearch and Kibana. Each folder contains a README.txt explaining how to use the files.

root@ubuntu-x64_02:/usr/local/elasticsearch9201#./bin/elasticsearch-certutil http
  • When asked if you want to generate a CSR, enter: n
## Do you wish to generate a Certificate Signing Request (CSR)?

A CSR is used when you want your certificate to be created by an existing
Certificate Authority (CA) that you do not control (that is, you do not have
access to the keys for that CA).

If you are in a corporate environment with a central security team, then you
May have an existing Corporate CA that can generate your certificate for you.
Infrastructure within your organization may already be configured to trust this
CA, so it may be easier for clients to connect to Elasticsearch if you use a
CSR and send that request to the team that controls your CA.

If you choose not to generate a CSR, this tool will generate a new certificate
for you. That certificate will be signed by a CA under your control.
quick and easy way to secure your cluster with TLS, but you will need to
configure all your clients to trust that custom CA.

Generate a CSR? [y/N]N
  • When asked if you want to use an existing CA, enter: y
## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?

If you have an existing CA certificate and key, then you can use that CA to
sign your new http certificate. This allows you to use the same CA across
multiple Elasticsearch clusters which can make it easier to configure clients,
and may be easier for you to manage.

If you do not have an existing CA, one will be generated for you.

Use an existing CA? [y/N]y
  • Enter the path to your CA. elastic-stack-ca.p12 This is the absolute path to the file you generated for your cluster. Such as: /usr/local/elasticsearch9201/elastic-stack-ca.p12
  • Enter your CA’s password. Empty if none, in which case you just press ENTER at the prompt
## What is the path to your CA?

Please enter the full pathname to the Certificate Authority that you wish to
use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
(.jks) or PEM (.crt, .key, .pem) format.
CA Path: /usr/local/elasticsearch9201/elastic-stack-ca.p12
Reading a PKCS12 keystore requires a password.
It is possible for the keystore is password to be blank,
in which case you can simply press <ENTER> at the prompt
Password for elastic-stack-ca.p12:
  • Enter an expiration value for the certificate. You can: Enter the validity period in years, months or days. For example, enter 90D
## How long should your certificates be valid?

Every certificate has an expiry date. When the expiry date is reached clients
will stop trusting your certificate and TLS connections will fail.

Best practice suggests that you should either:
(a) set this to a short duration (90 - 120 days) and have automatic processes
to generate a new certificate before the old one expires, or
(b) set it to a longer duration (3 - 5 years) and then perform a manual update
a few months before it expires.

You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)

For how long should your certificate be valid? [5y] 3Y
  • Enter Y when asked if you want to generate a certificate for each node, each certificate has its own private key and is issued for a specific hostname or IP address
## Do you wish to generate one certificate per node?

If you have multiple nodes in your cluster, then you may choose to generate a
separate certificate for each of these nodes. Each certificate will have its
own private key, and will be issued for a specific hostname or IP address.

Alternatively, you may wish to generate a single certificate that is valid
across all the hostnames or addresses in your cluster.

If all of your nodes will be accessed through a single domain
(e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
simpler to generate one certificate with a wildcard hostname (*.es.example.com)
and use that across all of your nodes.

However, if you do not have a common domain name, and you expect to add
additional nodes to your cluster in the future, then you should generate a
certificate per node so that you can more easily generate new certificates when
you provision new nodes.

Generate a certificate per node? [y/N]Y
  • When prompted, enter the name of the first node in the cluster. Use the same node name that was used when generating the node certificate.
## What is the name of node #1?

This name will be used as part of the certificate file name, and as a
descriptive name within the certificate.

You can use any descriptive name that you like, but we recommend using the name
of the Elasticsearch node.
 
node #1 name: node-1
  • Enter any hostnames used to connect to your first node. These hostnames are added as DNS names to the certificate’s Subject Alternative Name (SAN) field.
## Which hostnames will be used to connect to node-1?

These hostnames will be added as "DNS" names in the "Subject Alternative Name"
(SAN) field in your certificate.

You should list every hostname and variant that people will use to connect to
your cluster over http.
Do not list IP addresses here, you will be asked to enter them later.

If you wish to use a wildcard certificate (for example *.es.example.com) you
can enter that here.

Enter all the hostnames that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

node-1.es.example.com

You entered the following hostnames.

 - node-1.es.example.com

Is this correct [Y/n]Y
  • Enter the IP address that clients can use to connect to your node. The local loopback address is used here
## Which IP addresses will be used to connect to node-1?

If your clients will ever connect to your nodes by numeric IP address, then you
can list these as valid IP "Subject Alternative Name" (SAN) fields in your
certificate.

If you do not have fixed IP addresses, or not wish to support direct IP access
to your cluster then you can just press <ENTER> to skip this step.

Enter all the IP addresses that you need, one per line.
When you are done, press <ENTER> once more to move on to the next step.

127.0.0.1


You entered the following IP addresses.

 - 127.0.0.1

Is this correct [Y/n]Y
  • Repeat these steps for each additional node in the cluster.
  • After generating certificates for each node, enter the private key passphrase when prompted. Or press Enter directly to be empty.
## What password do you want for your private key(s)?

Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
This type of keystore is always password protected, but it is possible to use a
blank password.

If you wish to use a blank password, simply press <enter> at the prompt below.
Provide a password for the "http.p12" file: [<ENTER> for none]
  • Save to the current directory by default
## Where should we save the generated files?

A number of files will be generated including your private key(s),
public certificate(s), and sample configuration options for Elastic Stack products.

These files will be included in a single zip archive.

What filename should be used for the output zip file? [/usr/local/elasticsearch9201/elasticsearch-ssl-http.zip]

Zip file written to /usr/local/elasticsearch9201/elasticsearch-ssl-http.zip
  • Unzip the resulting elasticsearch-ssl-http.zip file. This zip file contains a directory for Elasticsearch and Kibana.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# unzip elasticsearch-ssl-http.zip
Archive: elasticsearch-ssl-http.zip
   creating: elasticsearch/
   creating: elasticsearch/node-1/
  inflating: elasticsearch/node-1/README.txt
  inflating: elasticsearch/node-1/http.p12
  inflating: elasticsearch/node-1/sample-elasticsearch.yml
   creating: elasticsearch/node-2/
  inflating: elasticsearch/node-2/README.txt
  inflating: elasticsearch/node-2/http.p12
  inflating: elasticsearch/node-2/sample-elasticsearch.yml
   creating: elasticsearch/node-3/
  inflating: elasticsearch/node-3/README.txt
  inflating: elasticsearch/node-3/http.p12
  inflating: elasticsearch/node-3/sample-elasticsearch.yml
   creating: kibana/
  inflating: kibana/README.txt
  inflating: kibana/elasticsearch-ca.pem
  inflating: kibana/sample-kibana.yml
  • View the certificate generation directory: es node certificate
root@ubuntu-x64_02:/usr/local/elasticsearch9201# tree elasticsearch
elasticsearch
├── node-1
│ ├── http.p12
│ ├── README.txt
│ └── sample-elasticsearch.yml
├── node-2
│ ├── http.p12
│ ├── README.txt
│ └── sample-elasticsearch.yml
└── node-3
    ├── http.p12
    ├── README.txt
    └── sample-elasticsearch.yml

3 directories, 9 files
  • View the certificate generation directory: kibana certificate
root@ubuntu-x64_02:/usr/local/elasticsearch9201# tree kibana/
kibana/
├── elasticsearch-ca.pem
├── README.txt
└── sample-kibana.yml

0 directories, 3 files
  • On each node in the cluster, complete the following steps:
  • Copy the relevant http.p12 certificates into the $ES_PATH_CONF directory.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp elasticsearch/node-1/http.p12 /usr/local/elasticsearch9201/config/
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp elasticsearch/node-2/http.p12 /usr/local/elasticsearch9202/config/
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp elasticsearch/node-3/http.p12 /usr/local/elasticsearch9203/config/
  • Edit the elasticsearch.yml file to enable HTTPS security and specify the location of the security certificate http.p12.
cat >> /usr/local/elasticsearch9201/config/elasticsearch.yml <<EOF

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

EOF


cat >> /usr/local/elasticsearch9202/config/elasticsearch.yml <<EOF

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

EOF

cat >> /usr/local/elasticsearch9203/config/elasticsearch.yml <<EOF

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

EOF
  • Add your private key passphrase to Elasticsearch’s security settings.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cd /usr/local/elasticsearch9201; ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Enter value for xpack.security.http.ssl.keystore.secure_password:

root@ubuntu-x64_02:/usr/local/elasticsearch9201# cd /usr/local/elasticsearch9202; ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Enter value for xpack.security.http.ssl.keystore.secure_password:

root@ubuntu-x64_02:/usr/local/elasticsearch9202# cd /usr/local/elasticsearch9203; ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
Enter value for xpack.security.http.ssl.keystore.secure_password:
  • Modify file permissions
root@ubuntu-x64_02:/usr/local/elasticsearch9203# chown -R elk:elk /usr/local/elasticsearch9201
root@ubuntu-x64_02:/usr/local/elasticsearch9203# chown -R elk:elk /usr/local/elasticsearch9202
root@ubuntu-x64_02:/usr/local/elasticsearch9203# chown -R elk:elk /usr/local/elasticsearch9203
  • Start Elasticsearch.
root@ubuntu-x64_02:/usr/local/elasticsearch9203# systemctl start elasticsearch9201
root@ubuntu-x64_02:/usr/local/elasticsearch9203# systemctl start elasticsearch9202
root@ubuntu-x64_02:/usr/local/elasticsearch9203# systemctl start elasticsearch9203
  • add hosts explanation
root@ubuntu-x64_02:/usr/local/elasticsearch9203# echo "127.0.0.1 node-1.es.example.com" >> /etc/hosts
root@ubuntu-x64_02:/usr/local/elasticsearch9203# echo "127.0.0.1 node-2.es.example.com" >> /etc/hosts
root@ubuntu-x64_02:/usr/local/elasticsearch9203# echo "127.0.0.1 node-3.es.example.com" >> /etc/hosts
  • Use the -k option to not verify the CA certificate, otherwise it will cause problems that cannot be verified
root@ubuntu-x64_02:/usr/local/elasticsearch9201# curl -k -u elastic:123456 https://node-1.es.example.com:9201/_cat/nodes/?v;
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
127.0.0.1 56 97 6 0.28 0.14 0.21 cdfhilmrstw-node-1
127.0.0.1 48 97 6 0.28 0.14 0.21 cdfhilmrstw-node-3
127.0.0.1 63 97 6 0.28 0.14 0.21 cdfhilmrstw * node-2

Encrypt HTTP client communication for Kibana

  • The browser sends traffic to Kibana, and Kibana sends traffic to Elasticsearch. These communication channels are individually configured to use TLS. You encrypt traffic between the browser and Kibana, and then encrypt traffic between Kibana and Elasticsearch.

Encrypt traffic between Kibana and Elasticsearch

  • When you run the elasticsearch-certutil tool with the http option, it creates a /kibana directory containing the elasticsearch-ca.pem file. You use this file to configure Kibana to trust the Elasticsearch CA at the HTTP layer.
  • Copy the file elasticsearch-ca.pem to the Kibana configuration directory, as defined by path $KBN_PATH_CONF.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp kibana/elasticsearch-ca.pem /usr/local/kibana/config/
  • Open kibana.yml and add the following line to specify the location of the security certificate for the HTTP layer.
cat >> /usr/local/kibana/config/kibana.yml <<EOF

elasticsearch.ssl.certificateAuthorities: /usr/local/kibana/config/elasticsearch-ca.pem

EOF
  • Add the following line to specify the HTTPS URL of the Elasticsearch cluster.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cat /usr/local/kibana/config/kibana.yml| grep "elasticsearch.hosts"
elasticsearch.hosts: ["https://127.0.0.1:9201" ,"https://127.0.0.1:9202" ,"https://127.0.0.1:9203"]
  • Modify permissions
root@ubuntu-x64_02:/usr/local/elasticsearch9201# chown -R elk:elk /usr/local/kibana
  • restart kibana
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl restart kibana.service

Encrypt traffic between browser and Kibana

  • Kibana creates the server certificate and private key for you. Kibana uses this server certificate and corresponding private key when receiving connections from web browsers.

  • The following creates a Certificate Signing Request (CSR) for Kibana. The CSR contains information used by the CA to generate and sign the security certificate. Certificates can be trusted (signed by a public, trusted CA) or untrusted (signed by an internal CA). Self-signed or internally signed certificates can be used in development environments and building proofs of concept, but should not be used in production environments.

  • Use a trusted CA such as Let’s Encrypt or your organization’s internal CA to sign the certificate before going into production. Use signed certificates to establish browser trust to connect to Kibana for internal access or access on the public internet.

  • Generate a server certificate and private key for Kibana.

root@ubuntu-x64_02:/usr/local/elasticsearch9201# ./bin/elasticsearch-certutil csr -name kibana-server -dns kibana.example.com
  • Unzip the csr-bundle.zip file to get kibana-server.csr unsigned security certificate and kibana-server.key unencrypted private key.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# unzip csr-bundle.zip
Archive: csr-bundle.zip
   creating: kibana-server/
  inflating: kibana-server/kibana-server.csr
  inflating: kibana-server/kibana-server.key
  • Send the certificate kibana-server.csr signing request to your internal CA or a trusted CA for signing to obtain a signing certificate. The signature file can be in different formats, .crt eg: kibana-server.crt.
  • Generate a certificate signature using a self-signed CA
root@ubuntu-x64_02:/usr/local/elasticsearch9201# tree kibana-server/
kibana-server/
├── kibana-server.crt
├── kibana-server.csr
└── kibana-server.key

0 directories, 3 files
  • Open kibana.yml and add the following lines to configure Kibana to access the server certificate and unencrypted private key.
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp ./kibana-server/kibana-server.crt /usr/local/kibana/config/
root@ubuntu-x64_02:/usr/local/elasticsearch9201# cp ./kibana-server/kibana-server.key /usr/local/kibana/config/

cat >> /usr/local/kibana/config/kibana.yml <<EOF

server.ssl.certificate: /usr/local/kibana/config/kibana-server.crt
server.ssl.key: /usr/local/kibana/config/kibana-server.key

EOF


# Directory Structure 
root@ubuntu-x64_02:/usr/local/elasticsearch9201# ls -tlh /usr/local/kibana/config/
total 44K
-rw-r--r-- 1 root root 989 Mar 23 15:24 kibana-server.crt
-rw-r--r-- 1 elk elk 5.5K Mar 23 15:20 kibana.yml
-rw-r--r-- 1 elk elk 1.7K Mar 23 14:56 kibana-server.key
-rw-r--r-- 1 elk elk 1.2K Mar 22 18:04 elasticsearch-ca.pem
-rw-r--r-- 1 elk elk 190 Mar 20 19:20 kibana.keystore
-rw-r--r-- 1 elk elk 5.2K Mar 20 17:26 kibana.yml.bak
-rwxr-xr-x 1 elk elk 5.2K Jul 9 2022 kibana.yml_20220709073134.bak
-rwxr-xr-x 1 elk elk 216 Jul 9 2022 node.options
  • kibana.yml add the following line to enable TLS for inbound connections
cat >> /usr/local/kibana/config/kibana.yml <<EOF

server.ssl.enabled: true

EOF


# Modify permissions
root@ubuntu-x64_02:/usr/local/elasticsearch9201# chown -R elk:elk /usr/local/kibana
  • After making these changes, you must always access Kibana over HTTPS. For example, https://.com.
server.publicBaseUrl: "https://192.168.88.12:5601"
  • start kibana
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl restart kibana.service
  • check status
root@ubuntu-x64_02:/usr/local/elasticsearch9201# systemctl status kibana.service
● kibana.service - kibana
   Loaded: loaded (/lib/systemd/system/kibana.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2023-03-23 15:26:50 CST; 6s ago
  Process: 4306 ExecStop=/usr/bin/kill -15 $MAINPID (code=exited, status=203/EXEC)
 Main PID: 4312 (node)
    Tasks: 18
   Memory: 200.5M
      CPU: 6.855s
   CGroup: /system.slice/kibana.service
           ├─4312 /usr/local/kibana/bin/../node/bin/node /usr/local/kibana/bin/../src/cli/dist
           └─4325 /usr/local/kibana/node/bin/node --preserve-symlinks-main --preserve-symlinks /usr/local/kibana/src/cli/dist

Mar 23 15:26:50 ubuntu-x64_02 systemd[1]: kibana.service: Unit entered failed state.
Mar 23 15:26:50 ubuntu-x64_02 systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 23 15:26:50 ubuntu-x64_02 systemd[1]: Started kibana.
  • Log in to kibana and enter the elastic:123456 login password. Since the CA is self-signed, the browser will prompt that it has not been verified, and the development environment can be used

Summary

  • This scheme builds on the base security scheme and uses TLS to secure all HTTP traffic. In addition to configuring TLS on the transport interface of the Elasticsearch cluster, you can also configure TLS on the HTTP interfaces of Elasticsearch and Kibana. If you need mutual (mutual) TLS over the HTTP layer, you need to configure mutual authenticated encryption.