A solution for connecting Android devices to mobile phone H5 to display real-time images (2)

Frontend: A solution for Android devices to connect to mobile phone H5 to display real-time images

Because of the pitfalls encountered in SSL verification

The real-time screen display solution ran well during local joint debugging and testing. However, after the pre-release environment was released, the real-time screen display could not be displayed and the browser reported an error.

Mixed Content: The page at ‘*****’ was loaded over HTTPS, but attempted to connect to the insecure WebSocket endpoint ‘ws://*****’. This request has been blocked; this endpoint must be available over WSS.
(anonymous)
Uncaught DOMException: Failed to construct ‘WebSocket’: An insecure WebSocket connection may not be initiated from a page loaded over HTTPS.

After investigation, both the pre-release and online environments are HTTPS environments. HTTPS is based on an SSL certificate to verify the identity of the server and encrypt the communication between the browser and the server, so the browser may block when certain non-SSL verified resources are called from an HTTPS site. For example, if you use ws://*** to call the WebSocket server or introduce a js file like http://***.js, an error will be reported. The local test is an HTTP environment, so this problem does not occur.

Due to security considerations, the online environment can only be an HTTPS environment, which requires data transmission to be changed to a wss format that supports the SSL protocol.

According to the research results, if you want to use wss, you can first build an nginx proxy to forward wss requests to a WebSocket service that does not support SSL; second, transform the WebSocket service to support the SSL protocol.

nginx

First try to build nginx on the device. In theory, you can download the latest version installation package from the nginx official website and install it through the shell command. However, it was blocked by device permissions during actual installation and could not be installed.

According to the research, nginx can be installed under the Android system through Termux. Termux is an advanced terminal emulator for Android. It is open source and does not require root. It supports apt management software packages. It is very convenient to install software packages. It perfectly supports Python, PHP, and Ruby. , Nodejs, MySQL, etc.

Refer to the Termux advanced terminal installation, usage and configuration tutorial to install Termux and nginx successfully.

?

Modify the nginx configuration. According to the Internet, there are several requirements for using wss in nginx configuration:

  • wss does not support the ip + port connection method and can only be requested through the domain name

  • Later, during the actual test, I found that this item is not necessary. I guess it is because we are building a local WebSocket service in the LAN, not a public network environment, so we can continue to use the ip + port connection method.

  • nginx needs to configure the domain name path, the access path is: wss://domain name/wss/project access

# Establish websocket connection
location /wss/ {
  proxy_pass http://127.0.0.1:own project port number/;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "Upgrade";
  proxy_set_header X-real-ip $remote_addr;
  proxy_set_header X-Forwarded-For $remote_addr;
}

  • nginx needs to configure an SSL certificate. Since it is a local LAN environment, a self-signed certificate needs to be configured.

openssl req -x509 -nodes -days 36500 -newkey rsa:2048 -keyout cert.key -out cert.crt
server {
  listen port;
  server_name domain name address;
  #Adjust to your own certificate
  ssl_certificate /usr/local/nginx/conf/ssl/xxxx.crt;
  ssl_certificate_key /usr/local/nginx/conf/ssl/xxxx.key;
  ssl_session_timeout 5m;

  # Establish websocket connection
  location /wss/ {
    proxy_pass http://127.0.0.1:own project port number/;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-real-ip $remote_addr;
    proxy_set_header X-Forwarded-For $remote_addr;
  }
}

Actual debugging record:

  • nginx can establish China Unicom normally, and non-SSL connection configuration port forwarding can also work normally;

  • Configure domain name to forward IP requests, but the connection fails. My understanding is that nginx itself does not support domain name resolution capabilities and requires DNS service support. But we are in a local LAN environment and need to set up a local DNS server. Later, users will also be required to configure a local DNS server on their mobile phones. Even if it is technically feasible, it also increases user operations;

  • After configuring the SSL self-signed certificate, nginx fails to connect. The reason is unknown.

I later thought and analyzed that the applicable scenarios for configuring nginx suggested on the Internet should be that both HTTPS pages and WebSocket services are assumed to be in the same service environment on the public network; but our actual scenario is that the HTTPS page is served on the public network, and the WebSocket is called within the page. The service is in the local area network and the application scenarios are different, so whether it is domain name proxy or SSL certificate configuration, it is more complicated to implement than the online solution.

Even if the nginx solution is feasible, it still requires the cooperation of the manufacturer to pre-install nginx on the device and support starting nginx at boot.

All things considered, the plan was abandoned.

WebSocket supports SSL

WebSocket supports the SSL protocol and needs to be configured with an SSL certificate when starting the service.

  • Since we are in a LAN environment, we cannot use the public network CA certificate and need to apply for a self-signed certificate;

  • The device is an Android system and does not support JKS format certificates, but only supports BKS format certificates;

keytool -genkeypair -alias socketKey -keyalg RSA -keysize 2048 -validity 36500 -keystore socket_keystore.bks -storetype BKS -provider org.bouncycastle.jce.provider.BouncyCastleProvider -providerpath bcprov-jdk18on-173.jar

This command will generate a 2048-bit RSA key pair and store it in a keystore file named socket_keystore.bks. The socketKey and 36500 parameters are the alias and validity period. -provider and -providerpath are options to the keytool command-line tool that specify the provider and provider path for the keystore.

When actually executing the command, an error occurs:

keytool error: java.security.KeyStoreException: BKS not found

This is due to using the -provider and -providerpath options in the command to specify Bouncy Castle as the provider of the keystore and setting its path to bcprov-jdk18on-173.jar.

BKS is a keystore format provided by Bouncy Castle. If you want to use Bouncy Castle as the keystore provider, you need to install the Bouncy Castle provider in the Java runtime environment. You can download and install the corresponding provider from the official website of Bouncy Castle. The download address is: Bouncy Castle.

After the installation is complete, you can find the bouncycastle.jar file in the jre/lib/security directory under the installation directory of the Java runtime environment. This file contains the provider implementation of Bouncy Castle. Then, you need to add it to the lib/security/java.security file in the installation directory of the Java runtime environment, and add the following line of code:

security.provider.10=org.bouncycastle.jce.provider.BouncyCastleProvider

Among them, 10 is the provider’s priority, which can be replaced with the required priority. Configure bcprov-ext-jdk jar file in Mac environment_bcprov mac-CSDN blog

After the configuration is completed, execute the apply for certificate command and fill in the certificate key, user, and organization information as prompted.

Also attached:

keytool -genkeypair -alias socketkey -keyalg RSA -keysize 2048 -validity 36500 -keystore socket_keystore.jks -storepass 123456

?

Then add support for the SSL protocol to the WebSocket service.

compile "org.java-websocket:Java-WebSocket:1.5.1"

import org.java_websocket.WebSocket
import org.java_websocket.handshake.ClientHandshake
import org.java_websocket.server.DefaultSSLWebSocketServerFactory
import org.java_websocket.server.WebSocketServer
import java.security.KeyStore
import javax.net.ssl.KeyManagerFactory
import javax.net.ssl.SSLContext
import javax.net.ssl.TrustManagerFactory


/**
 * @author zhoucj
 * @since 2023/7/7
 */
class WifiSocketManager : ISocketManager {
    private val TAG = "LocalSocketEngineImpl_WIFI"

    private lateinit var wifiSocketServer: WifiSocketServer

    override fun start() {
        // websocket connection
        Log.i(TAG, "Start ServerSocket...")
        try {
            url?.let {
                wifiSocketServer = WifiSocketServer(8087)
                wifiSocketServer.isReuseAddr = true
                wifiSocketServer.isTcpNoDelay = true
                wifiSocketServer.start()
            }
        } catch (e: Exception) {
            Log.e(TAG, e)
        }
    }

    inner class WifiSocketServer(port: Int) : WebSocketServer(InetSocketAddress(port)) {

        var connect: WebSocket? = null

        ...
        
        override fun onStart() {
            Log.i(TAG, "onStart:")
            try {
                //Add SSL support
                val sslContext = SSLContext.getInstance("TLS")
                val keyStore = KeyStore.getInstance("BKS")

                //Load certificate
                val certInputStream = context.resources.openRawResource(R.raw.socket_keystore)
                keyStore.load(certInputStream, "123456".toCharArray())
                certInputStream.close()

                val trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm())
                trustManagerFactory.init(keyStore)

                val keyManagerFactory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm())
                keyManagerFactory.init(keyStore, "123456".toCharArray())

                sslContext.init(keyManagerFactory.keyManagers, trustManagerFactory.trustManagers, null)

                setWebSocketFactory(DefaultSSLWebSocketServerFactory(sslContext))
            } catch (e: Exception) {
                Log.e(TAG, e)
            }
        }
    }
}

wifiSocketServer.isReuseAddr = true means allowing reuse of previously used ports. Normally, when a port is occupied, the operating system will keep the port in the TIME_WAIT state for a period of time to ensure that all data packets in the network are processed correctly. And if you need to restart a service in a short period of time, the operating system may refuse to allocate a previously used port because the port is still in the TIME_WAIT state. Setting the wifiSocketServer.isReuseAddr property to true allows you to reuse a previously used port, thus avoiding this problem. wifiSocketServer.isTcpNoDelay = true means enabling TCP_NODELAY. This option can disable the Nagle algorithm, thereby sending data more timely. The Nagle algorithm caches small packets and combines them into larger packets before sending them to reduce traffic on the network. But in some cases, this can cause delays in data transmission, as the sender needs to wait for a certain number of small packets before combining them into a larger packet. Enabling TCP_NODELAY disables Nagle’s algorithm, allowing data to be sent in a more timely manner.

Start the service, H5 connection, the browser reports an error, but there is no task error message:

WebSocket connection to ‘wss://xxxxxx’ failed:

No error is reported, and valid information cannot be found online. Try local debugging. Due to device permission restrictions, debugging is not possible, so I wrote a WebSocket service Demo to run debugging on the mobile phone. Track the code execution and find that an exception will be reported when the service receives the request and processes it.

javax.net.ssl.SSLHandshakeException: Read error: ssl=0xb40000702e129658: Failure in SSL library, usually a protocol errorerror:10000416:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN (external/boringssl/src/ssl/tls_record.cc:594 0 xb40000704e12cfb8:0x00000001 )

(Comment here, the WebSocket library actually digests the exception internally without throwing it out, causing the upper layer users to have no idea why the error occurred)

The reason for the error is that the self-signed certificate is a certificate issued or generated by yourself, so it is not recognized and trusted by the browser or client. When using a self-signed certificate, an SSLV3_ALERT_CERTIFICATE_UNKNOWN error may occur during the SSL handshake. This is because the client is unable to verify the validity and authenticity of the self-signed certificate, causing the SSL handshake to fail.

To resolve this issue, you can add the self-signed certificate to the Trusted Root Certification Authorities so that clients will trust the certificate.

When wss accesses the self-signed certificate address, the browser will still block access and no error message will be reported. However, when accessing an HTTPS address using a self-signed certificate, you will be prompted and allowed to trust the user.

Manually access https://xxxxx with the same address as wss://xxxxx, and select ‘Yes’ when prompted to trust the certificate (each browser is different, some browsers will not prompt whether to trust the certificate, but will display it on the page Interception message asking if you want to continue access).

Then open the real-time image display page in the same browser environment, the image is displayed normally, and it is successful.

Since there is an additional step that requires the user to trust the certificate, the product needs to modify the interaction logic.

?

Thinking further, when developing Native, the client can force trust of all certificates through code without user operation. Is H5 also possible?

According to online inquiries, someone has provided the ability to add configuration when wss is requested.

var socket = new WebSocket('wss://example.com', null, {
   rejectUnauthorized: false
});

But there are two problems here.

First, online answers (including chatgpt) all omit the intermediate parameter null, causing execution errors.

DOMException: Failed to construct ‘WebSocket’: The subprotocol ‘[object Object]’ is invalid.

This is because in H5, when creating a WebSocket object, you can pass three parameters:

  1. url (required): Specifies the URL of the WebSocket server.

  2. protocol (optional): Specifies the protocol or subprotocol to use. Can be a string or an array of strings.

  3. options (optional): Specify WebSocket options such as subprotocol, certificate trust, and timeout. This should be an object containing options properties.

When the third parameter exists, the second parameter cannot be omitted. Otherwise, an error will be reported.

Second, in fact, the { rejectUnauthorized: false } option is only available in Node.js and is not supported in browsers. Writing this in the browser will cause originally connectable requests to become unconnectable.

Therefore this configuration solution is not feasible.

?

Here chatgpt also seriously provides another wrong answer. Convert the certificate file to a Base64 encoded string. Add the Base64-encoded string to the JavaScript code and pass it as the second parameter to the WebSocket constructor as a string.

const cert = `
  -----BEGIN CERTIFICATE-----
  Base64-encoded certificate content
  -----END CERTIFICATE-----
`;

const socket = new WebSocket('wss://example.com', cert);

In fact, we can know from the parameter definition that the second parameter only supports the protocol used and does not support certificates.

Attached is the command to convert the BKS certificate into a Base64 string.

#Convert the BKS certificate to PEM format first
keytool -importkeystore -srckeystore socket_keystore.bks -srcstoretype BKS -destkeystore socket_keystore.p12 -deststoretype PKCS12

openssl pkcs12 -in socket_keystore.p12 -out socket_keystore.pem -nodes

#Convert PEM format certificate to Base64 encoded string
cat socket_keystore.pem | base64

Think about it carefully, whether to trust the certificate is the behavior of the browser. H5 pages do not have the ability to bypass the browser to decide whether to trust the certificate. They can only modify the logic implementation of the browser.

Final plan

According to previous program research, the technical stuck points for H5 connected devices to display real-time images are:

  1. Both the pre-release and online environments are HTTPS environments. Browsers may block calls to certain non-SSL verified resources on HTTPS sites. Therefore WebSocket must also support the SSL protocol.

  2. Since the mobile phone and device are connected in a LAN environment, enabling SSL support for the WebSocket service cannot use a public network CA certificate, and you need to apply for a self-signed certificate.

  3. Browsers cannot directly trust self-signed certificates and require users to manually trust them, which requires high user operation requirements. Moreover, during the test, it was also found that different browsers have different prompts for “whether to trust the certificate”, and Apple phones do not display this prompt at all.

  4. It was further discovered that the underlying network library of the “Self-Developed App” disabled self-signed certificates. If self-signed certificate support needs to be enabled, the brother team needs to modify the underlying network library, which is difficult to promote.

After investigating various solutions, we found that whether we are pushing the Brother team to enable self-signed certificate support, or pushing the Brother team to provide native layer websocket connections to bypass the HTTPS restrictions of H5, or transferring data through the server, considering development investment, user experience, etc. , are not very suitable.

Later, we realized that since the root cause of the problem was that HTTPS restricted the need for SSL verification, then if we did not use HTTPS but used an HTTP environment, the problem would not be solved.

After communicating with security classmates, it is possible to use HTTP environment to only display this page on the real-time screen (not involving user data).

Therefore, the final solution is:

  • Since the HTTPS environment does not allow direct opening of HTTP pages, the real-time screen page is displayed by opening a new browser.

  • This page uses HTTP requests, which bypasses the certificate verification process and can transmit data through ordinary WebSocket.

  • This new window should be restricted to only viewing real-time images, without transmitting any user data or performing any business interactions. All business operations must still return to the original HTTPS environment.