Troubleshooting TLS-enabled connections (2023)

general description

This guide covers a methodology and some tools that can help diagnose TLS connectivity problems and errors (TLS alerts). Continue accompanying the main tourTLS en RabbitMQ. The strategy is to test the required components with an alternative TLS implementation by process of elimination to identify the offending side (client or server).

Note that this process is not guaranteed to identify the problem if the interaction between two specific components is responsible for the problem.

The steps recommended in this guide are:

• To verifyeffective configuration
• Make sure the knotlisten for TLS connections
• To verifyfile permissions
• To verifyCompatibility with TLS in Erlang/OTP
• Verify certificate/key pairs and try an alternate TLS client or serverwith OpenSSL command line tools
• Check if it is available and configuredcipher suitesand options for using certificate keys
• Check client connectionswith a TLS termination proxy
• Finally, test a real client connection against a real server connection again

It is important to check this when testing with a RabbitMQ node and/or a real RabbitMQ client.recordsfor both the server and the client.

Verify the effective configuration of the node

Configuring a RabbitMQ node with TLS requires changing the configuration. Before attempting any other TLS troubleshooting steps, it is important to verify the location of the configuration file and the actual configuration (if node loaded it successfully). Seesetup guidefor details.

Check TLS listeners (ports)

This step verifies that the broker is listening on theexpected port(s), such as 5671 for AMQP 0-9-1 and 1.0, 8883 for MQTT, etc.

To verify that TLS has been enabled on the node, useRabbitmq-Diagnosislistenersor thelistenerssection oneRabbitmq-Diagnosisstate.

The listening area looks like this:

Interface: [::], Port: 25672, Protocol: Clustering, Purpose: Inter-node communication and CLI tool Interface: [::], Port: 5672, Protocol: amqp, Purpose: AMQP 0-9-1 and AMQP 1.0 interface : [::], Port: 5671, Protocol: amqp/ssl, Purpose: AMQP 0-9-1 and AMQP 1.0 over TLSInterface: [::], Port: 15672, Protocol: http, Purpose: HTTP APIInterface: [ : :], Port: 15671, Protocol: https, Purpose: HTTP API over TLS (HTTPS) Interface: [::], Port: 1883, Protocol: mqtt, Purpose: MQTT

In the example above, there are 6 TCP listeners on the node. Two of them accept TLS-enabled connections:

• Communication between the node and the CLI tool on the port25672
• Listen AMQP 0-9-1 (and 1.0 if enabled) for non-TLS connections on the port5672
• Listen AMQP 0-9-1 (and 1.0 if enabled) for TLS-enabled connections on the port5671
• HTTP-APIListeners on ports 15672 (HTTP) and 15671 (HTTPS)
• MQTTListener for non-TLS 1883 connections

If the above steps are not possible, check the nodeslog filemay be a reasonable alternative. It should contain an entry about an enabled TLS listener that looks like this:

2018-09-02 14:24:58.611 [info] <0.664.0> TCP listener started on [::]:56722018-09-02 14:24:58.614 [info] <0.680.0> SSL listener started the [ ::]:5671

If the node is configured to use TLS but no message similar to the above is logged, the configuration file may have been placed in the wrong location and not read by the agent, or the node may not have rebooted afterwards of the changes in the configuration file. Watch thesettings pagefor details on how to check the configuration file.

tools likelsofYnet statuscan be used to check which ports a node is listening on, as described inNetwork TroubleshootingGuide.

Check the file permissions for the CA certificate, private key, and package

RabbitMQ should be able to read your configured CA certificate bundle, server certificate, and private key. The files must exist and have the appropriate permissions. Incorrect permissions (for example, files owned byrootor another root account that installed them) is a very common problem with TLS configurations.

On Linux, BSD, and MacOS, directory permissions can also affect the node's ability to read files.

If the certificate or private key files are unreadable or not present, the node will not accept TLS-enabled connections or TLS connections will simply be blocked (behavior differs between Erlang/OTP versions).

Andnew style settings formatused to configure certificate paths and private keys, node will check if the files are present at boot time and refuse to start if they are not.

Check TLS support in Erlang

Another important requirement for establishing TLS connections with the broker is TLS support in the broker. Confirm that Erlang VM supports TLS by running it

Rabbitmq-diagnostics --silent tls_versions

O en Windows

rabbitmq-diagnostics.bat --silent tls_versions

The output will look like this:

tlsv1.2tlsv1.1tlsv1sslv3

With versions that do not offerRabbitmq-diagnostics tls_versions, to use

rabbitmqctl eval 'ssl:versiones().'

O en Windows

rabbitmqctl.bat eval 'ssl:versiones().'

The output in this case looks like this:

[{ssl_app,"9.1"}, {compatible,['tlsv1.2','tlsv1.1',tlsv1]}, {compatible_dtls,['dtlsv1.2',dtlsv1]}, {verfügbar,['tlsv1. 2','tlsv1.1',tlsv1,sslv3]}, {verfügbar_dtls,['dtlsv1.2',dtlsv1]}]

If an error is reported, confirm that the Erlang/OTP installationincludes TLS support.

It is also possible to list available cipher suites on a node:

rabbitmq-diagnostics cipher_suites --format openssl --silent

On Windows:

rabbitmq-diagnostics.bat cipher_suites --format openssl --silent

It is also possible to check which versions of TLS are supported by the local Erlang runtime. walk towards herErl(owerl.exeon Windows) at the command line to open an Erlang shell and type

%% trailing dot is significant!ssl:versions().

Note that this will report the supported versions on the local node (for the runtime found inFAR), which may differ from that used by the examined RabbitMQ nodes.

OpenSSLs_clientYs_serverare commonly used command line tools that can be used to test TLS connections and certificate/key pairs. They help isolate problems when testing with alternative TLS client and server implementations. For example, when a specific TLS client works fine withs_serverbut not a RabbitMQ node, the cause is probably server side. Likewise if as_clientThe client can successfully connect to a RabbitMQ node, but another client cannot. It is the client's configuration that should be closely examined first.

The following example tries to confirm that the certificates and keys can be used to establish a TLS connection by establishing a connections_clientcustomer to ones_serverServer in two separate shells (terminal windows).

The example assumes that you have the followingCertificate and key files(These file names are used bytls-gen):

In a terminal window or tab, run the following command:

openssl s_server -accept 8443 \ -cert server_certificate.pem -key server_key.pem -CAfile ca_certificate.pem

An OpenSSL is starteds_serverusing the provided CA certificate bundle, server certificate, and private key. It is used to verify the trust of certificates with test TLS connections to this sample server.

In another terminal window, run and replace the following commandCN_NAMEwith the expected hostname orCNCertificate name:

openssl s_client -connect localhost:8443 \ -cert client_certificate.pem -key client_key.pem -CAfile ca_certificate.pem \ -verify 8 -verify_hostname CN_NAME

Opens a new TLS connection to the sample TLS server started earlier. you can skip those-verify_hostnameargument, but OpenSSL no longer performs this check.

If the certificates and keys were created successfully, a TLS connection output will appear on both tabs. There is now a connection between the sample client and the sample server, similar totelnet.

If hechain of trustcould be set, the second terminal will display a verification confirmation with the verification code.0:

Check return code: 0 (ok)

As with command line tools, non-zero code reports some kind of error.

If an error is reported, confirm that the certificates and keys were generated correctly and that a matching certificate/private key pair is being used. In addition, certificates can have theirRestricted usage scenariosin generation time. This means that a server such as a RabbitMQ node will reject a certificate intended for clients to use for self-authentication.

For environments where self-signed certificates are appropriate, we recommend usingtls-genfor generations.

Validate available cipher suites

RabbitMQ nodes and clients can be restrictedcipher suitesIts use is allowed during the TLS handshake. It is important to make sure that the two parties share some cipher suites, otherwise the handshake will fail.

The certificate's key usage properties can also restrict which cipher suites can be used.

VerConfigure cipher suitesYExtensions for the use of public keysin the main TLS guide for more information.

ciphers openssl -v

Lists all cipher suites supported by the local build of OpenSSL.

Try to establish a TLS connection to a RabbitMQ node

Once a RabbitMQ node has been configured to listen on a TLS port, OpenSSLs_clientit can be used to test the establishment of the TLS connection, this time against the node. This check determines whether the broker is likely to be configured correctly without the need to configure a RabbitMQ client. The tool can also be useful for comparing the behavior of different customers. The example assumes a running node.facility hostAndStandard TLS port for AMQP 0-9-1 and AMQP 1.0, 5671:

openssl s_client -connect localhost:5671 -cert client_certificate.pem -key client_certificate.pem -CAfile ca_certificate.pem

The output should be similar to the case where port 8443 was used. The node log file shouldcontain a new entry when the connection is established:

2018-09-27 15:46:20 [info] <0.1082.0> Accepting AMQP connection <0.1082.0> (127.0.0.1:50915 -> 127.0.0.1:5671)2018-09-27 15:46:20 [info] <0.1082.0> connection <0.1082.0> (127.0.0.1:50915 -> 127.0.0.1:5671): authenticated user 'user' and access granted to vhost 'virtual_host'

The node expects clients to perform a protocol handshake (AMQP 0-9-1, AMQP 1.0, etc.). If this does not happen within a short period of time (10 seconds by default for most protocols), the node closes the connection.

Validate client connections with Stunnel

sneak outis a tool that can be used to validate TLS-enabled clients. In this configuration, clients establish a secure connection to Stunnel, which forwards the decrypted data to a "normal" port on the broker (for example, 5672 for AMQP 0-9-1 and AMQP 1.0). This provides some assurance that the client's TLS configuration is correct, regardless of the broker's TLS configuration.

sneak outit is a specialized proxy. In this example, it runs in daemon mode on the same host as the broker. The following discussion assumes that stunnel is only used temporarily. It is also possible to use Stunnel to perform TLS termination, but that is beyond the scope of this guide.

In this examplesneak outconnects to the broker's unencrypted port (5672) and accepts TLS connections from TLS-enabled clients on port 5679.

Parameters are passed through a configuration file calledstunnel.conf. It has the following content:

Vordergrund = ja[rabbit-amqp]connect = localhost:5672accept = 5679cert = client/key-cert.pemdebug = 7

sneak outstarts as follows:

cat client_key.pem client_certificate.pem > client/key-cert.pemstunnel stunnel.conf

sneak outrequires a certificate and associated private key. The certificate and private key files must be concatenated as shown aboveGatoDomain.sneak outrequires that the key is not password protected. TLS enabled clients should now be able to connect to port 5679 and any TLS errors will show up in the console wheresneak outit started.

Validate the connection from the RabbitMQ client to the RabbitMQ node

Assuming none of the previous steps resulted in any errors, you can safely connect the tested TLS-enabled client to the broker's TLS-enabled port and ensure that any running OpenSSL is stopped.s_serverosneak outinstances first.

Certificate chains and verification depth

When using a client certificatesigned by an intermediary certificate authority, it may be necessary to configure the RabbitMQ server to use a higher onecheck depth.

Insufficient verification depth causes TLS peer verification to fail.

Understanding TLS connection protocol errors

Many of the previous steps generate new entries in the broker log file. These entries, along with the diagnostic output from the commands in the console, should help identify the cause of TLS-related errors. Below is a list of the most common error entries: