Meilisearch mTLS Interoperability Issue with Go Client (TLS Error Decoding Message)

[hitesh22rana]

Description
Meilisearch server configured with mTLS (mutual TLS) using self-signed certificates fails to complete the TLS handshake when accessed via the official Go SDK (meilisearch-go). The client reports a tls: error decoding message error during the connection attempt. This issue appears to start from versions after v1.15.0 (e.g., v1.15.1, v1.16.2 and later), likely due to the upgrade of the underlying rustls library to v0.23.x, which defaults to the aws-lc-rs cryptography backend. This backend has known interoperability problems with Go's crypto/tls library, as documented in rustls issue rustls/rustls#1912.

The problem occurs specifically in mTLS setups; curl requests (using OpenSSL) succeed without issues. Forcing the rustls backend to "ring" (via a Cargo.toml patch during build) resolves the compatibility issue, allowing the Go client to connect successfully.

Expected behavior
The Go client should successfully establish a TLS connection to the Meilisearch server over HTTPS with mTLS enabled, perform operations like search queries, and return results without handshake errors, matching the behavior of curl.

Current behavior
The Go client panics with the error:

panic: MeilisearchCommunicationError unable to execute request (path "POST /indexes/logs/search" with method "Search"): Post "https://0.0.0.0:7700/indexes/logs/search": local error: tls: error decoding message

This happens even when forcing TLS 1.2 in the client's tls.Config (via MinVersion and MaxVersion). The server is reachable and functional, as confirmed by successful curl requests to the same endpoints.

Screenshots or Logs

• Error from Go Client:

panic: MeilisearchCommunicationError unable to execute request (path "POST /indexes/logs/search" with method "Search"): Post "https://0.0.0.0:7700/indexes/logs/search": local error: tls: error decoding message

goroutine 1 [running]:
main.main()
        /Users/hiteshrana/Work/chronoverse/test.go:729 +0x538
exit status 2

• Successful curl Health Check:

curl --cert certs/clients/client.crt --key certs/clients/client.key --cacert certs/ca/ca.crt https://127.0.0.1:7700/health | jq
{
  "status": "available"
}

• Successful curl Search Query:

curl --http2 --cert certs/clients/client.crt \
             --key certs/clients/client.key \
             --cacert certs/ca/ca.crt \
             -H "Authorization: Bearer 35b09c3c7b1001ed1fbed7db29a155d0201ab22d527b41bfba9003da7bb3b404" \
             -H "Content-Type: application/json" \
             -X POST https://127.0.0.1:7700/indexes/logs/search \
             -d '{
               "q": "{User:",
               "filter": "user_id = \"01989764-9541-7938-a03a-2d2fbf8e8abf\" AND workflow_id = \"0198c284-21aa-7bf8-9ace-1aa46ba59efd\" AND job_id = \"019923e9-14e6-7ed9-afc9-795fea3d9475\"",
               "limit": 1,
               "attributesToRetrieve": ["message", "sequence_num", "stream", "timestamp"],
               "attributesToHighlight": ["message"]
             }' | jq
{
  "hits": [
    {
      "message": "1) {User: torvalds, Profile: https://github.com/torvalds}",
      "sequence_num": 4,
      "stream": "stdout",
      "timestamp": 1758378448808961000,
      "_formatted": {
        "message": "1) {<em>User</em>: torvalds, Profile: https://github.com/torvalds}",
        "sequence_num": "4",
        "stream": "stdout",
        "timestamp": "1758378448808961000"
      }
    }
  ],
  "query": "{User:",
  "processingTimeMs": 14,
  "limit": 1,
  "offset": 0,
  "estimatedTotalHits": 200
}

• Verbose curl with TLS 1.2 (Negotiates to 1.3 Successfully):

curl -v --tlsv1.2 --cert certs/clients/client.crt --key certs/clients/client.key --cacert certs/ca/ca.crt https://127.0.0.1:7700/health
*   Trying 127.0.0.1:7700...
* Connected to 127.0.0.1 (127.0.0.1) port 7700
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: certs/ca/ca.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, CERT verify (15):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=Chronoverse meilisearch
*  start date: Sep 21 08:27:44 2025 GMT
*  expire date: Sep 21 08:27:44 2026 GMT
*  subjectAltName: host "127.0.0.1" matched cert's IP address!
*  issuer: CN=Chronoverse CA
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:7700/health
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 127.0.0.1:7700]
* [HTTP/2] [1] [:path: /health]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /health HTTP/2
> Host: 127.0.0.1:7700
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/2 200
< content-length: 22
< vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
< content-type: application/json
< date: Sun, 21 Sep 2025 08:33:28 GMT
<
* Connection #0 to host 127.0.0.1 left intact
{"status":"available"}

• Go Client Code (Failing Setup):

import (
	"bytes"
	"crypto/tls"
	"crypto/x509"
	"encoding/json"
	"fmt"
	"os"

	"github.com/meilisearch/meilisearch-go"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
)

func newTLSConfig(certFile, keyFile, caFile string) (*tls.Config, error) {
	cert, err := tls.LoadX509KeyPair(certFile, keyFile)
	if err != nil {
		return nil, status.Errorf(codes.Internal, "failed to load client key pair: %v", err)
	}

	caCert, err := os.ReadFile(caFile)
	if err != nil {
		return nil, status.Errorf(codes.Internal, "failed to read CA certificate: %v", err)
	}

	caCertPool := x509.NewCertPool()
	if !caCertPool.AppendCertsFromPEM(caCert) {
		return nil, fmt.Errorf("failed to append CA cert")
	}

	return &tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs:      caCertPool,
		MinVersion:   tls.VersionTLS12,
	}, nil
}

func main() {
	var (
		certFile string = "certs/clients/client.crt"
		keyFile  string = "certs/clients/client.key"
		caFile   string = "certs/ca/ca.crt"
	)
	tlsConfig, err := newTLSConfig(certFile, keyFile, caFile)
	if err != nil {
		panic(err)
	}

	client := meilisearch.New(
		"https://0.0.0.0:7700",
		meilisearch.WithAPIKey("35b09c3c7b1001ed1fbed7db29a155d0201ab22d527b41bfba9003da7bb3b404"),
		meilisearch.WithCustomClientWithTLS(tlsConfig),
	)

	var (
		UserID     string = "user_id"
		WorkflowID string = "workflow_id"
		JobID      string = "job_id"
	)

	var filters string = fmt.Sprintf(
		`user_id = "%s" AND workflow_id = "%s" AND job_id = "%s" AND (timestamp > %d OR (timestamp = %d AND sequence_num > %d))`,
		UserID,
		WorkflowID,
		JobID,
		0,
		0,
		0,
	)

	searchRes, err := client.Index("logs").Search(
		`"level":"info","timestamp":"`,
		&meilisearch.SearchRequest{
			AttributesToRetrieve:  []string{"message", "sequence_num", "stream", "timestamp"},
			AttributesToHighlight: []string{"message"},
			Filter:                filters,
			Limit:                 100,
		},
	)
	if err != nil {
		panic(err)
	}

	for _, hit := range searchRes.Hits {
		var result map[string]any
		if formattedRaw, ok := hit["_formatted"]; ok {
			if err := json.Unmarshal(formattedRaw, &result); err != nil {
				// fallback to raw hit
				result = make(map[string]any)
				for k, v := range hit {
					var val any
					_ = json.Unmarshal(v, &val)
					result[k] = val
				}
			}
		} else {
			result = make(map[string]any)
			for k, v := range hit {
				var val any
				_ = json.Unmarshal(v, &val)
				result[k] = val
			}
		}

		printJSONNoEscape(result)
	}
}

func printJSONNoEscape(v any) {
	buf := &bytes.Buffer{}
	encoder := json.NewEncoder(buf)
	encoder.SetEscapeHTML(false)
	encoder.SetIndent("", "  ")
	if err := encoder.Encode(v); err != nil {
		fmt.Fprintln(os.Stderr, "Error encoding JSON:", err)
		return
	}
	fmt.Print(buf.String())
}

• Meilisearch Docker Run Command:

docker run -it --rm \ 
            -p 7700:7700 \
            -e MEILI_MASTER_KEY='35b09c3c7b1001ed1fbed7db29a155d0201ab22d527b41bfba9003da7bb3b404' \
            -e MEILI_SSL_AUTH_PATH='/certs/ca/ca.crt' \
            -e MEILI_SSL_CERT_PATH='/certs/meilisearch/meilisearch.crt' \
            -e MEILI_SSL_KEY_PATH='/certs/meilisearch/meilisearch.key' \
            -v $(pwd)/meili_data:/meili_data \
            -v $(pwd)/certs:/certs \
            getmeili/meilisearch:latest

• (If relevant) Certificate Generation Script:

mkdir -p /certs/ca

  SERVICES="meilisearch"
  for svc in $$SERVICES; do
    mkdir -p /certs/$$svc
  done

# Generate CA if not exists
  if [ ! -f /certs/ca/ca.key ]; then
    echo "🛡️ Generating CA certificate..."
    openssl genrsa -out /certs/ca/ca.key 4096
    openssl req -x509 -new -nodes -key /certs/ca/ca.key -sha256 \
      -days 365 -out /certs/ca/ca.crt \
      -subj "/CN=Chronoverse CA"
    echo "✅ CA certificate created"
  fi

for svc in $$SERVICES; do
    CERT_PATH="/certs/$$svc"
    echo "🔧 Generating certificate for $$svc..."
    openssl genrsa -out "$$CERT_PATH/$$svc.key" 4096
    openssl req -new -key "$$CERT_PATH/$$svc.key" \
      -out "$$CERT_PATH/$$svc.csr" \
      -subj "/CN=Chronoverse $$svc"

    echo "subjectAltName=IP:0.0.0.0,IP:127.0.0.1,DNS:$$svc" > "$$CERT_PATH/$$svc-ext.cnf"

    openssl x509 -req -in "$$CERT_PATH/$$svc.csr" \
      -CA /certs/ca/ca.crt -CAkey /certs/ca/ca.key \
      -CAcreateserial -out "$$CERT_PATH/$$svc.crt" -days 365 \
      -extfile "$$CERT_PATH/$$svc-ext.cnf"

    rm "$$CERT_PATH/$$svc.csr" "$$CERT_PATH/$$svc-ext.cnf"

    echo "✅ Certificate created for $$svc"
  done

# Client certificate for mTLS
  echo "🔐 Generating client certificate for mTLS..."
  mkdir -p /certs/clients
  openssl genrsa -out /certs/clients/client.key 4096
  openssl req -new -key /certs/clients/client.key \
    -out /certs/clients/client.csr \
    -subj "/CN=Chronoverse Client"

  echo "subjectAltName=DNS:client" > /certs/clients/client-ext.cnf

  openssl x509 -req -in /certs/clients/client.csr \
    -CA /certs/ca/ca.crt -CAkey /certs/ca/ca.key \
    -CAcreateserial -out /certs/clients/client.crt -days 365 \
    -extfile /certs/clients/client-ext.cnf

  rm /certs/clients/client.csr /certs/clients/client-ext.cnf
  rm /certs/ca/ca.srl
  echo "✅ Client certificate created"

  echo "🔐 Setting permissions for all certificates..."

  chmod 444 /certs/ca/ca.crt
  chmod 444 /certs/ca/ca.key

  for svc in $$SERVICES; do
    chmod 444 /certs/$$svc/$$svc.crt
    chmod 444 /certs/$$svc/$$svc.key
  done

  chmod 444 /certs/clients/client.crt
  chmod 444 /certs/clients/client.key

  echo "✅ Permissions set successfully"
  echo "🎉 TLS certificates initialized successfully!"

Environment (please complete the following information):

• OS: macOS 15 Tahoe
• Meilisearch version: latest v1.21.0
• meilisearch-go version: v0.34.0

Additional Notes:

• The issue does not occur when Meilisearch is run without SSL/TLS enabled (i.e., over plain HTTP without mTLS). In that case, the Go client connects and performs operations successfully using the standard configuration without a custom TLS setup.
• The issue is reproducible with self-signed certs generated via OpenSSL as shown.
• Workaround: In my opinion building Meilisearch from source with a Cargo.toml patch to force rustls to use the "ring" backend might resolves the problem: (haven't tried yet).

[patch.crates-io]
rustls = { git = "https://github.com/rustls/rustls", rev = "a8d875fb2096e2eee57735189868db7d7e2ba0c0", default-features = false, features = ["ring", "logging", "std"] }

• Reference: rustls GitHub issue discussing aws-lc-rs vs. Go TLS compatibility: AWS LC fails against golang TLS server while ring works fine rustls/rustls#1912
• No existing issues found in Meilisearch repo matching this exact problem.

[ja7ad]

@ElyarSadig Possible check issue?

[ElyarSadig]

@ja7ad
I investigated further and found the TLS handshake fails inside Go's standard crypto/tls when Meilisearch is built with the aws-lc-rs backend for rustls; the failure occurs before any HTTP request is sent. Replacing Go's crypto/tls is not practical because nearly all Go HTTP clients use it.
The practical workarounds for this now could be:

• Rebuild Meilisearch to use rustls with the ring backend (this resolves the incompatibility).
• Terminate TLS with a proxy (nginx/Envoy) that uses OpenSSL/ring between the Go client and Meilisearch. It's possible a future Go or aws-lc-rs update will fix the interoperability.

[hitesh22rana]

@ja7ad I investigated further and found the TLS handshake fails inside Go's standard crypto/tls when Meilisearch is built with the aws-lc-rs backend for rustls; the failure occurs before any HTTP request is sent. Replacing Go's crypto/tls is not practical because nearly all Go HTTP clients use it. The practical workarounds for this now could be:

• Rebuild Meilisearch to use rustls with the ring backend (this resolves the incompatibility).
• Terminate TLS with a proxy (nginx/Envoy) that uses OpenSSL/ring between the Go client and Meilisearch. It's possible a future Go or aws-lc-rs update will fix the interoperability.

Thanks, @ElyarSadig.
I agree the root cause is the aws lc rs and Go crypto tls handshake mismatch under mTLS. Using a reverse proxy feels like a stopgap rather than a fix. The better path is to rebuild Meilisearch with rustls and the ring backend so the handshake completes reliably.

Would the team consider a build time switch or official image variant that selects ring for rustls, so users on Go clients can avoid the proxy workaround?

[ja7ad]

@ja7ad I investigated further and found the TLS handshake fails inside Go's standard crypto/tls when Meilisearch is built with the aws-lc-rs backend for rustls; the failure occurs before any HTTP request is sent. Replacing Go's crypto/tls is not practical because nearly all Go HTTP clients use it. The practical workarounds for this now could be:

• Rebuild Meilisearch to use rustls with the ring backend (this resolves the incompatibility).
• Terminate TLS with a proxy (nginx/Envoy) that uses OpenSSL/ring between the Go client and Meilisearch. It's possible a future Go or aws-lc-rs update will fix the interoperability.

Thanks, @ElyarSadig.
I agree the root cause is the aws lc rs and Go crypto tls handshake mismatch under mTLS. Using a reverse proxy feels like a stopgap rather than a fix. The better path is to rebuild Meilisearch with rustls and the ring backend so the handshake completes reliably.

Would the team consider a build time switch or official image variant that selects ring for rustls, so users on Go clients can avoid the proxy workaround?

Thanks for raising this. You’re right that the handshake failures stem from the AWS-LC backend in Rustls and its interaction with Go’s crypto/tls under mTLS. The Rustls team has already addressed this upstream (rustls#1924) by lowering the priority of ECDSA_NISTP521_SHA512, so once that release is available and Meilisearch updates its dependency, Go clients should work without needing a reverse proxy.

As for providing an official ring build: technically Rustls supports a ring backend via feature flags, but we currently default to AWS-LC because it offers stronger performance and FIPS-validated cryptography. Maintaining two parallel image variants (AWS-LC and ring) would expand our support surface and complicate long-term maintenance. For that reason, our plan is to stay on AWS-LC and benefit from the upstream fix rather than publish a separate ring image.

If you need an immediate solution, you can already compile Meilisearch from source with the ring feature enabled in Cargo.toml. Otherwise, once the patched Rustls release is integrated, the default images should interoperate cleanly with Go clients without additional workarounds.