Skip to content

25.3.8-fips: openssl-cmake: shim fixes for MongoDB#1451

Merged
Enmk merged 2 commits intoreleases/25.3.8-fipsfrom
mkmkme/fips/mongodb
Feb 27, 2026
Merged

25.3.8-fips: openssl-cmake: shim fixes for MongoDB#1451
Enmk merged 2 commits intoreleases/25.3.8-fipsfrom
mkmkme/fips/mongodb

Conversation

@mkmkme
Copy link
Copy Markdown
Collaborator

@mkmkme mkmkme commented Feb 26, 2026

This commit fixes three issues that led to MongoDB integration tests failing. I tried to explain each of the fixes with a proper documentation to the function/macro.

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • Tiered Storage (2h)

@mkmkme
Copy link
Copy Markdown
Collaborator Author

mkmkme commented Feb 27, 2026

@codex review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7223ea8e64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@mkmkme
Copy link
Copy Markdown
Collaborator Author

mkmkme commented Feb 27, 2026

@codex review

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2ccd7df1c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +123 to +124
SSL *ssl = (SSL *)BIO_get_data(b);
if (!ssl)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Validate BIO type before casting BIO_get_data to SSL

BIO_do_handshake now assumes b is always an SSL BIO and immediately casts BIO_get_data(b) to SSL *, but BIO_get_data for non-SSL BIOs returns other internal structs. If a caller invokes handshake on a wrapped chain head (e.g., a filter/connect BIO above SSL), SSL_get_rbio/SSL_do_handshake will run on an invalid pointer and can crash. The previous BIO_ctrl(...BIO_C_DO_STATE_MACHINE...) path delegated through BIO ctrl handlers instead of dereferencing arbitrary BIO payloads.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only call site for this function in the entire codebase is mongoc-stream-tls-openssl.c:569, and openssl->bio is assigned at line 793 from bio_ssl, which is the SSL BIO created by BIO_new_ssl.

Comment on lines 103 to +104
#define BIO_get_ssl(b, sslp) \
BIO_ctrl(b, BIO_C_GET_SSL, 0, (char *)(sslp))
#define BIO_do_handshake(b) \
BIO_ctrl(b, BIO_C_DO_STATE_MACHINE, 0, NULL)
(*(SSL **)(sslp) = (SSL *)BIO_get_data(b))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve BIO_ctrl chain semantics in BIO_get_ssl shim

Replacing BIO_get_ssl with direct BIO_get_data(b) extraction drops BIO_ctrl forwarding semantics and only works when b is the SSL BIO itself. On chained BIOs, this macro will store the top BIO's private data pointer into sslp rather than the SSL object, which can later cause invalid SSL operations. OpenSSL's macro (BIO_ctrl(...BIO_C_GET_SSL...)) supports wrappers by routing through ctrl handlers.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In our use case (MongoDB contrib), every call for BIO_get_ssl is proven to be on the SSL BIO directly and called right after BIO_new_ssl

Copy link
Copy Markdown
Member

@Enmk Enmk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Enmk Enmk merged commit 24e5691 into releases/25.3.8-fips Feb 27, 2026
137 of 167 checks passed
@svb-alt svb-alt added fips Work related to Altinity FIPS releases fips-25.3 labels Mar 11, 2026
@DimensionWieldr
Copy link
Copy Markdown
Collaborator

AI audit returned 1 defect (potential memory leak). I think this one needs review.

Confirmed defects



  Medium: BIO reference count leak in `BIO_do_handshake`

  • Impact: The transport BIO (bio_mongoc_shim) is leaked on every MongoDB TLS connection teardown. Each leaked BIO is ~64-128 bytes. The freed-but-referenced BIO_METHOD (freed
    separately at mongoc-stream-tls-openssl.c:93) creates a dangling pointer inside the leaked BIO, though it is never dereferenced.
  • Anchor: contrib/openssl-cmake/include/evp_API_shim.h / BIO_do_handshake() lines 127-129
  • Trigger: Any MongoDB TLS connection in FIPS mode. Lifecycle: mongoc_stream_tls_openssl_new (line 766: BIO_push) → _mongoc_stream_tls_openssl_handshake (line 569:
    BIO_do_handshake) → _mongoc_stream_tls_openssl_destroy (line 90: BIO_free_all).
  • Why defect: SSL_set_bio(ssl, bio, bio) with rbio == wbio internally calls BIO_up_ref(rbio) once (OpenSSL ssl_lib.c:1557-1558: "If the two arguments are equal then one fewer 
    reference is granted by the caller than we want to take"). SSL_free later calls BIO_free_all on both rbio and wbio (2 decrements). The BIO chain walk adds 1 more decrement.
    Total decrements at cleanup = 3. The shim adds 2 manual BIO_up_ref + 1 internal = 3 extra refs on top of the initial refcount of 1 → final refcount = 4 − 3 = 1 (never freed).
    Only 1 manual BIO_up_ref is needed.
  • Fix direction: Remove one BIO_up_ref(BIO_next(b)) call (line 127 or 128).
  • Regression test: Open and close MongoDB TLS connections in FIPS mode under LSAN/ASAN; verify no leaked BIO allocations.


   contrib/openssl-cmake/include/evp_API_shim.h lines 127-129

          BIO_up_ref(BIO_next(b));
          BIO_up_ref(BIO_next(b));   // one too many
          SSL_set_bio(ssl, BIO_next(b), BIO_next(b));

@DimensionWieldr DimensionWieldr added the verified-with-issues Verified by QA and issues found. label Apr 2, 2026
@mkmkme
Copy link
Copy Markdown
Collaborator Author

mkmkme commented Apr 2, 2026

AI audit returned 1 defect (potential memory leak). I think this one needs review.

Confirmed defects



  Medium: BIO reference count leak in `BIO_do_handshake`

  • Impact: The transport BIO (bio_mongoc_shim) is leaked on every MongoDB TLS connection teardown. Each leaked BIO is ~64-128 bytes. The freed-but-referenced BIO_METHOD (freed
    separately at mongoc-stream-tls-openssl.c:93) creates a dangling pointer inside the leaked BIO, though it is never dereferenced.
  • Anchor: contrib/openssl-cmake/include/evp_API_shim.h / BIO_do_handshake() lines 127-129
  • Trigger: Any MongoDB TLS connection in FIPS mode. Lifecycle: mongoc_stream_tls_openssl_new (line 766: BIO_push) → _mongoc_stream_tls_openssl_handshake (line 569:
    BIO_do_handshake) → _mongoc_stream_tls_openssl_destroy (line 90: BIO_free_all).
  • Why defect: SSL_set_bio(ssl, bio, bio) with rbio == wbio internally calls BIO_up_ref(rbio) once (OpenSSL ssl_lib.c:1557-1558: "If the two arguments are equal then one fewer 
    reference is granted by the caller than we want to take"). SSL_free later calls BIO_free_all on both rbio and wbio (2 decrements). The BIO chain walk adds 1 more decrement.
    Total decrements at cleanup = 3. The shim adds 2 manual BIO_up_ref + 1 internal = 3 extra refs on top of the initial refcount of 1 → final refcount = 4 − 3 = 1 (never freed).
    Only 1 manual BIO_up_ref is needed.
  • Fix direction: Remove one BIO_up_ref(BIO_next(b)) call (line 127 or 128).
  • Regression test: Open and close MongoDB TLS connections in FIPS mode under LSAN/ASAN; verify no leaked BIO allocations.


   contrib/openssl-cmake/include/evp_API_shim.h lines 127-129

          BIO_up_ref(BIO_next(b));
          BIO_up_ref(BIO_next(b));   // one too many
          SSL_set_bio(ssl, BIO_next(b), BIO_next(b));

This seems to be a real issue. Thanks! I'll fix it asap.

Reference: https://github.com/aws/aws-lc/blob/AWS-LC-FIPS-2.0.0/ssl/ssl_lib.cc#L748

@DimensionWieldr
Copy link
Copy Markdown
Collaborator

Fixed by #1616 . LGTM

@DimensionWieldr DimensionWieldr added verified Approved for release and removed verified-with-issues Verified by QA and issues found. labels Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fips Work related to Altinity FIPS releases fips-25.3 fips-25.3.8.30001 verified Approved for release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants