Support for FIPS compliance mode

[beanuwave]

Description

This PR makes FIPS mode available through the OPENSEARCH_CRYPTO_STANDARD=FIPS-140-3 environmental parameter instead of the tests.fips.enabled setting. It provides FIPS 140-3 support by replacing all BC dependencies with BCFIPS dependencies and making FIPS approved-only mode configurable at launch. Running this mode restricts the BCFIPS provider to rely solely on FIPS-certified ciphers.

• The fips.gradle build script is removed in order to support a single-build solution.
• All BC dependencies are replaced by BCFIPS.
• Fixed creation of password protected internal keystore for valuable settings at node start. The add-string command was made with additional --stdin option, which interferes with password input.
• The Password Matcher inside Identity-Shiro that relies on BC to check if hashed passwords match with OpenBSDBCrypt is replaced by the password4j implementation.
• Adds full support for the BCFKS format (*.bcfks) for key and trust stores, also making it the default.
• KeyStore instantiation was added to forbidden-apis in favour of KeyStoreFactory.
• Google's truststore is converted to the BCFKS format.
• Makes the best guess of which store type is provided based on the filename extension.
• Store types are strictly limited to JKS, PKCS12, PKCS11, and BCFKS.
• Refactors PemUtils to parse private keys in formats EC, PKCS8, PKCS1, and DSA, with or without encryption, and with or without parameters.
• dependency ':libs:opensearch-common' was added to rest-client build, to support strict keystore types. It's also the reason for JVM bump JAVA8 to JAVA11.
• allow ingest-attachment plugin to run in FIPS mode, since BC dependencies find no use.
• The java.security file is added to the build to distinguish between FIPS and non-FIPS environments.
• The fips_java.security file is altered due to evolving security standards.
• The security.policy file is altered to grant necessary security permissions.
• Increased security standards in KeyTabs and algorithms for Kerberos.
• SecureRandom gets instantiated in to different way, depending on if it runs with FIPS or not.
• Uninstalls SunJCE provider from security providers list at runtime when FIPS mode is enabled.

Runtime limitations (known so far) that come with enabling FIPS mode:

Admins can continue to manage their systems without being impacted by this change. However, for those keen on FIPS compliance, the most common obstacle will likely be the requirement to set a stronger password for the internal keystore and also convert key and truststores to *.bcfks format.

• Does not allow empty passwords for keystores or private keys (they need to be at least 112 bits in strength).
• The ssl.verification_mode=NONE setting is not permitted.
• JKS and PKCS12 key and trust store types are not supported at all.
• The internal keystore cannot be auto-migrated from versions 1 or 2.
• Azure Classic Discovery plugin -> deprecated.
• SQL-CLI client.
• HDFS plugin won't connect since it's using RC4 cipher for token authentication.

Reasons for refactoring PemUtils, which is used by the Reindex API in cases of migrating data from a remote cluster that is TLS protected:

• Lack of support for evolving standards like PKCS#8.
• Password-Based Key Derivation Functions such as PBKDF-OPENSSL are not supported in FIPS mode in favor of the PBKDF2 standard.
• Java type safety.
• It is generally a good idea to let ASN1 annotation parsing be done by external security libraries.

Related Issues

opensearch-project/security#3420

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8e8ed47: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6016d5d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Could use some help maybe from @cwperks or @peternied reviewing this, please.

[KarstenSchnitter]

@reta I totally get your point. But I also see a downside of your proposal. Splitting up this PR will create considerable effort and time to do so. This will inevitably prolong the time the changes remain unmerged. I am not sure, it will be faster creating separate PRs out of this one. It might also create some confusion due to the immense number of comments already discussed in this PR's reviews.

Given the nature of this PR, there is quite some potential for conflicts with the upcoming SecurityManager disablement/removal. Or am I judging this wrongly? We would really like to see this effort coming to a close with the PR being merged.

In the end, we will handle this change by the recommendation of this project's maintainers. If you tell us to split, we will do it. Please let us know, what way you want to pursue.

[reta]

@reta I totally get your point. But I also see a downside of your proposal. Splitting up this PR will create considerable effort and time to do so.

@KarstenSchnitter thank you, the whole intent of the suggestion is to get this change reviewed, approved and merged. I am (personally) not comfortable with some of these (fips deps bundled / leaked with some modules, tons of conditionals still present in some tests, ...) and I thought going with small steps would allow us to tackle the scoped problems one by one (plus cutting the large 200 files change into smaller ones would certainly help). But I am totally fine if other maintainers see it not being necessary.

[cwperks]

What is breaking with the inclusion of the FIPS deps even if not running in FIPS-140-3 compliance mode? I know there's been a lot of due diligence already done with this PR so I'd like to see it merged in and any follow-up items identified and completed with sufficient headway before 3.0.0 release.

I see that the gradle check has been failing, do you need any help in identifying issues in the failing test suite?

[peternied]

In the end, we will handle this change by the recommendation of this project's maintainers. If you tell us to split, we will do it. Please let us know, what way you want to pursue.

@iigonin @beanuwave @KarstenSchnitter I appreciate the position you are in - thank you work working with us, this is a big feature and its only gotten bigger as we've got into the details. I think this grow is also causing turn arounds to be slower - I know it has with my involvement.

With smaller increments will make it easier to get more maintainer engagement and codify decisions that don't need to be revisited. While breaking the changes apart will take more time in energy the trade-off is better confidence for the project.

Cherry-picking from components I've reviewed in the past some candidates could be such refactoring around KeyStoreFactory or RestClientFipsAwareTestCase that provide a better foundation for the OpenSearch and work well in isolation.

It might also create some confusion due to the immense number of comments already discussed in this PR's reviews.

100% and I would hate to lose that context, I think this PR should be preserved at least for the great dialogs and decision making. One way this could be managed is by making new PRs and creating links to relevant comments in advance of requesting the review - that creates continuity.

Another thought, this PR's size will winnows down over time as incremental progress is merged in from main.

---

I know this has been a huge undertaking and we are converging - thanks for working with all of us.

[beanuwave]

@reta I would also appreciate multiple smaller commits so that they are easier to review. The reason I found it difficult with the modular approach is that the project already had BC dependencies in multiple modules. Unfortunately, the process of replacing old BC with FIPS counterparts cannot be split because they cannot coexist on the classpath at the same time.

@cwperks The build fails on SimpleSecureNetty4TransportTests due to a race condition. Despite minimizing randomness and tweaking various cluster/node transport settings, I still cannot pinpoint the cause of the closed connection and handshake issues. Any help would be greatly appreciated.

[reta]

@reta I would also appreciate multiple smaller commits so that they are easier to review. The reason I found it difficult with the modular approach is that the project already had BC dependencies in multiple modules. Unfortunately, the process of replacing old BC with FIPS counterparts cannot be split because they cannot coexist on the classpath at the same time.

Thank you @beanuwave, I think we could clearly start with build tools and removing deprecated / not used settings (as I suggested) - it does not have any dependencies on FIPS but will shave off 20-30 files immediately. The other pieces will follow, at least I don't see the blockers there, but certainly some work.

[cwperks]

@cwperks The build fails on SimpleSecureNetty4TransportTests due to a race condition. Despite minimizing randomness and tweaking various cluster/node transport settings, I still cannot pinpoint the cause of the closed connection and handshake issues. Any help would be greatly appreciated.

I once encountered a similar thread leak failure on #16778, but not sure how to resolve the failure seen in SimpleSecureNetty4TransportTests.testFailToSend. It passes when I run the test locally or on my dev desktop, but fails consistently in the jenkins runner.

[github-actions]

❌ Gradle check result for 96524cf: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?