Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way to dynamically update TLS certificates #5228

Merged
merged 43 commits into from
Nov 7, 2024

Conversation

my4-dev
Copy link
Contributor

@my4-dev my4-dev commented Oct 9, 2023

Motivation:

#5033

API design note:

  • TlsKeyPair represents a pair of PrivateKey and X509Certificate chain.
    • This API is used as an official key pair type in Armeria.
    • All APIs for specifying key pairs in TlsSetters have been deprecated in favor of TlsSetters tls(TlsKeyPair).
      TlsKeyPair.of(privateKey, certificate);
      TlsKeyPair.of(privateKey, keyPassword, certificate);
  • TlsProvider dynamically resolves a TlsKeyPair for the given hostname when a connection is established.
    • DNS wildcard format is supported as a hostname.
    • * is used as a special hostname to get the TlsKeyPair for the default virtual host.
      TlsProvider
         .builder()
          // Set the default key pair.
         .keyPair(TlsKeyPair.of(...))
         // Set the key pair for "*.example.com".
         .keyPair("*.example.com", TlsKeyPair.of(...))
         .build();
    • To dynamically update/reload TlsKeyPair, a custom TlsProvider can be implemented.
      class DynamicTlsProvider implements TlsProvider {
         @Override
         public TlsKeyPair keyPair(String hostname) {
             // relodableCache will be updated periodically by a scheduler
             return relodableCache.get(hostname);
         }
      }
      • The newly returned key pair is used for the TLS handshake of new connections.
  • ServerTlsConfig and ClientTlsConfig are added to override the default values and customize SslContextBuilder.
    • Unlike TlsProvider, *TlsConfig are immutable so all TlsKeyPairs returned by a TlsProvider build SslContext with the same configuration.
  • Both server and client allow TlsProvider and TlsKeyPair for TLS configurations.
    Server
      .builder()
      // For dynamic usage
      .tlsProvider(tlsProvider)
      // For customizing TLS
      .tlsProvider(tlsProvider, serverTlsConfig)
      // For sample usage
      .tls(tlsKeyPair)
      .build()
    
    ClientFactory
      .builder()
      // For dynamic usage
      .tlsProvider(tlsProvider)
      // For customizing TLS
      .tlsProvider(tlsProvider, clientTlsConfig)
      // For sample usage
      .tls(tlsKeyPair)
      .build()
    • Some internal implementations for TLS handshake have been changed to create SslContext dynamically.

Modifications:

  • Server
    • Add TlsProviderMapping that converts TlsProvider into SslContext Mapping for SniHandler.
      • A dynamic TlsProvider can be used to update the certificates without Server.reconfigure().
    • Add a setter method for TlsProvider to ServerBuilder.
      • A builder method for VirtualHost isn't added because a TlsProvider can contain multiple certificates.
        • If necessary, I will consider TlsProvider at the virtual host level later.
  • Client
    • Fix Bootstraps to create a Bootstrap with a TlsKeyPair returned by TlsProvider when a new connection is created.
      • If no TlsProvider is set, the original behavior that returns predefined BootStraap is used.
    • Add options for TlsProvider to ClientFactoryBuilder.
  • Common
    • TlsProvider provides separate builders for the client and server.
    • TlsKeyPair provides various factory methods to easily create a key pair from different resources.
    • Cache SslContexts and expire them after 1 hour of inactivity.
      • If you think that the caching strategy will not be effective, I am willing to revert it.
    • Add CloseableMeterBinder to unregister when the associated resource is unused.
  • Deprecate) The following APIs have been deprecated:
    • TlsSetters tls(File keyCertChainFile, File keyFile)
    • TlsSetters tls(File keyCertChainFile, File keyFile, @Nullable String keyPassword)
    • TlsSetters tls(InputStream keyCertChainInputStream, InputStream keyInputStream)
    • TlsSetters tls(InputStream keyCertChainInputStream, InputStream keyInputStream, @Nullable String keyPassword)
    • TlsSetters tls(PrivateKey key, X509Certificate... keyCertChain)
    • TlsSetters tls(PrivateKey key, Iterable<? extends X509Certificate> keyCertChain)
    • TlsSetters tls(PrivateKey key, @Nullable String keyPassword, X509Certificate... keyCertChain)
    • TlsSetters tls(PrivateKey key, @Nullable String keyPassword, Iterable<? extends X509Certificate> keyCertChain)

Result:

@my4-dev
Copy link
Contributor Author

my4-dev commented Oct 9, 2023

Hi, @ikhoon !
This PR is just a poc and incomplete.
Please confirm that this implementation is appropriate to solve this Issue.
(This is a little different from your suggestion)

@minwoox minwoox added this to the 1.27.0 milestone Oct 11, 2023
@ikhoon
Copy link
Contributor

ikhoon commented Oct 31, 2023

The priority of this issue has been raised. Please let me finish this PR. 🙇‍♂️

@my4-dev
Copy link
Contributor Author

my4-dev commented Oct 31, 2023

I'm sorry to be late for this reply.
I was so busy that I couldn't confirm this PR🙇‍♂️

@ikhoon
Copy link
Contributor

ikhoon commented Oct 31, 2023

No worries. I’m really grateful for your contribution. 🙇‍♂️

@ikhoon ikhoon changed the title Specify a client TLS key pair without ClientFactory Provide a way to dynamically update TLS certificates Dec 20, 2023
Copy link

codecov bot commented Jan 3, 2024

Codecov Report

Attention: Patch coverage is 65.77017% with 140 lines in your changes missing coverage. Please review.

Project coverage is 73.62%. Comparing base (e08527d) to head (cb798b2).
Report is 7 commits behind head on main.

Current head cb798b2 differs from pull request most recent head be0752d

Please upload reports for the commit be0752d to get more accurate results.

Files Patch % Lines
...ecorp/armeria/internal/common/TlsProviderUtil.java 73.61% 10 Missing and 9 partials ⚠️
.../armeria/internal/common/util/CertificateUtil.java 40.00% 12 Missing ⚠️
...n/java/com/linecorp/armeria/common/TlsKeyPair.java 72.50% 9 Missing and 2 partials ⚠️
...ava/com/linecorp/armeria/server/ServerBuilder.java 60.71% 5 Missing and 6 partials ⚠️
...a/com/linecorp/armeria/client/HttpChannelPool.java 64.28% 5 Missing and 5 partials ⚠️
...om/linecorp/armeria/common/DefaultTlsProvider.java 74.19% 7 Missing and 1 partial ⚠️
...corp/armeria/common/metric/CertificateMetrics.java 75.75% 8 Missing ⚠️
...ia/common/metric/AbstractCloseableMeterBinder.java 50.00% 7 Missing ⚠️
...om/linecorp/armeria/common/TlsProviderBuilder.java 68.42% 5 Missing and 1 partial ⚠️
.../java/com/linecorp/armeria/common/TlsProvider.java 16.66% 5 Missing ⚠️
... and 19 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5228      +/-   ##
============================================
- Coverage     74.75%   73.62%   -1.14%     
+ Complexity    21409    20168    -1241     
============================================
  Files          1877     1748     -129     
  Lines         79126    74634    -4492     
  Branches      10201     9526     -675     
============================================
- Hits          59152    54946    -4206     
+ Misses        15179    15139      -40     
+ Partials       4795     4549     -246     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ikhoon ikhoon changed the title Provide a way to dynamically update TLS certificates Provide a way to dynamically update TLS certificates for server and client Jan 3, 2024
@ikhoon
Copy link
Contributor

ikhoon commented Jan 3, 2024

This PR is ready to review. PTAL. The PR description has also been updated for the API design and implementation details.

Copy link
Contributor

@jrhee17 jrhee17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! Server side looks good to me 👍 Left some minor questions on the client side 🙇

Copy link
Contributor

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks promising! 👍

@ikhoon ikhoon removed this from the 1.27.0 milestone Jan 24, 2024
Comment on lines 104 to 115
@Deprecated
default boolean allowsUnsafeCiphers() {
return false;
}

/**
* Returns the {@link Consumer} which can arbitrarily configure the {@link SslContextBuilder} that will be
* applied to the SSL session.
*/
default Consumer<SslContextBuilder> tlsCustomizer() {
return builder -> {};
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That will move the responsibility of managing the life cycle of SslContext to TlsProvider, which may complicate things.

What do you think of modifying the method for creating the SSL context so that the TlsProvider doesn't have to manage the lifecycle at all? Here's a proposed interface change:

interface TlsProvider {
  SslContext createSslContext(String hostname, SessionProtocol sessionProtocol, TlsEngineType tlsEngineType);
}

I'm suggesting this because it seems like the current TlsProvider functions more as a data class than as a provider.

TlsProvider is designed to be used at both client and server.

At present, we can use TlsProvider.ofServer() for the client side as well. By validating the TlsProvider when it's assigned to a client or a server, we can prevent this misuse.
If you still prefer to use TlsProvider for both, we could add enum parameters to TlsEngineType such as CLIENT_JDK, SERVER_JDK, CLIENT_OPENSSL, etc.

Users who implement their own TlsProvider may find it difficult to build SslContext, or we might need to expose a factory method to make it easier.

That is true. I also worry about exposing Netty's SslContext API to the public. However, creating an SslContext might not be too difficult since Netty provides a well-designed SslContextBuilder API. To address the issue of exposing the API, we could provide our own abstraction, similar to what we did with DnsEndpointGroupBuilder.

@ikhoon ikhoon modified the milestones: 1.30.0, 1.31.0 Aug 1, 2024
@github-actions github-actions bot added the Stale label Sep 9, 2024
@ikhoon ikhoon changed the title Provide a way to dynamically update TLS certificates for server Provide a way to dynamically update TLS certificates Sep 20, 2024
@ikhoon
Copy link
Contributor

ikhoon commented Sep 24, 2024

This PR is ready to review again.

@ikhoon ikhoon removed the Stale label Sep 24, 2024
Copy link
Contributor

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! 👍 👍 👍 Left minor comments and questions. 🙇


final Bootstrap baseBootstrap = isDomainSocket ? unixBaseBootstrap : inetBaseBootstrap;
assert baseBootstrap != null;
return newBootstrap(baseBootstrap, remoteAddress, desiredProtocol, serializationFormat);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, we should create bootstrap every time when connecting if sslContextFactory is set?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Because we need a new ChannelInitializer for a different SslContext.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so, I'm a bit worried about the performance. 🤔
Maybe @trustin can give some idea.

Copy link
Contributor

@ikhoon ikhoon Sep 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I saw Bootstrap, I thought it was a simple class and did not refer to heavy objects.

Copy link
Contributor

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! 🙇 🙇 🙇

Comment on lines 180 to 184
final List<X509Certificate> trustedCerts = tlsProvider.trustedCertificates();
if (trustedCerts != null && !trustedCerts.isEmpty()) {
contextBuilder.trustManager(trustedCerts);
}
applyTlsConfig(contextBuilder);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that tlsProvider.trustedCertificates can be overwritten by clientTlsConfig.tlsNoVerifySet() or clientTlsConfig.insecureHosts() was surprising to me.

Is this intentional?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have the same behavior before. Trusted certificates set with ClientFactoryBuilder.tlsCustomizer(c -> c.trustManager(certs)) got overwritten by ClientFactoryBuilder.tlsNoVerifySet().

tlsProvider.trustedCertificates and clientTlsConfig.tlsNoVerifySet() or clientTlsConfig.insecureHosts() are mutually exclusive. If we respect trustedCertificates, tlsNoVerifySet is also ignored.

What do you think of adding a note for the behavior?

* Provides client-side TLS configuration for {@link TlsProvider}.
*/
@UnstableApi
public final class ClientTlsConfig extends AbstractTlsConfig {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the record, can you explain what properties go into KeyPair and what properties go into TlsProvider?

i.e. tlsNoVerifySet, insecureHosts seems to modify the trusted certificates, yet they are in ClientTlsConfig. On the other hand, TlsProvider has a trustedCertificates method

I'm wondering if it's just easier to expose an interface which is a simplified version of TrustManager (similarly to how KeyPair is a pojo version of KeyManager)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you imagine to return the simplified TrustManager instead of List<X509Certificate> for trustedCertificates()?

I hesitated to add the simplified TrustManager because TlsProvider was designed to use both server and client but tlsNoVerifySet and insecureHosts would be only useful for clients. Moreover, they are not a recommended option for production.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you imagine to return the simplified TrustManager instead of List for trustedCertificates()?

Right, I imagined that TlsProvider contains a trustManager()-esque API

I hesitated to add the simplified TrustManager because TlsProvider was designed to use both server and client but tlsNoVerifySet and insecureHosts would be only useful for clients. Moreover, they are not a recommended option for production.

I thought TrustManager is used by both server and client side.
I still find it odd that TrustManager related APIs exist in both APIs (TlsProvider, ClientTlsConfig), but I'd rather not delay this PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think TlsProvider's function is to provide the data required for TLS handshake rather than associating with TrustedManger.

tlsNoVerifySet, insecureHosts are options that exist at the code level so it would be ambiguous to express them as data. I thought ClientTlsConfig was a better place to set them.

I understand your concerns and this API may cause confusion or inconvenience. Let's get user feedback and decide whether the current design needs refactoring.

final TlsKeyPair tlsKeyPair = key.tlsKeyPair();
if (mode == SslContextMode.SERVER) {
assert tlsKeyPair != null;
return createSslContext(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: the call to createSslContext modifies the global security providers during run-time (as opposed to start-up time when the ClientFactory is initialized).

If users aren't using pure armeria, I think it's possible that MinifiedBouncyCastleProvider can be used inadvertently.

I'm not sure if this is a good idea, but I don't see a way around this so SGTM for now

* Creates a new {@link TlsKeyPair} from the specified key file, key password and certificate chain
* file.
*/
public static TlsKeyPair of(File keyFile, @Nullable String keyPassword, File certificateChainFile) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: while this is a convenient way to load a TlsKeyPair, we may want to note in javadocs that TlsProvider#find shouldn't block.

e.g.

Server.builder()
      .tlsSelfSigned()
      .tlsProvider(hostname -> TlsKeyPair.of(new File(""), new File("")))
      .build();

return refCnt == 0;
}

void destroy() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question) Is there no need for ReferenceCountUtil.release(sslContext) in case SslContext is an instance of ReferenceCountedOpenSslServerContext?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReferenceCountedOpenSslServerContext can't be used in Armeria. I added a TODO to support it.
https://github.com/line/armeria/pull/5228/files#diff-3fb600f6cb71c42141597949d05d9e1a8879ada1c3385f3aaaac9c774f5d0038R74-R75

@@ -420,6 +440,8 @@ public ClientFactoryBuilder tls(KeyManagerFactory keyManagerFactory) {
@Override
public ClientFactoryBuilder tlsCustomizer(Consumer<? super SslContextBuilder> tlsCustomizer) {
requireNonNull(tlsCustomizer, "tlsCustomizer");
ensureNoTlsProvider();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question) Is there a reason tlsCustomizer isn't deprecated? It seems like all functionality is covered by ClientTlsConfig

Copy link
Contributor

@ikhoon ikhoon Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagined TlsProvider is an optional feature for the TLS handshake.
For users who don't need dynamic TLS, ClientFactoryBuilder.tls(TlsKeyPair) and ClientFactoryBuilder.tlsCustomizer() would be enough for simplicity.

However, the API may be deprecated in the future.

* <p>Note that this method mutually exclusive with {@link #tls(TlsKeyPair)} and other static TLS settings.
*/
@UnstableApi
public ServerBuilder tlsProvider(TlsProvider tlsProvider) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I still think it may be useful for VirtualHosts to set TlsProvider since

  1. The hostname pattern can be set with a wildcard.
  2. Users may host multiple domains with a single server, and want to rotate each domain independently

I'm curious how hard it would be to modify s.t. each VirtualHost can set a TlsProvider:

  • The defaultVirtualHost can set either a SslContext or a TlsProvider
  • Each VirtualHost can set either a SslContext or a TlsProvider
  • We define a DelegatingTlsMapping where each VirtualHost.hostnamePattern maps to a TlsProvider or a SslContext

Copy link
Contributor

@ikhoon ikhoon Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible to support TlsProvider for VirtualHosts. I didn't implement it because a TlsProvider can contain multiple certificates so one TlsProvider covers all VirtualHosts.

If there are any requirements for it, I'd be happy to implement it.

- Return associated certificates in `MappedTlsProvider`
- Rename `TlsProvider.set{Default}()` to `TlsProvider.keyPair()`
Copy link
Contributor

@jrhee17 jrhee17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 👍 👍

@jrhee17
Copy link
Contributor

jrhee17 commented Nov 7, 2024

@ikhoon is this ready for merging?

@ikhoon
Copy link
Contributor

ikhoon commented Nov 7, 2024

Yes.
Thanks to @my4-dev for the initial work! 🙇‍♂️🙇‍♂️

@ikhoon ikhoon merged commit 6628513 into line:main Nov 7, 2024
11 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Specify a client TLS key pair without ClientFactory
6 participants