Skip to content

Commit

Permalink
Release 2.0.0 of the Amazon Kinesis Client for .NET (#33)
Browse files Browse the repository at this point in the history 
* Added support for Enhanced Fan-Out: https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/.
  Enhanced Fan-Out provides dedicated throughput per stream consumer, and uses an HTTP/2 push API (SubscribeToShard) to deliver records with lower latency.
* Updated the Amazon Kinesis Client Library for Java to version 2.1.2.
  * Version 2.1.2 uses 4 additional Kinesis API's
    __WARNING: These additional API's may require updating any explicit IAM policies__
    * `RegisterStreamConsumer`: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_RegisterStreamConsumer.html
    * `SubscribeToShard`: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html
    * `DescribeStreamConsumer`: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamConsumer.html
    * `DescribeStreamSummary`: https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamSummary.html
  * For more information about Enhanced Fan-Out with the Amazon
    Kinesis Client Library please see the
    announcement: https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/ and
    developer documentation: https://docs.aws.amazon.com/streams/latest/dev/introduction-to-enhanced-consumers.html.
* Added a new record processor interface `IShardRecordProcessor` interface closely matches the Java `ShardRecordProcessor`
  `IShardRecordProcessor`: https://github.com/awslabs/amazon-kinesis-client-net/blob/95fd04a5702c287358eb3f58057017a6fd96000d/ClientLibrary/IShardRecordProcessor.cs#L18.
  `ShardRecordProcessor`: https://github.com/awslabs/amazon-kinesis-client/blob/258be9a504a0e179d9cf9e0eaa6e0cf99003578b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/ShardRecordProcessor.java#L27.
  While the original `IRecordProcessor` interface remains present, and will continue to work it's recommended to upgrade to the newer interface.
  * The `Shutdown` method from `IRecordProcessor` has been replaced by `LeaseLost` and `ShardEnded`.
  * Added the `LeaseLost` method which is invoked when a lease is lost.
    `LeaseLost` replaces `Shutdown` where `ShutdownInput.Reason` was `ShutdownReason.ZOMBIE`.
  * Added the `ShardEnded` method which is invoked when all records from a split or merge have been processed.
    `ShardEnded`  replaces `Shutdown` where `ShutdownInput.Reason` was `ShutdownReason.TERMINATE`.
  * Added `ShutdownRequested` which provides the record processor a last chance to checkpoint during the Amazon Kinesis Client Library shutdown process before the lease is canceled.
    * To control how long the Amazon Kinesis Client Library waits for the record processors to complete shutdown, add `timeoutInSeconds=<seconds to wait>` to your properties file.
* Updated the AWS Java SDK version to 2.4.0
* MultiLangDaemon now provides logging using Logback.
  * MultiLangDaemon supports custom configurations for logging via a Logback XML configuration file.
  * The Bootstrap program was been updated to accept either `-l` or `--log-configuration` to provide a Logback XML configuration file.
  • Loading branch information
@pfifer
pfifer authored Feb 27, 2019
1 parent 95fd04a commit c709cda
Showing 1 changed file with 61 additions and 11 deletions.
72 changes: 61 additions & 11 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,36 +9,59 @@ This package wraps and manages the interaction with the *MultiLangDaemon*, which
A record processor in C# typically looks something like the following:

```csharp
using System;
using System.Collections.Generic;
using Amazon.Kinesis.ClientLibrary;

namespace Sample
{
class SampleRecordProcessor : IRecordProcessor
public class RecordProcessor : IShardRecordProcessor
{
public void Initialize(InitializationInput input)
{
// initialize
//
// Initialize the record processor
//
}

public void ProcessRecords(ProcessRecordsInput input)
{
// process batch of records (input.Records) and
// checkpoint (using input.Checkpointer)
//
// Process a batch of records from input.Records, and optionally checkpoint by calling
// input.Checkpointer.Checkpoint()
//
}

public void Shutdown(ShutdownInput input)
public void LeaseLost(LeaseLossInput leaseLossInput)
{
// cleanup
//
// Perform any cleanup required.
// This record processor has lost it's lease so checkpointing is not possible.
// This is why LeaseLostInput does not provide a Checkpointer property,
//
}

public void ShardEnded(ShardEndedInput shardEndedInput)
{
//
// The record process has processed all records in the shard, and will no longer receive records.
// It is required that this method call shardEndedInput.Checkpointer.Checkpoint() to inform the KCL
// that the record processor has acknowledged the completion of the shard.
//
}

public void ShutdownRequested(ShutdownRequestedInput shutdownRequestedInput)
{
//
// This is called when the KCL is being shutdown, and if desired the record processor can checkpoint here
// by calling shutdownRequestedInput.Checkpointer.Checkpoint(...)
//
}
}

class MainClass
internal class MainClass
{
public static void Main(string[] args)
{
KCLProcess.Create(new SampleRecordProcessor()).Run();
KclProcess.Create(new RecordProcessor()).Run();
}
}
}
Expand Down Expand Up @@ -97,7 +120,7 @@ To run the processor, first **build the SampleConsumer project**, then **run the
* You must have [Java][jvm] installed.
* If you omit the `--execute` argument, the bootstrap program outputs a command that can be used to start the KCL directly.
* The *MultiLangDaemon* reads its configuration from the `kcl.properties` file, which contains a few important settings:
* **executableName = SampleProcessor.exe**
* **executableName = dotnet SampleProcessor.dll**
The name of the processor executable.
* **streamName = myTestStream**
The name of the Kinesis stream from which to read data. This must match the stream name used by your producer.
Expand Down Expand Up @@ -125,6 +148,33 @@ you to focus on writing record processing logic in C#. This approach enables the
be language-agnostic, while providing identical features and similar parallel processing model across
all languages.

## Release Notes

### Release 2.0.0 (February 27, 2019)
* Added support for [Enhanced Fan-Out](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/).
Enhanced Fan-Out provides dedicated throughput per stream consumer, and uses an HTTP/2 push API (SubscribeToShard) to deliver records with lower latency.
* Updated the Amazon Kinesis Client Library for Java to version 2.1.2.
* Version 2.1.2 uses 4 additional Kinesis API's
__WARNING: These additional API's may require updating any explicit IAM policies__
* [`RegisterStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_RegisterStreamConsumer.html)
* [`SubscribeToShard`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html)
* [`DescribeStreamConsumer`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamConsumer.html)
* [`DescribeStreamSummary`](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStreamSummary.html)
* For more information about Enhanced Fan-Out with the Amazon Kinesis Client Library please see the [announcement](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/) and [developer documentation](https://docs.aws.amazon.com/streams/latest/dev/introduction-to-enhanced-consumers.html).
* Added a new record processor interface [`IShardRecordProcessor`](https://github.com/awslabs/amazon-kinesis-client-net/blob/95fd04a5702c287358eb3f58057017a6fd96000d/ClientLibrary/IShardRecordProcessor.cs#L18). This interface closely matches the Java [`ShardRecordProcessor`](https://github.com/awslabs/amazon-kinesis-client/blob/258be9a504a0e179d9cf9e0eaa6e0cf99003578b/amazon-kinesis-client/src/main/java/software/amazon/kinesis/processor/ShardRecordProcessor.java#L27) interface.
While the original `IRecordProcessor` interface remains present, and will continue to work it's recommended to upgrade to the newer interface.
* The `Shutdown` method from `IRecordProcessor` has been replaced by `LeaseLost` and `ShardEnded`.
* Added the `LeaseLost` method which is invoked when a lease is lost.
`LeaseLost` replaces `Shutdown` where `ShutdownInput.Reason` was `ShutdownReason.ZOMBIE`.
* Added the `ShardEnded` method which is invoked when all records from a split or merge have been processed.
`ShardEnded` replaces `Shutdown` where `ShutdownInput.Reason` was `ShutdownReason.TERMINATE`.
* Added `ShutdownRequested` which provides the record processor a last chance to checkpoint during the Amazon Kinesis Client Library shutdown process before the lease is canceled.
* To control how long the Amazon Kinesis Client Library waits for the record processors to complete shutdown, add `timeoutInSeconds=<seconds to wait>` to your properties file.
* Updated the AWS Java SDK version to 2.4.0
* MultiLangDaemon now provides logging using Logback.
* MultiLangDaemon supports custom configurations for logging via a Logback XML configuration file.
* The Bootstrap program was been updated to accept either `-l` or `--log-configuration` to provide a Logback XML configuration file.

## See Also

* [Developing Processor Applications for Amazon Kinesis Using the Amazon Kinesis Client Library][amazon-kcl]
Expand Down

0 comments on commit c709cda

Please sign in to comment.