http-stream

There are already so many data integration tools for mass、fast、accurate data processing:

• datax.
• sqoop.
• kettle.
• hop.
• airbyte.
• flinkx(chunjun).
• nifi.
• seatunnel.
• gobblin.

But how to sync data through http is also a big problem for developers.

• various request and response. Different http api provider, different rules have to obey.

  • request params. header、query、path、request body
  • date format. such as timestamp, yyyy-MM-dd HH:mm:ss or YYYY-MM-DDTHH:MM:SSZ.

• authentication and authorization.

• ratelimiter. qps(query per second) or concurrency.

• network failure. not threaten or caution, frequent network failure would never disappear.

Challenges and Solutions

various request

• pagination。
  • Rethinking Streaming Workloads with Akka Streams: Part II。
  • PagedSource。
• entire or incremental extraction。

Excellent Practices

fault tolerance

retry and ratelimiter

• resilience4j
• failsafe
• guava-retrying

detailed query context

query param and response result

metrics

• micrometer
• metrics