[Task]: Extending Ballerina's Transaction Support to Include Transaction Recovery

[dsplayerX]

Description

Ballerina doesn't have native support for recovery in distributed transactions. It offers recovery only for database transactions utilizing the Atomikos library's transaction manager but lacks the support for transactional microservices or other XA resources. The goal of this task is to extend Ballerina's transaction support to include native recovery functionality for distributed transactions, according to the XA spec, eliminating the need for the Atomikos library. It aims to mitigate risks from network failures, resource manager issues, and application errors, ensuring data consistency, fault tolerance, and overall application reliability in distributed transactions.

Describe your task(s)

[Phase 1] Recovery for Direct XA Resource Transactions

• Implement a Log Manager:
  Implement a component that manages an in-memory log, allowing dynamic tracking of transaction status during runtime. This log stores relevant information about ongoing transactions' states.
  And a file-based log manager responsible for persistently storing transaction logs. This ensures that transaction information is durably saved and can be recovered even after system restarts or crashes.
• Log Transaction States:
  Identify and add logs where necessary to capture required transaction state information, to be used in the recovery processes.
• Implement Recovery Pass after System Crash:
  Implement a recovery process that runs during startup to perform an initial pass and recover any incomplete transactions or states resulting from a previous system crash before returning to normal operation. Tracked in Implement Transaction Recovery for XA Resources #42080
• Handle Transaction Failures During Runtime:
  Develop a mechanism to handle recovery for runtime transaction failures, preserving data integrity and reducing hazard outcomes. For this, the improvements [Improvement]: Set Transaction Timeout Functionality for Ballerina Transactions #42061 and [Improvement]: Improve Ballerina transactions with XA Resource connection check/refresh before commit, rollback operations #41933 were created.

[Phase 2] Coordinator-Participant Recovery

• Design and Implement Coordinator-Participant Recovery Mechanism:
  Design and implement a recovery mechanism for both coordinators and participant nodes. This mechanism should allow communication between two nodes and gracefully recover from failures, ensuring that transactions can either be completed or rolled back consistently.

Related area

-> Compilation

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[dsplayerX]

Changes and New Additions

• A recovery manager instance was integrated into the transaction manager, along side the log manager which handles in-memory and a file-based logging.
• XIDGenerator was changed to use the transactionId as the gtrid and the transactionBlockId as the bqual. The default_format value is obtained by combining the ASCII values of the characters 'B', 'A', and 'L'. This value will be unique for ballerina transactions but same for each transaction.
• RecoveryStates enum was introduced and used to manage and identify different states of transactions during logging and recovery.
• TransactionLogRecord was introduced which holds the information of a transaction including transaction id, transaction state and the log time (as of now, will need to include more information for coordinator-participant recovery).
• RecoveryLogManager was introduced that manages transaction recovery logs with a combination of file-based FileRecoveryLog and in-memory InMemoryRecoveryLog logs, allowing the adding and retrieving of transaction log records.
  • The in-memory log manager is used for dynamic tracking of transaction status during runtime.
  • The file-based log manager is responsible for persistently storing transaction logs, ensuring recovery after system restarts or crashes.
  • The file log can be customized by changing the directory, filename, checkpoint interval and whether or not to delete old logs. Configuration values for recoveryLogName, recoveryLogDir, checkpointInterval (use "-1" to disable checkpointing), deleteOldLogs can be provided under [ballerina.lang.transaction].
• RecoveryManager was introduced which is responsible for recovering transactions.
  • As of now, it manages a collection of pending transaction log records and a set of XA resources to be recovered.
  • The recovery process involves checking the transaction state, retrieving prepared XIDs from the XA resources, and replaying commit or rollback operations based on the transaction state.
  • This also handles heuristic terminations of transactions in XA Resources.

Recovery Pass

• XA Resources for recovery will be gathered using an exposed addXAResourceToRecover method from the library side during their initialization. Transaction recovery will not be supported for resources created/initialized within the transaction block.
• The transaction manager features a startupCrashRecovery method designed to initiate recovery post-system crashes.
• A boolean variable is utilized within the transaction manager to track the success status of the initial recovery pass.
• Recovery process will run during system startup, performing an initial pass to recover incomplete transactions resulting from a previous system crash before returning to normal operation. Failed transactions (prepared and in-doubt) are identified and recovered from the saved log during startup crash recovery releasing currently held locks and allowing the system to operate normally.

Update 23/01/2024

[dsplayerX]

Recovery Process

The recovery process involves retrieving failed transactions from the XAResources using xa_recover(). This would return a list of XIDs (transaction identifiers) for transactions that were in progress but failed to complete in that specific resource. Once we have these XIDs, we search for corresponding log records to determine the decision (commit/abort) that was previously made by the coordinator for each transaction and then act on it accordingly. This typically involves either committing or aborting the transaction, depending on the decision recorded in the logs. If there are mixed/hazard outcomes, the user is warned of those outcomes and those need to be manually handled.

Update 11/02/24

[dsplayerX]

The recovery process involves retrieving failed transactions from the XAResources using xa_recover(). This would return a list of XIDs (transaction identifiers) for transactions that were in progress but failed to complete in that specific resource. Once we have these XIDs, we search for corresponding log records to determine the decision (commit/abort) that was previously made by the coordinator for each transaction and then act on it accordingly. This typically involves either committing or aborting the transaction, depending on the decision recorded in the logs. If there are mixed/hazard outcomes, the user is warned of those outcomes and those need to be manually handled.

As discussed, retrieving prepared transactions from the database and matching them with corresponding log records to act based on the coordinator's decision was deemed unnecessary overhead.

Instead, we'll broadcast the coordinator's decision (commit/abort) to all resources. Resources without active or failed transactions for that XID will respond with XAER_INVAL or XAER_NOTA, indicating that the XID is no longer known to the resource and the transaction has terminated through a concurrent commit or rollback. This approach would streamline the process and minimizing unnecessary calls.

Update 12/02/2024