From 61ac04aacc20a2e4987cac9615ea877891c2845b Mon Sep 17 00:00:00 2001 From: Denis Angell Date: Wed, 20 Nov 2024 01:54:03 +0100 Subject: [PATCH 01/17] Sync: Ripple(d) 1.11.0 (#299) * Add jss fields used by Clio `nft_info`: (#4320) Add Clio-specific JSS constants to ensure a common vocabulary of keywords in Clio and this project. By providing visibility of the full API keyword namespace, it reduces the likelihood of developers introducing minor variations on names used by Clio, or unknowingly claiming a keyword that Clio has already claimed. This change moves this project slightly away from having only the code necessary for running the core server, but it is a step toward the goal of keeping this server's and Clio's APIs similar. The added JSS constants are annotated to indicate their relevance to Clio. Clio can be found here: https://github.com/XRPLF/clio Signed-off-by: ledhed2222 * Introduce support for a slabbed allocator: (#4218) When instantiating a large amount of fixed-sized objects on the heap the overhead that dynamic memory allocation APIs impose will quickly become significant. In some cases, allocating a large amount of memory at once and using a slabbing allocator to carve the large block into fixed-sized units that are used to service requests for memory out will help to reduce memory fragmentation significantly and, potentially, improve overall performance. This commit introduces a new `SlabAllocator<>` class that exposes an API that is _similar_ to the C++ concept of an `Allocator` but it is not meant to be a general-purpose allocator. It should not be used unless profiling and analysis of specific memory allocation patterns indicates that the additional complexity introduced will improve the performance of the system overall, and subsequent profiling proves it. A helper class, `SlabAllocatorSet<>` simplifies handling of variably sized objects that benefit from slab allocations. This commit incorporates improvements suggested by Greg Popovitch (@greg7mdp). Commit 1 of 3 in #4218. * Optimize `SHAMapItem` and leverage new slab allocator: (#4218) The `SHAMapItem` class contains a variable-sized buffer that holds the serialized data associated with a particular item inside a `SHAMap`. Prior to this commit, the buffer for the serialized data was allocated separately. Coupled with the fact that most instances of `SHAMapItem` were wrapped around a `std::shared_ptr` meant that an instantiation might result in up to three separate memory allocations. This commit switches away from `std::shared_ptr` for `SHAMapItem` and uses `boost::intrusive_ptr` instead, allowing the reference count for an instance to live inside the instance itself. Coupled with using a slab-based allocator to optimize memory allocation for the most commonly sized buffers, the net result is significant memory savings. In testing, the reduction in memory usage hovers between 400MB and 650MB. Other scenarios might result in larger savings. In performance testing with NFTs, this commit reduces memory size by about 15% sustained over long duration. Commit 2 of 3 in #4218. * Avoid using std::shared_ptr when not necessary: (#4218) The `Ledger` class contains two `SHAMap` instances: the state and transaction maps. Previously, the maps were dynamically allocated using `std::make_shared` despite the fact that they did not require lifetime management separate from the lifetime of the `Ledger` instance to which they belong. The two `SHAMap` instances are now regular member variables. Some smart pointers and dynamic memory allocation was avoided by using stack-based alternatives. Commit 3 of 3 in #4218. * Prevent replay attacks with NetworkID field: (#4370) Add a `NetworkID` field to help prevent replay attacks on and from side-chains. The new field must be used when the server is using a network id > 1024. To preserve legacy behavior, all chains with a network ID less than 1025 retain the existing behavior. This includes Mainnet, Testnet, Devnet, and hooks-testnet. If `sfNetworkID` is present in any transaction submitted to any of the nodes on one of these chains, then `telNETWORK_ID_MAKES_TX_NON_CANONICAL` is returned. Since chains with a network ID less than 1025, including Mainnet, retain the existing behavior, there is no need for an amendment. The `NetworkID` helps to prevent replay attacks because users specify a `NetworkID` field in every transaction for that chain. This change introduces a new UINT32 field, `sfNetworkID` ("NetworkID"). There are also three new local error codes for transaction results: - `telNETWORK_ID_MAKES_TX_NON_CANONICAL` - `telREQUIRES_NETWORK_ID` - `telWRONG_NETWORK` To learn about the other transaction result codes, see: https://xrpl.org/transaction-results.html Local error codes were chosen because a transaction is not necessarily malformed if it is submitted to a node running on the incorrect chain. This is a local error specific to that node and could be corrected by switching to a different node or by changing the `network_id` on that node. See: https://xrpl.org/connect-your-rippled-to-the-xrp-test-net.html In addition to using `NetworkID`, it is still generally recommended to use different accounts and keys on side-chains. However, people will undoubtedly use the same keys on multiple chains; for example, this is common practice on other blockchain networks. There are also some legitimate use cases for this. A `app.NetworkID` test suite has been added, and `core.Config` was updated to include some network_id tests. * Fix the fix for std::result_of (#4496) Newer compilers, such as Apple Clang 15.0, have removed `std::result_of` as part of C++20. The build instructions provided a fix for this (by adding a preprocessor definition), but the fix was broken. This fixes the fix by: * Adding the `conf` prefix for tool configurations (which had been forgotten). * Passing `extra_b2_flags` to `boost` package to fix its build. * Define `BOOST_ASIO_HAS_STD_INVOKE_RESULT` in order to build boost 1.77 with a newer compiler. * Use quorum specified via command line: (#4489) If `--quorum` setting is present on the command line, use the specified value as the minimum quorum. This allows for the use of a potentially fork-unsafe quorum, but it is sometimes necessary for small and test networks. Fix #4488. --------- Co-authored-by: RichardAH * Fix errors for Clang 16: (#4501) Address issues related to the removal of `std::{u,bi}nary_function` in C++17 and some warnings with Clang 16. Some warnings appeared with the upgrade to Apple clang version 14.0.3 (clang-1403.0.22.14.1). - `std::{u,bi}nary_function` were removed in C++17. They were empty classes with a few associated types. We already have conditional code to define the types. Just make it unconditional. - libc++ checks a cast in an unevaluated context to see if a type inherits from a binary function class in the standard library, e.g. `std::equal_to`, and this causes an error when the type privately inherits from such a class. Change these instances to public inheritance. - We don't need a middle-man for the empty base optimization. Prefer to inherit directly from an empty class than from `beast::detail::empty_base_optimization`. - Clang warns when all the uses of a variable are removed by conditional compilation of assertions. Add a `[[maybe_unused]]` annotation to suppress it. - As a drive-by clean-up, remove commented code. See related work in #4486. * Fix typo (#4508) * fix!: Prevent API from accepting seed or public key for account (#4404) The API would allow seeds (and public keys) to be used in place of accounts at several locations in the API. For example, when calling account_info, you could pass `"account": "foo"`. The string "foo" is treated like a seed, so the method returns `actNotFound` (instead of `actMalformed`, as most developers would expect). In the early days, this was a convenience to make testing easier. However, it allows for poor security practices, so it is no longer a good idea. Allowing a secret or passphrase is now considered a bug. Previously, it was controlled by the `strict` option on some methods. With this commit, since the API does not interpret `account` as `seed`, the option `strict` is no longer needed and is removed. Removing this behavior from the API is a [breaking change](https://xrpl.org/request-formatting.html#breaking-changes). One could argue that it shouldn't be done without bumping the API version; however, in this instance, there is no evidence that anyone is using the API in the "legacy" way. Furthermore, it is a potential security hole, as it allows users to send secrets to places where they are not needed, where they could end up in logs, error messages, etc. There's no reason to take such a risk with a seed/secret, since only the public address is needed. Resolves: #3329, #3330, #4337 BREAKING CHANGE: Remove non-strict account parsing (#3330) * Add nftoken_id, nftoken_ids, offer_id fields for NFTokens (#4447) Three new fields are added to the `Tx` responses for NFTs: 1. `nftoken_id`: This field is included in the `Tx` responses for `NFTokenMint` and `NFTokenAcceptOffer`. This field indicates the `NFTokenID` for the `NFToken` that was modified on the ledger by the transaction. 2. `nftoken_ids`: This array is included in the `Tx` response for `NFTokenCancelOffer`. This field provides a list of all the `NFTokenID`s for the `NFToken`s that were modified on the ledger by the transaction. 3. `offer_id`: This field is included in the `Tx` response for `NFTokenCreateOffer` transactions and shows the OfferID of the `NFTokenOffer` created. The fields make it easier to track specific tokens and offers. The implementation includes code (by @ledhed2222) from the Clio project to extract NFTokenIDs from mint transactions. * Ensure that switchover vars are initialized before use: (#4527) Global variables in different TUs are initialized in an undefined order. At least one global variable was accessing a global switchover variable. This caused the switchover variable to be accessed in an uninitialized state. Since the switchover is always explicitly set before transaction processing, this bug can not effect transaction processing, but could effect unit tests (and potentially the value of some global variables). Note: at the time of this patch the offending bug is not yet in production. * Move faulty assert (#4533) This assert was put in the wrong place, but it only triggers if shards are configured. This change moves the assert to the right place and updates it to ensure correctness. The assert could be hit after the server downloads some shards. It may be necessary to restart after the shards are downloaded. Note that asserts are normally checked only in debug builds, so release packages should not be affected. Introduced in: #4319 (66627b26cfae8e1c902546f0778dd9b013aedc5a) * Fix unaligned load and stores: (#4528) (#4531) Misaligned load and store operations are supported by both Intel and ARM CPUs. However, in C++, these operations are undefined behavior (UB). Substituting these operations with a `memcpy` fixes this UB. The compiled assembly code is equivalent to the original, so there is no performance penalty to using memcpy. For context: The unaligned load and store operations fixed here were originally introduced in the slab allocator (#4218). * Add missing includes for gcc 13.1: (#4555) gcc 13.1 failed to compile due to missing headers. This patch adds the needed headers. * Trivial: add comments for NFToken-related invariants (#4558) * fix node size estimation (#4536) Fix a bug in the `NODE_SIZE` auto-detection feature in `Config.cpp`. Specifically, this patch corrects the calculation for the total amount of RAM available, which was previously returned in bytes, but is now being returned in units of the system's memory unit. Additionally, the patch adjusts the node size based on the number of available hardware threads of execution. * fix: remove redundant moves (#4565) - Resolve gcc compiler warning: AccountObjects.cpp:182:47: warning: redundant move in initialization [-Wredundant-move] - The std::move() operation on trivially copyable types may generate a compile warning in newer versions of gcc. - Remove extraneous header (unused imports) from a unit test file. * Revert "Fix the fix for std::result_of (#4496)" This reverts commit cee8409d607519aabd13a0504ab12a9f18aca6e7. * Revert "Fix typo (#4508)" This reverts commit 2956f14de850291dc1d5c6cb79be333c9562a8f8. * clang * [fold] bad merge * [fold] fix bad merge - add back filter for ripple state on account_channels - add back network id test (env auto adds network id in xahau) * [fold] fix build error --------- Signed-off-by: ledhed2222 Co-authored-by: ledhed2222 Co-authored-by: Nik Bougalis Co-authored-by: RichardAH Co-authored-by: John Freeman Co-authored-by: Mark Travis Co-authored-by: solmsted Co-authored-by: drlongle Co-authored-by: Shawn Xie <35279399+shawnxie999@users.noreply.github.com> Co-authored-by: Scott Determan Co-authored-by: Ed Hennis Co-authored-by: Scott Schurr Co-authored-by: Chenna Keshava B S <21219765+ckeshava@users.noreply.github.com> --- Builds/CMake/RippledCore.cmake | 3 + src/ripple/app/consensus/RCLConsensus.cpp | 11 +- src/ripple/app/consensus/RCLCxTx.h | 11 +- src/ripple/app/ledger/Ledger.cpp | 150 +++--- src/ripple/app/ledger/Ledger.h | 27 +- src/ripple/app/ledger/LedgerHistory.cpp | 39 +- src/ripple/app/ledger/LedgerHistory.h | 2 +- src/ripple/app/ledger/LedgerReplayer.h | 2 +- src/ripple/app/ledger/TransactionMaster.h | 2 +- .../ledger/impl/LedgerReplayMsgHandler.cpp | 26 +- src/ripple/app/ledger/impl/LedgerReplayer.cpp | 2 +- src/ripple/app/ledger/impl/LedgerToJson.cpp | 12 +- .../app/ledger/impl/SkipListAcquire.cpp | 2 +- src/ripple/app/ledger/impl/SkipListAcquire.h | 2 +- .../app/ledger/impl/TransactionMaster.cpp | 2 +- src/ripple/app/main/Main.cpp | 10 +- src/ripple/app/misc/AmendmentTable.h | 3 +- src/ripple/app/misc/FeeVoteImpl.cpp | 2 +- src/ripple/app/misc/NegativeUNLVote.cpp | 12 +- src/ripple/app/misc/impl/ValidatorList.cpp | 24 +- src/ripple/app/rdb/impl/UnitaryShard.cpp | 16 +- src/ripple/app/tx/impl/InvariantCheck.h | 24 + src/ripple/basics/IOUAmount.h | 15 +- src/ripple/basics/SlabAllocator.h | 432 ++++++++++++++++++ src/ripple/basics/StringUtilities.h | 2 + src/ripple/basics/base64.h | 1 + src/ripple/basics/impl/IOUAmount.cpp | 27 +- .../basics/impl/partitioned_unordered_map.cpp | 6 +- .../container/detail/aged_ordered_container.h | 55 +-- .../detail/aged_unordered_container.h | 55 +-- src/ripple/beast/hash/impl/xxhash.cpp | 12 +- src/ripple/consensus/Consensus.h | 2 +- src/ripple/core/impl/Config.cpp | 23 +- src/ripple/json/impl/json_reader.cpp | 2 + src/ripple/net/ShardDownloader.md | 2 +- src/ripple/net/impl/RPCCall.cpp | 26 +- src/ripple/nodestore/DeterministicShard.md | 3 +- src/ripple/nodestore/ShardSizeTuning.md | 2 +- src/ripple/nodestore/impl/DecodedBlob.cpp | 1 - src/ripple/nodestore/impl/Shard.cpp | 8 +- src/ripple/peerfinder/impl/Bootcache.h | 7 - src/ripple/peerfinder/impl/Livecache.h | 11 - src/ripple/protocol/STAmount.h | 13 +- src/ripple/protocol/impl/STAmount.cpp | 35 +- src/ripple/protocol/impl/TER.cpp | 1 - src/ripple/protocol/impl/TxMeta.cpp | 6 +- src/ripple/protocol/jss.h | 6 + src/ripple/rpc/NFTSyntheticSerializer.h | 58 +++ src/ripple/rpc/NFTokenID.h | 68 +++ src/ripple/rpc/NFTokenOfferID.h | 64 +++ src/ripple/rpc/handlers/AccountChannels.cpp | 44 +- .../rpc/handlers/AccountCurrenciesHandler.cpp | 13 +- src/ripple/rpc/handlers/AccountInfo.cpp | 18 +- src/ripple/rpc/handlers/AccountLines.cpp | 53 +-- src/ripple/rpc/handlers/AccountObjects.cpp | 31 +- src/ripple/rpc/handlers/AccountOffers.cpp | 13 +- src/ripple/rpc/handlers/AccountTx.cpp | 3 + src/ripple/rpc/handlers/DepositAuthorized.cpp | 24 +- src/ripple/rpc/handlers/GatewayBalances.cpp | 28 +- src/ripple/rpc/handlers/NoRippleCheck.cpp | 14 +- src/ripple/rpc/handlers/OwnerInfo.cpp | 20 +- src/ripple/rpc/handlers/Tx.cpp | 3 + .../rpc/impl/NFTSyntheticSerializer.cpp | 50 ++ src/ripple/rpc/impl/NFTokenID.cpp | 202 ++++++++ src/ripple/rpc/impl/NFTokenOfferID.cpp | 85 ++++ src/ripple/shamap/SHAMap.h | 34 +- .../shamap/SHAMapAccountStateLeafNode.h | 4 +- src/ripple/shamap/SHAMapItem.h | 146 +++++- src/ripple/shamap/SHAMapLeafNode.h | 13 +- src/ripple/shamap/SHAMapTxLeafNode.h | 4 +- src/ripple/shamap/SHAMapTxPlusMetaLeafNode.h | 4 +- src/ripple/shamap/impl/SHAMap.cpp | 77 ++-- src/ripple/shamap/impl/SHAMapDelta.cpp | 38 +- src/ripple/shamap/impl/SHAMapInnerNode.cpp | 2 +- src/ripple/shamap/impl/SHAMapLeafNode.cpp | 10 +- src/ripple/shamap/impl/SHAMapSync.cpp | 4 +- src/ripple/shamap/impl/SHAMapTreeNode.cpp | 8 +- src/test/app/LedgerReplay_test.cpp | 4 +- src/test/app/NFToken_test.cpp | 233 ++++++++++ src/test/app/NetworkID_test.cpp | 11 +- src/test/app/ValidatorList_test.cpp | 10 +- src/test/basics/FileUtilities_test.cpp | 1 + .../consensus/ByzantineFailureSim_test.cpp | 2 +- src/test/csf/Tx.h | 6 + src/test/csf/ledgers.h | 2 +- src/test/rpc/AccountCurrencies_test.cpp | 15 +- src/test/rpc/AccountInfo_test.cpp | 17 +- src/test/rpc/AccountLinesRPC_test.cpp | 8 +- src/test/rpc/AccountObjects_test.cpp | 2 +- src/test/rpc/AccountOffers_test.cpp | 8 +- src/test/rpc/NoRippleCheck_test.cpp | 4 +- src/test/rpc/OwnerInfo_test.cpp | 10 +- src/test/rpc/RPCCall_test.cpp | 207 ++------- src/test/rpc/ReportingETL_test.cpp | 1 - src/test/shamap/FetchPack_test.cpp | 9 +- src/test/shamap/SHAMapSync_test.cpp | 11 +- src/test/shamap/SHAMap_test.cpp | 44 +- 97 files changed, 2028 insertions(+), 849 deletions(-) create mode 100644 src/ripple/basics/SlabAllocator.h create mode 100644 src/ripple/rpc/NFTSyntheticSerializer.h create mode 100644 src/ripple/rpc/NFTokenID.h create mode 100644 src/ripple/rpc/NFTokenOfferID.h create mode 100644 src/ripple/rpc/impl/NFTSyntheticSerializer.cpp create mode 100644 src/ripple/rpc/impl/NFTokenID.cpp create mode 100644 src/ripple/rpc/impl/NFTokenOfferID.cpp diff --git a/Builds/CMake/RippledCore.cmake b/Builds/CMake/RippledCore.cmake index bf24760b03..5a25d2741e 100644 --- a/Builds/CMake/RippledCore.cmake +++ b/Builds/CMake/RippledCore.cmake @@ -671,6 +671,9 @@ target_sources (rippled PRIVATE src/ripple/rpc/impl/ShardVerificationScheduler.cpp src/ripple/rpc/impl/Status.cpp src/ripple/rpc/impl/TransactionSign.cpp + src/ripple/rpc/impl/NFTokenID.cpp + src/ripple/rpc/impl/NFTokenOfferID.cpp + src/ripple/rpc/impl/NFTSyntheticSerializer.cpp #[===============================[ main sources: subdir: perflog diff --git a/src/ripple/app/consensus/RCLConsensus.cpp b/src/ripple/app/consensus/RCLConsensus.cpp index 24b695e89d..6816456060 100644 --- a/src/ripple/app/consensus/RCLConsensus.cpp +++ b/src/ripple/app/consensus/RCLConsensus.cpp @@ -186,7 +186,7 @@ RCLConsensus::Adaptor::share(RCLCxTx const& tx) if (app_.getHashRouter().shouldRelay(tx.id())) { JLOG(j_.debug()) << "Relaying disputed tx " << tx.id(); - auto const slice = tx.tx_.slice(); + auto const slice = tx.tx_->slice(); protocol::TMTransaction msg; msg.set_rawtransaction(slice.data(), slice.size()); msg.set_status(protocol::tsNEW); @@ -330,7 +330,7 @@ RCLConsensus::Adaptor::onClose( tx.first->add(s); initialSet->addItem( SHAMapNodeType::tnTRANSACTION_NM, - SHAMapItem(tx.first->getTransactionID(), s.slice())); + make_shamapitem(tx.first->getTransactionID(), s.slice())); } // Add pseudo-transactions to the set @@ -374,7 +374,8 @@ RCLConsensus::Adaptor::onClose( RCLCensorshipDetector::TxIDSeqVec proposed; initialSet->visitLeaves( - [&proposed, seq](std::shared_ptr const& item) { + [&proposed, + seq](boost::intrusive_ptr const& item) { proposed.emplace_back(item->key(), seq); }); @@ -539,7 +540,7 @@ RCLConsensus::Adaptor::doAccept( std::vector accepted; result.txns.map_->visitLeaves( - [&accepted](std::shared_ptr const& item) { + [&accepted](boost::intrusive_ptr const& item) { accepted.push_back(item->key()); }); @@ -614,7 +615,7 @@ RCLConsensus::Adaptor::doAccept( << "Test applying disputed transaction that did" << " not get in " << dispute.tx().id(); - SerialIter sit(dispute.tx().tx_.slice()); + SerialIter sit(dispute.tx().tx_->slice()); auto txn = std::make_shared(sit); // Disputed pseudo-transactions that were not accepted diff --git a/src/ripple/app/consensus/RCLCxTx.h b/src/ripple/app/consensus/RCLCxTx.h index f1c34238c2..c6abfdfee9 100644 --- a/src/ripple/app/consensus/RCLCxTx.h +++ b/src/ripple/app/consensus/RCLCxTx.h @@ -42,7 +42,7 @@ class RCLCxTx @param txn The transaction to wrap */ - RCLCxTx(SHAMapItem const& txn) : tx_{txn} + RCLCxTx(boost::intrusive_ptr txn) : tx_(std::move(txn)) { } @@ -50,11 +50,11 @@ class RCLCxTx ID const& id() const { - return tx_.key(); + return tx_->key(); } //! The SHAMapItem that represents the transaction. - SHAMapItem const tx_; + boost::intrusive_ptr tx_; }; /** Represents a set of transactions in RCLConsensus. @@ -90,8 +90,7 @@ class RCLTxSet bool insert(Tx const& t) { - return map_->addItem( - SHAMapNodeType::tnTRANSACTION_NM, SHAMapItem{t.tx_}); + return map_->addItem(SHAMapNodeType::tnTRANSACTION_NM, t.tx_); } /** Remove a transaction from the set. @@ -145,7 +144,7 @@ class RCLTxSet code uses the shared_ptr semantics to know whether the find was successful and properly creates a Tx as needed. */ - std::shared_ptr const& + boost::intrusive_ptr const& find(Tx::ID const& entry) const { return map_->peekItem(entry); diff --git a/src/ripple/app/ledger/Ledger.cpp b/src/ripple/app/ledger/Ledger.cpp index 07ab293113..081ed9c8f4 100644 --- a/src/ripple/app/ledger/Ledger.cpp +++ b/src/ripple/app/ledger/Ledger.cpp @@ -119,9 +119,8 @@ class Ledger::sles_iter_impl : public sles_type::iter_base sles_type::value_type dereference() const override { - auto const item = *iter_; - SerialIter sit(item.slice()); - return std::make_shared(sit, item.key()); + SerialIter sit(iter_->slice()); + return std::make_shared(sit, iter_->key()); } }; @@ -168,7 +167,7 @@ class Ledger::txs_iter_impl : public txs_type::iter_base txs_type::value_type dereference() const override { - auto const item = *iter_; + auto const& item = *iter_; if (metadata_) return deserializeTxPlusMeta(item); return {deserializeTx(item), nullptr}; @@ -183,8 +182,8 @@ Ledger::Ledger( std::vector const& amendments, Family& family) : mImmutable(false) - , txMap_(std::make_shared(SHAMapType::TRANSACTION, family)) - , stateMap_(std::make_shared(SHAMapType::STATE, family)) + , txMap_(SHAMapType::TRANSACTION, family) + , stateMap_(SHAMapType::STATE, family) , rules_{config.features} , j_(beast::Journal(beast::Journal::getNullSink())) { @@ -247,7 +246,7 @@ Ledger::Ledger( rawInsert(sle); } - stateMap_->flushDirty(hotACCOUNT_NODE); + stateMap_.flushDirty(hotACCOUNT_NODE); setImmutable(); } @@ -259,12 +258,8 @@ Ledger::Ledger( Family& family, beast::Journal j) : mImmutable(true) - , txMap_(std::make_shared( - SHAMapType::TRANSACTION, - info.txHash, - family)) - , stateMap_( - std::make_shared(SHAMapType::STATE, info.accountHash, family)) + , txMap_(SHAMapType::TRANSACTION, info.txHash, family) + , stateMap_(SHAMapType::STATE, info.accountHash, family) , rules_(config.features) , info_(info) , j_(j) @@ -272,7 +267,7 @@ Ledger::Ledger( loaded = true; if (info_.txHash.isNonZero() && - !txMap_->fetchRoot(SHAMapHash{info_.txHash}, nullptr)) + !txMap_.fetchRoot(SHAMapHash{info_.txHash}, nullptr)) { if (config.reporting()) { @@ -284,7 +279,7 @@ Ledger::Ledger( } if (info_.accountHash.isNonZero() && - !stateMap_->fetchRoot(SHAMapHash{info_.accountHash}, nullptr)) + !stateMap_.fetchRoot(SHAMapHash{info_.accountHash}, nullptr)) { if (config.reporting()) { @@ -295,8 +290,8 @@ Ledger::Ledger( JLOG(j.warn()) << "Don't have state data root for ledger" << info_.seq; } - txMap_->setImmutable(); - stateMap_->setImmutable(); + txMap_.setImmutable(); + stateMap_.setImmutable(); defaultFees(config); if (!setup()) @@ -313,10 +308,8 @@ Ledger::Ledger( // Create a new ledger that follows this one Ledger::Ledger(Ledger const& prevLedger, NetClock::time_point closeTime) : mImmutable(false) - , txMap_(std::make_shared( - SHAMapType::TRANSACTION, - prevLedger.stateMap_->family())) - , stateMap_(prevLedger.stateMap_->snapShot(true)) + , txMap_(SHAMapType::TRANSACTION, prevLedger.txMap_.family()) + , stateMap_(prevLedger.stateMap_, true) , fees_(prevLedger.fees_) , rules_(prevLedger.rules_) , j_(beast::Journal(beast::Journal::getNullSink())) @@ -345,12 +338,8 @@ Ledger::Ledger(Ledger const& prevLedger, NetClock::time_point closeTime) Ledger::Ledger(LedgerInfo const& info, Config const& config, Family& family) : mImmutable(true) - , txMap_(std::make_shared( - SHAMapType::TRANSACTION, - info.txHash, - family)) - , stateMap_( - std::make_shared(SHAMapType::STATE, info.accountHash, family)) + , txMap_(SHAMapType::TRANSACTION, info.txHash, family) + , stateMap_(SHAMapType::STATE, info.accountHash, family) , rules_{config.features} , info_(info) , j_(beast::Journal(beast::Journal::getNullSink())) @@ -364,8 +353,8 @@ Ledger::Ledger( Config const& config, Family& family) : mImmutable(false) - , txMap_(std::make_shared(SHAMapType::TRANSACTION, family)) - , stateMap_(std::make_shared(SHAMapType::STATE, family)) + , txMap_(SHAMapType::TRANSACTION, family) + , stateMap_(SHAMapType::STATE, family) , rules_{config.features} , j_(beast::Journal(beast::Journal::getNullSink())) { @@ -383,16 +372,16 @@ Ledger::setImmutable(bool rehash) // place the hash transitions to valid if (!mImmutable && rehash) { - info_.txHash = txMap_->getHash().as_uint256(); - info_.accountHash = stateMap_->getHash().as_uint256(); + info_.txHash = txMap_.getHash().as_uint256(); + info_.accountHash = stateMap_.getHash().as_uint256(); } if (rehash) info_.hash = calculateLedgerHash(info_); mImmutable = true; - txMap_->setImmutable(); - stateMap_->setImmutable(); + txMap_.setImmutable(); + stateMap_.setImmutable(); setup(); } @@ -415,8 +404,8 @@ bool Ledger::addSLE(SLE const& sle) { auto const s = sle.getSerializer(); - SHAMapItem item(sle.key(), s.slice()); - return stateMap_->addItem(SHAMapNodeType::tnACCOUNT_STATE, std::move(item)); + return stateMap_.addItem( + SHAMapNodeType::tnACCOUNT_STATE, make_shamapitem(sle.key(), s.slice())); } //------------------------------------------------------------------------------ @@ -451,20 +440,20 @@ bool Ledger::exists(Keylet const& k) const { // VFALCO NOTE Perhaps check the type for debug builds? - return stateMap_->hasItem(k.key); + return stateMap_.hasItem(k.key); } bool Ledger::exists(uint256 const& key) const { - return stateMap_->hasItem(key); + return stateMap_.hasItem(key); } std::optional Ledger::succ(uint256 const& key, std::optional const& last) const { - auto item = stateMap_->upper_bound(key); - if (item == stateMap_->end()) + auto item = stateMap_.upper_bound(key); + if (item == stateMap_.end()) return std::nullopt; if (last && item->key() >= last) return std::nullopt; @@ -479,7 +468,7 @@ Ledger::read(Keylet const& k) const assert(false); return nullptr; } - auto const& item = stateMap_->peekItem(k.key); + auto const& item = stateMap_.peekItem(k.key); if (!item) return nullptr; auto sle = std::make_shared(SerialIter{item->slice()}, item->key()); @@ -493,45 +482,44 @@ Ledger::read(Keylet const& k) const auto Ledger::slesBegin() const -> std::unique_ptr { - return std::make_unique(stateMap_->begin()); + return std::make_unique(stateMap_.begin()); } auto Ledger::slesEnd() const -> std::unique_ptr { - return std::make_unique(stateMap_->end()); + return std::make_unique(stateMap_.end()); } auto Ledger::slesUpperBound(uint256 const& key) const -> std::unique_ptr { - return std::make_unique(stateMap_->upper_bound(key)); + return std::make_unique(stateMap_.upper_bound(key)); } auto Ledger::txsBegin() const -> std::unique_ptr { - return std::make_unique(!open(), txMap_->begin()); + return std::make_unique(!open(), txMap_.begin()); } auto Ledger::txsEnd() const -> std::unique_ptr { - return std::make_unique(!open(), txMap_->end()); + return std::make_unique(!open(), txMap_.end()); } bool Ledger::txExists(uint256 const& key) const { - return txMap_->hasItem(key); + return txMap_.hasItem(key); } auto Ledger::txRead(key_type const& key) const -> tx_type { - assert(txMap_); - auto const& item = txMap_->peekItem(key); + auto const& item = txMap_.peekItem(key); if (!item) return {}; if (!open()) @@ -548,7 +536,7 @@ Ledger::digest(key_type const& key) const -> std::optional SHAMapHash digest; // VFALCO Unfortunately this loads the item // from the NodeStore needlessly. - if (!stateMap_->peekItem(key, digest)) + if (!stateMap_.peekItem(key, digest)) return std::nullopt; return digest.as_uint256(); } @@ -558,14 +546,14 @@ Ledger::digest(key_type const& key) const -> std::optional void Ledger::rawErase(std::shared_ptr const& sle) { - if (!stateMap_->delItem(sle->key())) + if (!stateMap_.delItem(sle->key())) LogicError("Ledger::rawErase: key not found"); } void Ledger::rawErase(uint256 const& key) { - if (!stateMap_->delItem(key)) + if (!stateMap_.delItem(key)) LogicError("Ledger::rawErase: key not found"); } @@ -574,9 +562,9 @@ Ledger::rawInsert(std::shared_ptr const& sle) { Serializer ss; sle->add(ss); - if (!stateMap_->addGiveItem( + if (!stateMap_.addGiveItem( SHAMapNodeType::tnACCOUNT_STATE, - std::make_shared(sle->key(), ss.slice()))) + make_shamapitem(sle->key(), ss.slice()))) LogicError("Ledger::rawInsert: key already exists"); } @@ -585,9 +573,9 @@ Ledger::rawReplace(std::shared_ptr const& sle) { Serializer ss; sle->add(ss); - if (!stateMap_->updateGiveItem( + if (!stateMap_.updateGiveItem( SHAMapNodeType::tnACCOUNT_STATE, - std::make_shared(sle->key(), ss.slice()))) + make_shamapitem(sle->key(), ss.slice()))) LogicError("Ledger::rawReplace: key not found"); } @@ -603,9 +591,8 @@ Ledger::rawTxInsert( Serializer s(txn->getDataLength() + metaData->getDataLength() + 16); s.addVL(txn->peekData()); s.addVL(metaData->peekData()); - if (!txMap().addGiveItem( - SHAMapNodeType::tnTRANSACTION_MD, - std::make_shared(key, s.slice()))) + if (!txMap_.addGiveItem( + SHAMapNodeType::tnTRANSACTION_MD, make_shamapitem(key, s.slice()))) LogicError("duplicate_tx: " + to_string(key)); } @@ -621,9 +608,9 @@ Ledger::rawTxInsertWithHash( Serializer s(txn->getDataLength() + metaData->getDataLength() + 16); s.addVL(txn->peekData()); s.addVL(metaData->peekData()); - auto item = std::make_shared(key, s.slice()); + auto item = make_shamapitem(key, s.slice()); auto hash = sha512Half(HashPrefix::txNode, item->slice(), item->key()); - if (!txMap().addGiveItem(SHAMapNodeType::tnTRANSACTION_MD, std::move(item))) + if (!txMap_.addGiveItem(SHAMapNodeType::tnTRANSACTION_MD, std::move(item))) LogicError("duplicate_tx: " + to_string(key)); return hash; @@ -723,7 +710,7 @@ Ledger::defaultFees(Config const& config) std::shared_ptr Ledger::peek(Keylet const& k) const { - auto const& value = stateMap_->peekItem(k.key); + auto const& value = stateMap_.peekItem(k.key); if (!value) return nullptr; auto sle = std::make_shared(SerialIter{value->slice()}, value->key()); @@ -845,8 +832,8 @@ Ledger::walkLedger(beast::Journal j, bool parallel) const std::vector missingNodes1; std::vector missingNodes2; - if (stateMap_->getHash().isZero() && !info_.accountHash.isZero() && - !stateMap_->fetchRoot(SHAMapHash{info_.accountHash}, nullptr)) + if (stateMap_.getHash().isZero() && !info_.accountHash.isZero() && + !stateMap_.fetchRoot(SHAMapHash{info_.accountHash}, nullptr)) { missingNodes1.emplace_back( SHAMapType::STATE, SHAMapHash{info_.accountHash}); @@ -854,9 +841,9 @@ Ledger::walkLedger(beast::Journal j, bool parallel) const else { if (parallel) - return stateMap_->walkMapParallel(missingNodes1, 32); + return stateMap_.walkMapParallel(missingNodes1, 32); else - stateMap_->walkMap(missingNodes1, 32); + stateMap_.walkMap(missingNodes1, 32); } if (!missingNodes1.empty()) @@ -868,15 +855,15 @@ Ledger::walkLedger(beast::Journal j, bool parallel) const } } - if (txMap_->getHash().isZero() && info_.txHash.isNonZero() && - !txMap_->fetchRoot(SHAMapHash{info_.txHash}, nullptr)) + if (txMap_.getHash().isZero() && info_.txHash.isNonZero() && + !txMap_.fetchRoot(SHAMapHash{info_.txHash}, nullptr)) { missingNodes2.emplace_back( SHAMapType::TRANSACTION, SHAMapHash{info_.txHash}); } else { - txMap_->walkMap(missingNodes2, 32); + txMap_.walkMap(missingNodes2, 32); } if (!missingNodes2.empty()) @@ -893,9 +880,9 @@ Ledger::walkLedger(beast::Journal j, bool parallel) const bool Ledger::assertSensible(beast::Journal ledgerJ) const { - if (info_.hash.isNonZero() && info_.accountHash.isNonZero() && stateMap_ && - txMap_ && (info_.accountHash == stateMap_->getHash().as_uint256()) && - (info_.txHash == txMap_->getHash().as_uint256())) + if (info_.hash.isNonZero() && info_.accountHash.isNonZero() && + (info_.accountHash == stateMap_.getHash().as_uint256()) && + (info_.txHash == txMap_.getHash().as_uint256())) { return true; } @@ -1057,15 +1044,14 @@ pendSaveValidated( return true; } - JobType const jobType{isCurrent ? jtPUBLEDGER : jtPUBOLDLEDGER}; - char const* const jobName{ - isCurrent ? "Ledger::pendSave" : "Ledger::pendOldSave"}; - // See if we can use the JobQueue. if (!isSynchronous && - app.getJobQueue().addJob(jobType, jobName, [&app, ledger, isCurrent]() { - saveValidatedLedger(app, ledger, isCurrent); - })) + app.getJobQueue().addJob( + isCurrent ? jtPUBLEDGER : jtPUBOLDLEDGER, + std::to_string(ledger->seq()), + [&app, ledger, isCurrent]() { + saveValidatedLedger(app, ledger, isCurrent); + })) { return true; } @@ -1077,15 +1063,15 @@ pendSaveValidated( void Ledger::unshare() const { - stateMap_->unshare(); - txMap_->unshare(); + stateMap_.unshare(); + txMap_.unshare(); } void Ledger::invariants() const { - stateMap_->invariants(); - txMap_->invariants(); + stateMap_.invariants(); + txMap_.invariants(); } //------------------------------------------------------------------------------ diff --git a/src/ripple/app/ledger/Ledger.h b/src/ripple/app/ledger/Ledger.h index 84e65ecffc..051b322e27 100644 --- a/src/ripple/app/ledger/Ledger.h +++ b/src/ripple/app/ledger/Ledger.h @@ -83,6 +83,10 @@ class Ledger final : public std::enable_shared_from_this, Ledger& operator=(Ledger const&) = delete; + Ledger(Ledger&&) = delete; + Ledger& + operator=(Ledger&&) = delete; + /** Create the Genesis ledger. The Genesis ledger contains a single account whose @@ -290,10 +294,10 @@ class Ledger final : public std::enable_shared_from_this, void setFull() const { - txMap_->setFull(); - stateMap_->setFull(); - txMap_->setLedgerSeq(info_.seq); - stateMap_->setLedgerSeq(info_.seq); + txMap_.setFull(); + txMap_.setLedgerSeq(info_.seq); + stateMap_.setFull(); + stateMap_.setLedgerSeq(info_.seq); } void @@ -305,25 +309,25 @@ class Ledger final : public std::enable_shared_from_this, SHAMap const& stateMap() const { - return *stateMap_; + return stateMap_; } SHAMap& stateMap() { - return *stateMap_; + return stateMap_; } SHAMap const& txMap() const { - return *txMap_; + return txMap_; } SHAMap& txMap() { - return *txMap_; + return txMap_; } // returns false on error @@ -401,8 +405,11 @@ class Ledger final : public std::enable_shared_from_this, bool mImmutable; - std::shared_ptr txMap_; - std::shared_ptr stateMap_; + // A SHAMap containing the transactions associated with this ledger. + SHAMap mutable txMap_; + + // A SHAMap containing the state objects for this ledger. + SHAMap mutable stateMap_; // Protects fee variables std::mutex mutable mutex_; diff --git a/src/ripple/app/ledger/LedgerHistory.cpp b/src/ripple/app/ledger/LedgerHistory.cpp index b239889859..2a98634d08 100644 --- a/src/ripple/app/ledger/LedgerHistory.cpp +++ b/src/ripple/app/ledger/LedgerHistory.cpp @@ -51,7 +51,9 @@ LedgerHistory::LedgerHistory( } bool -LedgerHistory::insert(std::shared_ptr ledger, bool validated) +LedgerHistory::insert( + std::shared_ptr const& ledger, + bool validated) { if (!ledger->isImmutable()) LogicError("mutable Ledger in insert"); @@ -72,12 +74,9 @@ LedgerHash LedgerHistory::getLedgerHash(LedgerIndex index) { std::unique_lock sl(m_ledgers_by_hash.peekMutex()); - auto it = mLedgersByIndex.find(index); - - if (it != mLedgersByIndex.end()) + if (auto it = mLedgersByIndex.find(index); it != mLedgersByIndex.end()) return it->second; - - return uint256(); + return {}; } std::shared_ptr @@ -167,19 +166,19 @@ log_metadata_difference( uint256 const& tx, beast::Journal j) { - auto getMeta = [](ReadView const& ledger, - uint256 const& txID) -> std::shared_ptr { - auto meta = ledger.txRead(txID).second; - if (!meta) - return {}; - return std::make_shared(txID, ledger.seq(), *meta); + auto getMeta = [](ReadView const& ledger, uint256 const& txID) { + std::optional ret; + if (auto meta = ledger.txRead(txID).second) + ret.emplace(txID, ledger.seq(), *meta); + return ret; }; auto validMetaData = getMeta(validLedger, tx); auto builtMetaData = getMeta(builtLedger, tx); - assert(validMetaData != nullptr || builtMetaData != nullptr); - if (validMetaData != nullptr && builtMetaData != nullptr) + assert(validMetaData || builtMetaData); + + if (validMetaData && builtMetaData) { auto const& validNodes = validMetaData->getNodes(); auto const& builtNodes = builtMetaData->getNodes(); @@ -280,17 +279,21 @@ log_metadata_difference( << validNodes.getJson(JsonOptions::none); } } + + return; } - else if (validMetaData != nullptr) + + if (validMetaData) { JLOG(j.error()) << "MISMATCH on TX " << tx - << ": Metadata Difference (built has none)\n" + << ": Metadata Difference. Valid=\n" << validMetaData->getJson(JsonOptions::none); } - else // builtMetaData != nullptr + + if (builtMetaData) { JLOG(j.error()) << "MISMATCH on TX " << tx - << ": Metadata Difference (valid has none)\n" + << ": Metadata Difference. Built=\n" << builtMetaData->getJson(JsonOptions::none); } } diff --git a/src/ripple/app/ledger/LedgerHistory.h b/src/ripple/app/ledger/LedgerHistory.h index be5c559bee..5733ca7637 100644 --- a/src/ripple/app/ledger/LedgerHistory.h +++ b/src/ripple/app/ledger/LedgerHistory.h @@ -44,7 +44,7 @@ class LedgerHistory @return `true` if the ledger was already tracked */ bool - insert(std::shared_ptr ledger, bool validated); + insert(std::shared_ptr const& ledger, bool validated); /** Get the ledgers_by_hash cache hit rate @return the hit rate diff --git a/src/ripple/app/ledger/LedgerReplayer.h b/src/ripple/app/ledger/LedgerReplayer.h index e9e94548b7..6866250485 100644 --- a/src/ripple/app/ledger/LedgerReplayer.h +++ b/src/ripple/app/ledger/LedgerReplayer.h @@ -105,7 +105,7 @@ class LedgerReplayer final void gotSkipList( LedgerInfo const& info, - std::shared_ptr const& data); + boost::intrusive_ptr const& data); /** * Process a ledger delta (extracted from a TMReplayDeltaResponse message) diff --git a/src/ripple/app/ledger/TransactionMaster.h b/src/ripple/app/ledger/TransactionMaster.h index c8ad336609..abac6da82c 100644 --- a/src/ripple/app/ledger/TransactionMaster.h +++ b/src/ripple/app/ledger/TransactionMaster.h @@ -68,7 +68,7 @@ class TransactionMaster std::shared_ptr fetch( - std::shared_ptr const& item, + boost::intrusive_ptr const& item, SHAMapNodeType type, std::uint32_t uCommitLedger); diff --git a/src/ripple/app/ledger/impl/LedgerReplayMsgHandler.cpp b/src/ripple/app/ledger/impl/LedgerReplayMsgHandler.cpp index 780b466497..57c2fd0834 100644 --- a/src/ripple/app/ledger/impl/LedgerReplayMsgHandler.cpp +++ b/src/ripple/app/ledger/impl/LedgerReplayMsgHandler.cpp @@ -163,15 +163,15 @@ LedgerReplayMsgHandler::processProofPathResponse( JLOG(journal_.debug()) << "Bad message: Cannot deserialize"; return false; } - auto item = static_cast(node.get())->peekItem(); - if (!item) + + if (auto item = static_cast(node.get())->peekItem()) { - JLOG(journal_.debug()) << "Bad message: Cannot get ShaMapItem"; - return false; + replayer_.gotSkipList(info, item); + return true; } - replayer_.gotSkipList(info, item); - return true; + JLOG(journal_.debug()) << "Bad message: Cannot get ShaMapItem"; + return false; } protocol::TMReplayDeltaResponse @@ -206,9 +206,10 @@ LedgerReplayMsgHandler::processReplayDeltaRequest( reply.set_ledgerheader(nData.getDataPtr(), nData.getLength()); // pack transactions auto const& txMap = ledger->txMap(); - txMap.visitLeaves([&](std::shared_ptr const& txNode) { - reply.add_transaction(txNode->data(), txNode->size()); - }); + txMap.visitLeaves( + [&](boost::intrusive_ptr const& txNode) { + reply.add_transaction(txNode->data(), txNode->size()); + }); JLOG(journal_.debug()) << "getReplayDelta for ledger " << ledgerHash << " txMap hash " << txMap.getHash().as_uint256(); @@ -264,10 +265,9 @@ LedgerReplayMsgHandler::processReplayDeltaResponse( STObject meta(metaSit, sfMetadata); orderedTxns.emplace(meta[sfTransactionIndex], std::move(tx)); - auto item = - std::make_shared(tid, shaMapItemData.slice()); - if (!item || - !txMap.addGiveItem(SHAMapNodeType::tnTRANSACTION_MD, item)) + if (!txMap.addGiveItem( + SHAMapNodeType::tnTRANSACTION_MD, + make_shamapitem(tid, shaMapItemData.slice()))) { JLOG(journal_.debug()) << "Bad message: Cannot deserialize"; return false; diff --git a/src/ripple/app/ledger/impl/LedgerReplayer.cpp b/src/ripple/app/ledger/impl/LedgerReplayer.cpp index c7aa5d9ca0..903f72dd11 100644 --- a/src/ripple/app/ledger/impl/LedgerReplayer.cpp +++ b/src/ripple/app/ledger/impl/LedgerReplayer.cpp @@ -172,7 +172,7 @@ LedgerReplayer::createDeltas(std::shared_ptr task) void LedgerReplayer::gotSkipList( LedgerInfo const& info, - std::shared_ptr const& item) + boost::intrusive_ptr const& item) { std::shared_ptr skipList = {}; { diff --git a/src/ripple/app/ledger/impl/LedgerToJson.cpp b/src/ripple/app/ledger/impl/LedgerToJson.cpp index e32303c492..558757d511 100644 --- a/src/ripple/app/ledger/impl/LedgerToJson.cpp +++ b/src/ripple/app/ledger/impl/LedgerToJson.cpp @@ -131,14 +131,14 @@ fillJsonTx( if (stMeta) { txJson[jss::metaData] = stMeta->getJson(JsonOptions::none); + + // If applicable, insert delivered amount if (txnType == ttPAYMENT || txnType == ttCHECK_CASH) - { - // Insert delivered amount - auto txMeta = std::make_shared( - txn->getTransactionID(), fill.ledger.seq(), *stMeta); RPC::insertDeliveredAmount( - txJson[jss::metaData], fill.ledger, txn, *txMeta); - } + txJson[jss::metaData], + fill.ledger, + txn, + {txn->getTransactionID(), fill.ledger.seq(), *stMeta}); } } diff --git a/src/ripple/app/ledger/impl/SkipListAcquire.cpp b/src/ripple/app/ledger/impl/SkipListAcquire.cpp index 00340d24bc..aa9b8564eb 100644 --- a/src/ripple/app/ledger/impl/SkipListAcquire.cpp +++ b/src/ripple/app/ledger/impl/SkipListAcquire.cpp @@ -137,7 +137,7 @@ SkipListAcquire::pmDowncast() void SkipListAcquire::processData( std::uint32_t ledgerSeq, - std::shared_ptr const& item) + boost::intrusive_ptr const& item) { assert(ledgerSeq != 0 && item); ScopedLockType sl(mtx_); diff --git a/src/ripple/app/ledger/impl/SkipListAcquire.h b/src/ripple/app/ledger/impl/SkipListAcquire.h index 3901e10800..df24d68312 100644 --- a/src/ripple/app/ledger/impl/SkipListAcquire.h +++ b/src/ripple/app/ledger/impl/SkipListAcquire.h @@ -96,7 +96,7 @@ class SkipListAcquire final void processData( std::uint32_t ledgerSeq, - std::shared_ptr const& item); + boost::intrusive_ptr const& item); /** * Add a callback that will be called when the skipList is ready or failed. diff --git a/src/ripple/app/ledger/impl/TransactionMaster.cpp b/src/ripple/app/ledger/impl/TransactionMaster.cpp index 1575b201fa..0301d95dca 100644 --- a/src/ripple/app/ledger/impl/TransactionMaster.cpp +++ b/src/ripple/app/ledger/impl/TransactionMaster.cpp @@ -111,7 +111,7 @@ TransactionMaster::fetch( std::shared_ptr TransactionMaster::fetch( - std::shared_ptr const& item, + boost::intrusive_ptr const& item, SHAMapNodeType type, std::uint32_t uCommitLedger) { diff --git a/src/ripple/app/main/Main.cpp b/src/ripple/app/main/Main.cpp index 7ee3372d59..86b7b48279 100644 --- a/src/ripple/app/main/Main.cpp +++ b/src/ripple/app/main/Main.cpp @@ -124,14 +124,12 @@ printHelp(const po::options_description& desc) << systemName() << "d [options] \n" << desc << std::endl << "Commands: \n" - " account_currencies [] [strict]\n" - " account_info ||| [] " - "[strict]\n" + " account_currencies []\n" + " account_info | []\n" " account_lines |\"\" []\n" " account_channels |\"\" []\n" - " account_objects [] [strict]\n" - " account_offers | [] " - "[strict]\n" + " account_objects []\n" + " account_offers | []\n" " account_tx accountID [ledger_index_min [ledger_index_max " "[limit " "]]] [binary]\n" diff --git a/src/ripple/app/misc/AmendmentTable.h b/src/ripple/app/misc/AmendmentTable.h index 0a5f6a011a..10396d8591 100644 --- a/src/ripple/app/misc/AmendmentTable.h +++ b/src/ripple/app/misc/AmendmentTable.h @@ -172,8 +172,7 @@ class AmendmentTable initialPosition->addGiveItem( SHAMapNodeType::tnTRANSACTION_NM, - std::make_shared( - amendTx.getTransactionID(), s.slice())); + make_shamapitem(amendTx.getTransactionID(), s.slice())); } } }; diff --git a/src/ripple/app/misc/FeeVoteImpl.cpp b/src/ripple/app/misc/FeeVoteImpl.cpp index 048f5a3fc6..0d60dc6b78 100644 --- a/src/ripple/app/misc/FeeVoteImpl.cpp +++ b/src/ripple/app/misc/FeeVoteImpl.cpp @@ -326,7 +326,7 @@ FeeVoteImpl::doVoting( if (!initialPosition->addGiveItem( SHAMapNodeType::tnTRANSACTION_NM, - std::make_shared(txID, s.slice()))) + make_shamapitem(txID, s.slice()))) { JLOG(journal_.warn()) << "Ledger already had fee change"; } diff --git a/src/ripple/app/misc/NegativeUNLVote.cpp b/src/ripple/app/misc/NegativeUNLVote.cpp index e057675421..0bc0b8dca3 100644 --- a/src/ripple/app/misc/NegativeUNLVote.cpp +++ b/src/ripple/app/misc/NegativeUNLVote.cpp @@ -22,6 +22,7 @@ #include #include #include +#include namespace ripple { @@ -147,7 +148,7 @@ NegativeUNLVote::addReportingTx( repUnlTx.add(s); if (!initalSet->addGiveItem( SHAMapNodeType::tnTRANSACTION_NM, - std::make_shared(txID, s.slice()))) + make_shamapitem(txID, s.slice()))) { JLOG(j_.warn()) << "R-UNL: ledger seq=" << seq << ", add ttUNL_REPORT tx failed"; @@ -202,7 +203,7 @@ NegativeUNLVote::addImportVLTx( repUnlTx.add(s); if (!initalSet->addGiveItem( SHAMapNodeType::tnTRANSACTION_NM, - std::make_shared(txID, s.slice()))) + make_shamapitem(txID, s.slice()))) { JLOG(j_.warn()) << "R-UNL: ledger seq=" << seq << ", add ttUNL_REPORT tx failed (import_vl_key)"; @@ -231,12 +232,11 @@ NegativeUNLVote::addTx( obj.setFieldVL(sfUNLModifyValidator, vp.slice()); }); - uint256 txID = negUnlTx.getTransactionID(); Serializer s; negUnlTx.add(s); if (!initialSet->addGiveItem( SHAMapNodeType::tnTRANSACTION_NM, - std::make_shared(txID, s.slice()))) + make_shamapitem(negUnlTx.getTransactionID(), s.slice()))) { JLOG(j_.warn()) << "N-UNL: ledger seq=" << seq << ", add ttUNL_MODIFY tx failed"; @@ -244,8 +244,8 @@ NegativeUNLVote::addTx( else { JLOG(j_.debug()) << "N-UNL: ledger seq=" << seq - << ", add a ttUNL_MODIFY Tx with txID: " << txID - << ", the validator to " + << ", add a ttUNL_MODIFY Tx with txID: " + << negUnlTx.getTransactionID() << ", the validator to " << (modify == ToDisable ? "disable: " : "re-enable: ") << vp; } diff --git a/src/ripple/app/misc/impl/ValidatorList.cpp b/src/ripple/app/misc/impl/ValidatorList.cpp index de0485ec6b..d17b85c484 100644 --- a/src/ripple/app/misc/impl/ValidatorList.cpp +++ b/src/ripple/app/misc/impl/ValidatorList.cpp @@ -1710,6 +1710,15 @@ ValidatorList::calculateQuorum( std::size_t effectiveUnlSize, std::size_t seenSize) { + // Use quorum if specified via command line. + if (minimumQuorum_ > 0) + { + JLOG(j_.warn()) << "Using potentially unsafe quorum of " + << *minimumQuorum_ + << " as specified on the command line"; + return *minimumQuorum_; + } + // Do not use achievable quorum until lists from all configured // publishers are available for (auto const& list : publisherLists_) @@ -1752,21 +1761,8 @@ ValidatorList::calculateQuorum( // Note that the negative UNL protocol introduced the // AbsoluteMinimumQuorum which is 60% of the original UNL size. The // effective quorum should not be lower than it. - auto quorum = static_cast(std::max( + return static_cast(std::max( std::ceil(effectiveUnlSize * 0.8f), std::ceil(unlSize * 0.6f))); - - // Use lower quorum specified via command line if the normal quorum - // appears unreachable based on the number of recently received - // validations. - if (minimumQuorum_) // < quorum && seenSize < quorum) - { - quorum = *minimumQuorum_; - - JLOG(j_.warn()) << "Using unsafe quorum of " << quorum - << " as specified in the command line"; - } - - return quorum; } TrustChanges diff --git a/src/ripple/app/rdb/impl/UnitaryShard.cpp b/src/ripple/app/rdb/impl/UnitaryShard.cpp index 56a14db58e..ab1758b485 100644 --- a/src/ripple/app/rdb/impl/UnitaryShard.cpp +++ b/src/ripple/app/rdb/impl/UnitaryShard.cpp @@ -103,22 +103,24 @@ updateLedgerDBs( for (auto const& item : ledger->txs) { - if (stop) + if (stop.load(std::memory_order_relaxed)) return false; - auto const txID{item.first->getTransactionID()}; - auto const sTxID{to_string(txID)}; - auto const txMeta{std::make_shared( - txID, ledger->seq(), *item.second)}; + TxMeta const txMeta{ + item.first->getTransactionID(), + ledger->seq(), + *item.second}; + + auto const sTxID = to_string(txMeta.getTxID()); session << "DELETE FROM AccountTransactions " "WHERE TransID = :txID;", soci::use(sTxID); - auto const& accounts = txMeta->getAffectedAccounts(); + auto const& accounts = txMeta.getAffectedAccounts(); if (!accounts.empty()) { - auto const sTxnSeq{std::to_string(txMeta->getIndex())}; + auto const sTxnSeq{std::to_string(txMeta.getIndex())}; auto const s{boost::str( boost::format("('%s','%s',%s,%s)") % sTxID % "%s" % sSeq % sTxnSeq)}; diff --git a/src/ripple/app/tx/impl/InvariantCheck.h b/src/ripple/app/tx/impl/InvariantCheck.h index 52a78c42e0..47d33a2edf 100644 --- a/src/ripple/app/tx/impl/InvariantCheck.h +++ b/src/ripple/app/tx/impl/InvariantCheck.h @@ -319,6 +319,17 @@ class ValidNewAccountRoot beast::Journal const&); }; +/** + * @brief Invariant: Validates several invariants for NFToken pages. + * + * The following checks are made: + * - The page is correctly associated with the owner. + * - The page is correctly ordered between the next and previous links. + * - The page contains at least one and no more than 32 NFTokens. + * - The NFTokens on this page do not belong on a lower or higher page. + * - The NFTokens are correctly sorted on the page. + * - Each URI, if present, is not empty. + */ class ValidNFTokenPage { bool badEntry_ = false; @@ -343,6 +354,19 @@ class ValidNFTokenPage beast::Journal const&); }; +/** + * @brief Invariant: Validates counts of NFTokens after all transaction types. + * + * The following checks are made: + * - The number of minted or burned NFTokens can only be changed by + * NFTokenMint or NFTokenBurn transactions. + * - A successful NFTokenMint must increase the number of NFTokens. + * - A failed NFTokenMint must not change the number of minted NFTokens. + * - An NFTokenMint transaction cannot change the number of burned NFTokens. + * - A successful NFTokenBurn must increase the number of burned NFTokens. + * - A failed NFTokenBurn must not change the number of burned NFTokens. + * - An NFTokenBurn transaction cannot change the number of minted NFTokens. + */ class NFTokenCountTracking { std::uint32_t beforeMintedTotal = 0; diff --git a/src/ripple/basics/IOUAmount.h b/src/ripple/basics/IOUAmount.h index 764aa38aae..2380a7d15e 100644 --- a/src/ripple/basics/IOUAmount.h +++ b/src/ripple/basics/IOUAmount.h @@ -186,7 +186,14 @@ mulRatio( std::uint32_t den, bool roundUp); -extern LocalValue stNumberSwitchover; +// Since many uses of the number class do not have access to a ledger, +// getSTNumberSwitchover needs to be globally accessible. + +bool +getSTNumberSwitchover(); + +void +setSTNumberSwitchover(bool v); /** RAII class to set and restore the Number switchover. */ @@ -198,16 +205,16 @@ class NumberSO public: ~NumberSO() { - *stNumberSwitchover = saved_; + setSTNumberSwitchover(saved_); } NumberSO(NumberSO const&) = delete; NumberSO& operator=(NumberSO const&) = delete; - explicit NumberSO(bool v) : saved_(*stNumberSwitchover) + explicit NumberSO(bool v) : saved_(getSTNumberSwitchover()) { - *stNumberSwitchover = v; + setSTNumberSwitchover(v); } }; diff --git a/src/ripple/basics/SlabAllocator.h b/src/ripple/basics/SlabAllocator.h new file mode 100644 index 0000000000..ece96d0b87 --- /dev/null +++ b/src/ripple/basics/SlabAllocator.h @@ -0,0 +1,432 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright 2022, Nikolaos D. Bougalis + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_BASICS_SLABALLOCATOR_H_INCLUDED +#define RIPPLE_BASICS_SLABALLOCATOR_H_INCLUDED + +#include + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#if BOOST_OS_LINUX +#include +#endif + +namespace ripple { + +template +class SlabAllocator +{ + static_assert( + sizeof(Type) >= sizeof(std::uint8_t*), + "SlabAllocator: the requested object must be larger than a pointer."); + + static_assert(alignof(Type) == 8 || alignof(Type) == 4); + + /** A block of memory that is owned by a slab allocator */ + struct SlabBlock + { + // A mutex to protect the freelist for this block: + std::mutex m_; + + // A linked list of appropriately sized free buffers: + std::uint8_t* l_ = nullptr; + + // The next memory block + SlabBlock* next_; + + // The underlying memory block: + std::uint8_t const* const p_ = nullptr; + + // The extent of the underlying memory block: + std::size_t const size_; + + SlabBlock( + SlabBlock* next, + std::uint8_t* data, + std::size_t size, + std::size_t item) + : next_(next), p_(data), size_(size) + { + // We don't need to grab the mutex here, since we're the only + // ones with access at this moment. + + while (data + item <= p_ + size_) + { + // Use memcpy to avoid unaligned UB + // (will optimize to equivalent code) + std::memcpy(data, &l_, sizeof(std::uint8_t*)); + l_ = data; + data += item; + } + } + + ~SlabBlock() + { + // Calling this destructor will release the allocated memory but + // will not properly destroy any objects that are constructed in + // the block itself. + } + + SlabBlock(SlabBlock const& other) = delete; + SlabBlock& + operator=(SlabBlock const& other) = delete; + + SlabBlock(SlabBlock&& other) = delete; + SlabBlock& + operator=(SlabBlock&& other) = delete; + + /** Determines whether the given pointer belongs to this allocator */ + bool + own(std::uint8_t const* p) const noexcept + { + return (p >= p_) && (p < p_ + size_); + } + + std::uint8_t* + allocate() noexcept + { + std::uint8_t* ret; + + { + std::lock_guard l(m_); + + ret = l_; + + if (ret) + { + // Use memcpy to avoid unaligned UB + // (will optimize to equivalent code) + std::memcpy(&l_, ret, sizeof(std::uint8_t*)); + } + } + + return ret; + } + + /** Return an item to this allocator's freelist. + + @param ptr The pointer to the chunk of memory being deallocated. + + @note This is a dangerous, private interface; the item being + returned should belong to this allocator. Debug builds + will check and assert if this is not the case. Release + builds will not. + */ + void + deallocate(std::uint8_t* ptr) noexcept + { + assert(own(ptr)); + + std::lock_guard l(m_); + + // Use memcpy to avoid unaligned UB + // (will optimize to equivalent code) + std::memcpy(ptr, &l_, sizeof(std::uint8_t*)); + l_ = ptr; + } + }; + +private: + // A linked list of slabs + std::atomic slabs_ = nullptr; + + // The alignment requirements of the item we're allocating: + std::size_t const itemAlignment_; + + // The size of an item, including the extra bytes requested and + // any padding needed for alignment purposes: + std::size_t const itemSize_; + + // The size of each individual slab: + std::size_t const slabSize_; + +public: + /** Constructs a slab allocator able to allocate objects of a fixed size + + @param count the number of items the slab allocator can allocate; note + that a count of 0 is valid and means that the allocator + is, effectively, disabled. This can be very useful in some + contexts (e.g. when mimimal memory usage is needed) and + allows for graceful failure. + */ + constexpr explicit SlabAllocator( + std::size_t extra, + std::size_t alloc = 0, + std::size_t align = 0) + : itemAlignment_(align ? align : alignof(Type)) + , itemSize_( + boost::alignment::align_up(sizeof(Type) + extra, itemAlignment_)) + , slabSize_(alloc) + { + assert((itemAlignment_ & (itemAlignment_ - 1)) == 0); + } + + SlabAllocator(SlabAllocator const& other) = delete; + SlabAllocator& + operator=(SlabAllocator const& other) = delete; + + SlabAllocator(SlabAllocator&& other) = delete; + SlabAllocator& + operator=(SlabAllocator&& other) = delete; + + ~SlabAllocator() + { + // FIXME: We can't destroy the memory blocks we've allocated, because + // we can't be sure that they are not being used. Cleaning the + // shutdown process up could make this possible. + } + + /** Returns the size of the memory block this allocator returns. */ + constexpr std::size_t + size() const noexcept + { + return itemSize_; + } + + /** Returns a suitably aligned pointer, if one is available. + + @return a pointer to a block of memory from the allocator, or + nullptr if the allocator can't satisfy this request. + */ + std::uint8_t* + allocate() noexcept + { + auto slab = slabs_.load(); + + while (slab != nullptr) + { + if (auto ret = slab->allocate()) + return ret; + + slab = slab->next_; + } + + // No slab can satisfy our request, so we attempt to allocate a new + // one here: + std::size_t size = slabSize_; + + // We want to allocate the memory at a 2 MiB boundary, to make it + // possible to use hugepage mappings on Linux: + auto buf = + boost::alignment::aligned_alloc(megabytes(std::size_t(2)), size); + + // clang-format off + if (!buf) [[unlikely]] + return nullptr; + // clang-format on + +#if BOOST_OS_LINUX + // When allocating large blocks, attempt to leverage Linux's + // transparent hugepage support. It is unclear and difficult + // to accurately determine if doing this impacts performance + // enough to justify using platform-specific tricks. + if (size >= megabytes(std::size_t(4))) + madvise(buf, size, MADV_HUGEPAGE); +#endif + + // We need to carve out a bit of memory for the slab header + // and then align the rest appropriately: + auto slabData = reinterpret_cast( + reinterpret_cast(buf) + sizeof(SlabBlock)); + auto slabSize = size - sizeof(SlabBlock); + + // This operation is essentially guaranteed not to fail but + // let's be careful anyways. + if (!boost::alignment::align( + itemAlignment_, itemSize_, slabData, slabSize)) + { + boost::alignment::aligned_free(buf); + return nullptr; + } + + slab = new (buf) SlabBlock( + slabs_.load(), + reinterpret_cast(slabData), + slabSize, + itemSize_); + + // Link the new slab + while (!slabs_.compare_exchange_weak( + slab->next_, + slab, + std::memory_order_release, + std::memory_order_relaxed)) + { + ; // Nothing to do + } + + return slab->allocate(); + } + + /** Returns the memory block to the allocator. + + @param ptr A pointer to a memory block. + @param size If non-zero, a hint as to the size of the block. + @return true if this memory block belonged to the allocator and has + been released; false otherwise. + */ + bool + deallocate(std::uint8_t* ptr) noexcept + { + assert(ptr); + + for (auto slab = slabs_.load(); slab != nullptr; slab = slab->next_) + { + if (slab->own(ptr)) + { + slab->deallocate(ptr); + return true; + } + } + + return false; + } +}; + +/** A collection of slab allocators of various sizes for a given type. */ +template +class SlabAllocatorSet +{ +private: + // The list of allocators that belong to this set + boost::container::static_vector, 64> allocators_; + + std::size_t maxSize_ = 0; + +public: + class SlabConfig + { + friend class SlabAllocatorSet; + + private: + std::size_t extra; + std::size_t alloc; + std::size_t align; + + public: + constexpr SlabConfig( + std::size_t extra_, + std::size_t alloc_ = 0, + std::size_t align_ = alignof(Type)) + : extra(extra_), alloc(alloc_), align(align_) + { + } + }; + + constexpr SlabAllocatorSet(std::vector cfg) + { + // Ensure that the specified allocators are sorted from smallest to + // largest by size: + std::sort( + std::begin(cfg), + std::end(cfg), + [](SlabConfig const& a, SlabConfig const& b) { + return a.extra < b.extra; + }); + + // We should never have two slabs of the same size + if (std::adjacent_find( + std::begin(cfg), + std::end(cfg), + [](SlabConfig const& a, SlabConfig const& b) { + return a.extra == b.extra; + }) != cfg.end()) + { + throw std::runtime_error( + "SlabAllocatorSet<" + beast::type_name() + + ">: duplicate slab size"); + } + + for (auto const& c : cfg) + { + auto& a = allocators_.emplace_back(c.extra, c.alloc, c.align); + + if (a.size() > maxSize_) + maxSize_ = a.size(); + } + } + + SlabAllocatorSet(SlabAllocatorSet const& other) = delete; + SlabAllocatorSet& + operator=(SlabAllocatorSet const& other) = delete; + + SlabAllocatorSet(SlabAllocatorSet&& other) = delete; + SlabAllocatorSet& + operator=(SlabAllocatorSet&& other) = delete; + + ~SlabAllocatorSet() + { + } + + /** Returns a suitably aligned pointer, if one is available. + + @param extra The number of extra bytes, above and beyond the size of + the object, that should be returned by the allocator. + + @return a pointer to a block of memory, or nullptr if the allocator + can't satisfy this request. + */ + std::uint8_t* + allocate(std::size_t extra) noexcept + { + if (auto const size = sizeof(Type) + extra; size <= maxSize_) + { + for (auto& a : allocators_) + { + if (a.size() >= size) + return a.allocate(); + } + } + + return nullptr; + } + + /** Returns the memory block to the allocator. + + @param ptr A pointer to a memory block. + + @return true if this memory block belonged to one of the allocators + in this set and has been released; false otherwise. + */ + bool + deallocate(std::uint8_t* ptr) noexcept + { + for (auto& a : allocators_) + { + if (a.deallocate(ptr)) + return true; + } + + return false; + } +}; + +} // namespace ripple + +#endif // RIPPLE_BASICS_SLABALLOCATOR_H_INCLUDED diff --git a/src/ripple/basics/StringUtilities.h b/src/ripple/basics/StringUtilities.h index 48de772ca4..8af81a3740 100644 --- a/src/ripple/basics/StringUtilities.h +++ b/src/ripple/basics/StringUtilities.h @@ -25,7 +25,9 @@ #include #include + #include +#include #include #include #include diff --git a/src/ripple/basics/base64.h b/src/ripple/basics/base64.h index ef34192d0b..05a61133f8 100644 --- a/src/ripple/basics/base64.h +++ b/src/ripple/basics/base64.h @@ -57,6 +57,7 @@ #ifndef RIPPLE_BASICS_BASE64_H_INCLUDED #define RIPPLE_BASICS_BASE64_H_INCLUDED +#include #include namespace ripple { diff --git a/src/ripple/basics/impl/IOUAmount.cpp b/src/ripple/basics/impl/IOUAmount.cpp index c9b52874ab..e3c3411057 100644 --- a/src/ripple/basics/impl/IOUAmount.cpp +++ b/src/ripple/basics/impl/IOUAmount.cpp @@ -27,7 +27,28 @@ namespace ripple { -LocalValue stNumberSwitchover(true); +namespace { + +// Use a static inside a function to help prevent order-of-initialzation issues +LocalValue& +getStaticSTNumberSwitchover() +{ + static LocalValue r{true}; + return r; +} +} // namespace + +bool +getSTNumberSwitchover() +{ + return *getStaticSTNumberSwitchover(); +} + +void +setSTNumberSwitchover(bool v) +{ + *getStaticSTNumberSwitchover() = v; +} /* The range for the mantissa when normalized */ static std::int64_t constexpr minMantissa = 1000000000000000ull; @@ -51,7 +72,7 @@ IOUAmount::normalize() return; } - if (*stNumberSwitchover) + if (getSTNumberSwitchover()) { Number const v{mantissa_, exponent_}; mantissa_ = v.mantissa(); @@ -117,7 +138,7 @@ IOUAmount::operator+=(IOUAmount const& other) return *this; } - if (*stNumberSwitchover) + if (getSTNumberSwitchover()) { *this = IOUAmount{Number{*this} + Number{other}}; return *this; diff --git a/src/ripple/basics/impl/partitioned_unordered_map.cpp b/src/ripple/basics/impl/partitioned_unordered_map.cpp index 6fb2cbec1d..3ced32eddf 100644 --- a/src/ripple/basics/impl/partitioned_unordered_map.cpp +++ b/src/ripple/basics/impl/partitioned_unordered_map.cpp @@ -31,7 +31,11 @@ namespace ripple { static std::size_t extract(uint256 const& key) { - return *reinterpret_cast(key.data()); + std::size_t result; + // Use memcpy to avoid unaligned UB + // (will optimize to equivalent code) + std::memcpy(&result, key.data(), sizeof(std::size_t)); + return result; } static std::size_t diff --git a/src/ripple/beast/container/detail/aged_ordered_container.h b/src/ripple/beast/container/detail/aged_ordered_container.h index 23534a26bb..10dca962b1 100644 --- a/src/ripple/beast/container/detail/aged_ordered_container.h +++ b/src/ripple/beast/container/detail/aged_ordered_container.h @@ -145,111 +145,78 @@ class aged_ordered_container }; // VFALCO TODO This should only be enabled for maps. - class pair_value_compare - : public beast::detail::empty_base_optimization -#ifdef _LIBCPP_VERSION - , - public std::binary_function -#endif + class pair_value_compare : public Compare { public: -#ifndef _LIBCPP_VERSION using first_argument = value_type; using second_argument = value_type; using result_type = bool; -#endif bool operator()(value_type const& lhs, value_type const& rhs) const { - return this->member()(lhs.first, rhs.first); + return Compare::operator()(lhs.first, rhs.first); } pair_value_compare() { } - pair_value_compare(pair_value_compare const& other) - : beast::detail::empty_base_optimization(other) + pair_value_compare(pair_value_compare const& other) : Compare(other) { } private: friend aged_ordered_container; - pair_value_compare(Compare const& compare) - : beast::detail::empty_base_optimization(compare) + pair_value_compare(Compare const& compare) : Compare(compare) { } }; // Compares value_type against element, used in insert_check // VFALCO TODO hoist to remove template argument dependencies - class KeyValueCompare - : public beast::detail::empty_base_optimization -#ifdef _LIBCPP_VERSION - , - public std::binary_function -#endif + class KeyValueCompare : public Compare { public: -#ifndef _LIBCPP_VERSION using first_argument = Key; using second_argument = element; using result_type = bool; -#endif KeyValueCompare() = default; - KeyValueCompare(Compare const& compare) - : beast::detail::empty_base_optimization(compare) - { - } - - // VFALCO NOTE WE might want only to enable these overloads - // if Compare has is_transparent -#if 0 - template - bool operator() (K const& k, element const& e) const + KeyValueCompare(Compare const& compare) : Compare(compare) { - return this->member() (k, extract (e.value)); } - template - bool operator() (element const& e, K const& k) const - { - return this->member() (extract (e.value), k); - } -#endif - bool operator()(Key const& k, element const& e) const { - return this->member()(k, extract(e.value)); + return Compare::operator()(k, extract(e.value)); } bool operator()(element const& e, Key const& k) const { - return this->member()(extract(e.value), k); + return Compare::operator()(extract(e.value), k); } bool operator()(element const& x, element const& y) const { - return this->member()(extract(x.value), extract(y.value)); + return Compare::operator()(extract(x.value), extract(y.value)); } Compare& compare() { - return beast::detail::empty_base_optimization::member(); + return *this; } Compare const& compare() const { - return beast::detail::empty_base_optimization::member(); + return *this; } }; diff --git a/src/ripple/beast/container/detail/aged_unordered_container.h b/src/ripple/beast/container/detail/aged_unordered_container.h index 920e6196bb..fcdccd2a63 100644 --- a/src/ripple/beast/container/detail/aged_unordered_container.h +++ b/src/ripple/beast/container/detail/aged_unordered_container.h @@ -148,115 +148,84 @@ class aged_unordered_container }; // VFALCO TODO hoist to remove template argument dependencies - class ValueHash : private beast::detail::empty_base_optimization -#ifdef _LIBCPP_VERSION - , - public std::unary_function -#endif + class ValueHash : public Hash { public: -#ifndef _LIBCPP_VERSION using argument_type = element; using result_type = size_t; -#endif ValueHash() { } - ValueHash(Hash const& h) - : beast::detail::empty_base_optimization(h) + ValueHash(Hash const& h) : Hash(h) { } std::size_t operator()(element const& e) const { - return this->member()(extract(e.value)); + return Hash::operator()(extract(e.value)); } Hash& hash_function() { - return this->member(); + return *this; } Hash const& hash_function() const { - return this->member(); + return *this; } }; // Compares value_type against element, used in find/insert_check // VFALCO TODO hoist to remove template argument dependencies - class KeyValueEqual - : private beast::detail::empty_base_optimization -#ifdef _LIBCPP_VERSION - , - public std::binary_function -#endif + class KeyValueEqual : public KeyEqual { public: -#ifndef _LIBCPP_VERSION using first_argument_type = Key; using second_argument_type = element; using result_type = bool; -#endif KeyValueEqual() { } - KeyValueEqual(KeyEqual const& keyEqual) - : beast::detail::empty_base_optimization(keyEqual) + KeyValueEqual(KeyEqual const& keyEqual) : KeyEqual(keyEqual) { } - // VFALCO NOTE WE might want only to enable these overloads - // if KeyEqual has is_transparent -#if 0 - template - bool operator() (K const& k, element const& e) const - { - return this->member() (k, extract (e.value)); - } - - template - bool operator() (element const& e, K const& k) const - { - return this->member() (extract (e.value), k); - } -#endif - bool operator()(Key const& k, element const& e) const { - return this->member()(k, extract(e.value)); + return KeyEqual::operator()(k, extract(e.value)); } bool operator()(element const& e, Key const& k) const { - return this->member()(extract(e.value), k); + return KeyEqual::operator()(extract(e.value), k); } bool operator()(element const& lhs, element const& rhs) const { - return this->member()(extract(lhs.value), extract(rhs.value)); + return KeyEqual::operator()(extract(lhs.value), extract(rhs.value)); } KeyEqual& key_eq() { - return this->member(); + return *this; } KeyEqual const& key_eq() const { - return this->member(); + return *this; } }; diff --git a/src/ripple/beast/hash/impl/xxhash.cpp b/src/ripple/beast/hash/impl/xxhash.cpp index 76d5e7997f..4a6c85db81 100644 --- a/src/ripple/beast/hash/impl/xxhash.cpp +++ b/src/ripple/beast/hash/impl/xxhash.cpp @@ -33,6 +33,8 @@ You can contact the author at : #include +#include + //************************************** // Tuning parameters //************************************** @@ -87,7 +89,7 @@ You can contact the author at : //************************************** // Includes & Memory related functions //************************************** -//#include "xxhash.h" +// #include "xxhash.h" // Modify the local functions below should you wish to use some other memory // routines for malloc(), free() #include @@ -260,7 +262,13 @@ FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align) { if (align == XXH_unaligned) - return endian == XXH_littleEndian ? A64(ptr) : XXH_swap64(A64(ptr)); + { + // Use memcpy to avoid unaligned UB + U64 tmp_aligned; + std::memcpy(&tmp_aligned, ptr, sizeof(U64)); + return endian == XXH_littleEndian ? tmp_aligned + : XXH_swap64(tmp_aligned); + } else return endian == XXH_littleEndian ? *(U64*)ptr : XXH_swap64(*(U64*)ptr); } diff --git a/src/ripple/consensus/Consensus.h b/src/ripple/consensus/Consensus.h index 09061aeb11..df5ec01cee 100644 --- a/src/ripple/consensus/Consensus.h +++ b/src/ripple/consensus/Consensus.h @@ -1644,7 +1644,7 @@ Consensus::createDisputes(TxSet_t const& o) (inThisSet && result_->txns.find(txId) && !o.find(txId)) || (!inThisSet && !result_->txns.find(txId) && o.find(txId))); - Tx_t tx = inThisSet ? *result_->txns.find(txId) : *o.find(txId); + Tx_t tx = inThisSet ? result_->txns.find(txId) : o.find(txId); auto txID = tx.id(); if (result_->disputes.find(txID) != result_->disputes.end()) diff --git a/src/ripple/core/impl/Config.cpp b/src/ripple/core/impl/Config.cpp index 5fe6b2c64a..37315d8f22 100644 --- a/src/ripple/core/impl/Config.cpp +++ b/src/ripple/core/impl/Config.cpp @@ -68,10 +68,8 @@ namespace detail { [[nodiscard]] std::uint64_t getMemorySize() { - struct sysinfo si; - - if (sysinfo(&si) == 0) - return static_cast(si.totalram); + if (struct sysinfo si; sysinfo(&si) == 0) + return static_cast(si.totalram) * si.mem_unit; return 0; } @@ -265,7 +263,8 @@ getEnvVar(char const* name) } Config::Config() - : j_(beast::Journal::getNullSink()), ramSize_(detail::getMemorySize()) + : j_(beast::Journal::getNullSink()) + , ramSize_(detail::getMemorySize() / (1024 * 1024 * 1024)) { } @@ -290,22 +289,18 @@ Config::setupControl(bool bQuiet, bool bSilent, bool bStandalone) threshold.second.begin(), threshold.second.end(), [this](std::size_t limit) { - return (ramSize_ / (1024 * 1024 * 1024)) < limit; + return (limit == 0) || (ramSize_ < limit); }); + assert(ns != threshold.second.end()); + if (ns != threshold.second.end()) NODE_SIZE = std::distance(threshold.second.begin(), ns); // Adjust the size based on the number of hardware threads of // execution available to us: - if (auto const hc = std::thread::hardware_concurrency()) - { - if (hc == 1) - NODE_SIZE = 0; - - if (hc < 4) - NODE_SIZE = std::min(NODE_SIZE, 1); - } + if (auto const hc = std::thread::hardware_concurrency(); hc != 0) + NODE_SIZE = std::min(hc / 2, NODE_SIZE); } assert(NODE_SIZE <= 4); diff --git a/src/ripple/json/impl/json_reader.cpp b/src/ripple/json/impl/json_reader.cpp index 6686b38f49..c92bea6d7e 100644 --- a/src/ripple/json/impl/json_reader.cpp +++ b/src/ripple/json/impl/json_reader.cpp @@ -19,8 +19,10 @@ #include #include + #include #include +#include #include #include diff --git a/src/ripple/net/ShardDownloader.md b/src/ripple/net/ShardDownloader.md index 9d8a33ae40..d961df61c6 100644 --- a/src/ripple/net/ShardDownloader.md +++ b/src/ripple/net/ShardDownloader.md @@ -195,7 +195,7 @@ three database entries upon completion. Since downloads execute serially by design, the entries in this table always correspond to the contents of a single file. -| Bytes | Size | Part | +| Bytes | size | Part | |:------:|:----------:|:----:| | 0x... | 2147483647 | 0 | | 0x... | 2147483647 | 1 | diff --git a/src/ripple/net/impl/RPCCall.cpp b/src/ripple/net/impl/RPCCall.cpp index cb63d997ce..3c0cdb6bdf 100644 --- a/src/ripple/net/impl/RPCCall.cpp +++ b/src/ripple/net/impl/RPCCall.cpp @@ -823,11 +823,9 @@ class RPCParser return jvRequest; } - // owner_info | [strict] - // owner_info || [] [strict] - // account_info | [strict] - // account_info || [] [strict] - // account_offers | [] [strict] + // owner_info + // account_info [] + // account_offers [] Json::Value parseAccountItems(Json::Value const& jvParams) { @@ -1050,10 +1048,7 @@ class RPCParser // Parameters 0 and 1 are accounts if (i < 2) { - if (parseBase58( - TokenType::AccountPublic, strParam) || - parseBase58(strParam) || - parseGenericSeed(strParam)) + if (parseBase58(strParam)) { jvRequest[accFields[i]] = std::move(strParam); } @@ -1079,16 +1074,8 @@ class RPCParser { std::string strIdent = jvParams[0u].asString(); unsigned int iCursor = jvParams.size(); - bool bStrict = false; - if (iCursor >= 2 && jvParams[iCursor - 1] == jss::strict) - { - bStrict = true; - --iCursor; - } - - if (!parseBase58(TokenType::AccountPublic, strIdent) && - !parseBase58(strIdent) && !parseGenericSeed(strIdent)) + if (!parseBase58(strIdent)) return rpcError(rpcACT_MALFORMED); // Get info on account. @@ -1096,9 +1083,6 @@ class RPCParser jvRequest[jss::account] = strIdent; - if (bStrict) - jvRequest[jss::strict] = 1; - if (iCursor == 2 && !jvParseLedger(jvRequest, jvParams[1u].asString())) return rpcError(rpcLGR_IDX_MALFORMED); diff --git a/src/ripple/nodestore/DeterministicShard.md b/src/ripple/nodestore/DeterministicShard.md index aff733a4fa..70d0584567 100644 --- a/src/ripple/nodestore/DeterministicShard.md +++ b/src/ripple/nodestore/DeterministicShard.md @@ -22,7 +22,7 @@ uint64 Appnum Application defined constant uint16 KeySize Key size in bytes uint64 Salt A random seed uint64 Pepper The salt hashed -uint16 BlockSize Size of a file block in bytes +uint16 BlockSize size of a file block in bytes uint16 LoadFactor Target fraction in 65536ths uint8[56] Reserved Zeroes uint8[] Reserved Zero-pad to block size @@ -160,4 +160,3 @@ Iteration 0: RIPEMD160[nudb.dat] = FAE6AE84C15968B0419FDFC014931EA12A396C71 Iteration 1: RIPEMD160[nudb.key] = F96BF2722AB2EE009FFAE4A36AAFC4F220E21951 Iteration 1: RIPEMD160[nudb.dat] = FAE6AE84C15968B0419FDFC014931EA12A396C71 ``` - diff --git a/src/ripple/nodestore/ShardSizeTuning.md b/src/ripple/nodestore/ShardSizeTuning.md index 3368fb69a7..bded73c43c 100644 --- a/src/ripple/nodestore/ShardSizeTuning.md +++ b/src/ripple/nodestore/ShardSizeTuning.md @@ -1,4 +1,4 @@ -# Shard Size Tuning +# Shard size Tuning The purpose of this document is to compare the sizes of shards containing varying amounts of ledgers. diff --git a/src/ripple/nodestore/impl/DecodedBlob.cpp b/src/ripple/nodestore/impl/DecodedBlob.cpp index 0c5a5de20c..13175d3629 100644 --- a/src/ripple/nodestore/impl/DecodedBlob.cpp +++ b/src/ripple/nodestore/impl/DecodedBlob.cpp @@ -38,7 +38,6 @@ DecodedBlob::DecodedBlob(void const* key, void const* value, int valueBytes) m_success = false; m_key = key; - // VFALCO NOTE Ledger indexes should have started at 1 m_objectType = hotUNKNOWN; m_objectData = nullptr; m_dataBytes = std::max(0, valueBytes - 9); diff --git a/src/ripple/nodestore/impl/Shard.cpp b/src/ripple/nodestore/impl/Shard.cpp index c52a2f758b..8d0eab8115 100644 --- a/src/ripple/nodestore/impl/Shard.cpp +++ b/src/ripple/nodestore/impl/Shard.cpp @@ -688,9 +688,6 @@ Shard::finalize(bool writeSQLite, std::optional const& referenceHash) ledger->stateMap().setLedgerSeq(ledgerSeq); ledger->txMap().setLedgerSeq(ledgerSeq); - // RH TODO: investigate why this assertion was failing in - // ripple.rpc.NodeToShardRPC - // assert(ledger->read(keylet::fees())); ledger->setImmutable(); if (!ledger->stateMap().fetchRoot( SHAMapHash{ledger->info().accountHash}, nullptr)) @@ -713,6 +710,11 @@ Shard::finalize(bool writeSQLite, std::optional const& referenceHash) if (writeSQLite && !storeSQLite(ledger)) return fail("failed storing to SQLite databases"); + assert( + ledger->info().seq == ledgerSeq && + (ledger->info().seq < XRP_LEDGER_EARLIEST_FEES || + ledger->read(keylet::fees()))); + hash = ledger->info().parentHash; next = std::move(ledger); diff --git a/src/ripple/peerfinder/impl/Bootcache.h b/src/ripple/peerfinder/impl/Bootcache.h index eb6455879c..b48f248ae4 100644 --- a/src/ripple/peerfinder/impl/Bootcache.h +++ b/src/ripple/peerfinder/impl/Bootcache.h @@ -91,17 +91,10 @@ class Bootcache using value_type = map_type::value_type; struct Transform -#ifdef _LIBCPP_VERSION - : std::unary_function< - map_type::right_map::const_iterator::value_type const&, - beast::IP::Endpoint const&> -#endif { -#ifndef _LIBCPP_VERSION using first_argument_type = map_type::right_map::const_iterator::value_type const&; using result_type = beast::IP::Endpoint const&; -#endif explicit Transform() = default; diff --git a/src/ripple/peerfinder/impl/Livecache.h b/src/ripple/peerfinder/impl/Livecache.h index 12e2373faa..8ecd68e845 100644 --- a/src/ripple/peerfinder/impl/Livecache.h +++ b/src/ripple/peerfinder/impl/Livecache.h @@ -69,14 +69,9 @@ class LivecacheBase public: // Iterator transformation to extract the endpoint from Element struct Transform -#ifdef _LIBCPP_VERSION - : public std::unary_function -#endif { -#ifndef _LIBCPP_VERSION using first_argument = Element; using result_type = Endpoint; -#endif explicit Transform() = default; @@ -239,15 +234,9 @@ class Livecache : protected detail::LivecacheBase template struct Transform -#ifdef _LIBCPP_VERSION - : public std:: - unary_function> -#endif { -#ifndef _LIBCPP_VERSION using first_argument = typename lists_type::value_type; using result_type = Hop; -#endif explicit Transform() = default; diff --git a/src/ripple/protocol/STAmount.h b/src/ripple/protocol/STAmount.h index 5ded2ae22a..ecfaaa3824 100644 --- a/src/ripple/protocol/STAmount.h +++ b/src/ripple/protocol/STAmount.h @@ -587,7 +587,12 @@ isAddable(STAmount const& amt1, STAmount const& amt2) // the low-level routine stAmountCanonicalize on an amendment switch. Only // transactions need to use this switchover. Outside of a transaction it's safe // to unconditionally use the new behavior. -extern LocalValue stAmountCanonicalizeSwitchover; + +bool +getSTAmountCanonicalizeSwitchover(); + +void +setSTAmountCanonicalizeSwitchover(bool v); /** RAII class to set and restore the STAmount canonicalize switchover. */ @@ -595,14 +600,14 @@ extern LocalValue stAmountCanonicalizeSwitchover; class STAmountSO { public: - explicit STAmountSO(bool v) : saved_(*stAmountCanonicalizeSwitchover) + explicit STAmountSO(bool v) : saved_(getSTAmountCanonicalizeSwitchover()) { - *stAmountCanonicalizeSwitchover = v; + setSTAmountCanonicalizeSwitchover(v); } ~STAmountSO() { - *stAmountCanonicalizeSwitchover = saved_; + setSTAmountCanonicalizeSwitchover(saved_); } private: diff --git a/src/ripple/protocol/impl/STAmount.cpp b/src/ripple/protocol/impl/STAmount.cpp index d1a878c8b4..02e3345944 100644 --- a/src/ripple/protocol/impl/STAmount.cpp +++ b/src/ripple/protocol/impl/STAmount.cpp @@ -34,7 +34,28 @@ namespace ripple { -LocalValue stAmountCanonicalizeSwitchover(true); +namespace { + +// Use a static inside a function to help prevent order-of-initialzation issues +LocalValue& +getStaticSTAmountCanonicalizeSwitchover() +{ + static LocalValue r{true}; + return r; +} +} // namespace + +bool +getSTAmountCanonicalizeSwitchover() +{ + return *getStaticSTAmountCanonicalizeSwitchover(); +} + +void +setSTAmountCanonicalizeSwitchover(bool v) +{ + *getStaticSTAmountCanonicalizeSwitchover() = v; +} static const std::uint64_t tenTo14 = 100000000000000ull; static const std::uint64_t tenTo14m1 = tenTo14 - 1; @@ -395,7 +416,7 @@ operator+(STAmount const& v1, STAmount const& v2) if (v1.native()) return {v1.getFName(), getSNValue(v1) + getSNValue(v2)}; - if (*stNumberSwitchover) + if (getSTNumberSwitchover()) { auto x = v1; x = v1.iou() + v2.iou(); @@ -717,7 +738,7 @@ STAmount::canonicalize() return; } - if (*stAmountCanonicalizeSwitchover) + if (getSTAmountCanonicalizeSwitchover()) { // log(cMaxNativeN, 10) == 17 if (mOffset > 17) @@ -725,7 +746,7 @@ STAmount::canonicalize() "Native currency amount out of range"); } - if (*stNumberSwitchover && *stAmountCanonicalizeSwitchover) + if (getSTNumberSwitchover() && getSTAmountCanonicalizeSwitchover()) { Number num( mIsNegative ? -mValue : mValue, mOffset, Number::unchecked{}); @@ -744,7 +765,7 @@ STAmount::canonicalize() while (mOffset > 0) { - if (*stAmountCanonicalizeSwitchover) + if (getSTAmountCanonicalizeSwitchover()) { // N.B. do not move the overflow check to after the // multiplication @@ -765,7 +786,7 @@ STAmount::canonicalize() mIsNative = false; - if (*stNumberSwitchover) + if (getSTNumberSwitchover()) { *this = iou(); return; @@ -1208,7 +1229,7 @@ multiply(STAmount const& v1, STAmount const& v2, Issue const& issue) return STAmount(v1.getFName(), minV * maxV); } - if (*stNumberSwitchover) + if (getSTNumberSwitchover()) return {IOUAmount{Number{v1} * Number{v2}}, issue}; std::uint64_t value1 = v1.mantissa(); diff --git a/src/ripple/protocol/impl/TER.cpp b/src/ripple/protocol/impl/TER.cpp index bc8a41fa96..be8be1dd42 100644 --- a/src/ripple/protocol/impl/TER.cpp +++ b/src/ripple/protocol/impl/TER.cpp @@ -136,7 +136,6 @@ transResults() MAKE_ERROR(telNON_LOCAL_EMITTED_TXN, "Emitted transaction cannot be applied because it was not generated locally."), MAKE_ERROR(telIMPORT_VL_KEY_NOT_RECOGNISED, "Import vl key was not recognized."), MAKE_ERROR(telCAN_NOT_QUEUE_IMPORT, "Import transaction was not able to be directly applied and cannot be queued."), - MAKE_ERROR(temMALFORMED, "Malformed transaction."), MAKE_ERROR(temBAD_AMOUNT, "Can only send positive amounts."), MAKE_ERROR(temBAD_CURRENCY, "Malformed: Bad currency."), diff --git a/src/ripple/protocol/impl/TxMeta.cpp b/src/ripple/protocol/impl/TxMeta.cpp index 4014b464ed..4b48f5eb4a 100644 --- a/src/ripple/protocol/impl/TxMeta.cpp +++ b/src/ripple/protocol/impl/TxMeta.cpp @@ -138,8 +138,7 @@ TxMeta::getAffectedAccounts() const if (index != -1) { - const STObject* inner = - dynamic_cast(&it.peekAtIndex(index)); + auto inner = dynamic_cast(&it.peekAtIndex(index)); assert(inner); if (inner) { @@ -157,8 +156,7 @@ TxMeta::getAffectedAccounts() const (field.getFName() == sfTakerPays) || (field.getFName() == sfTakerGets)) { - const STAmount* lim = - dynamic_cast(&field); + auto lim = dynamic_cast(&field); assert(lim); if (lim != nullptr) diff --git a/src/ripple/protocol/jss.h b/src/ripple/protocol/jss.h index 824c61a6e8..963434090e 100644 --- a/src/ripple/protocol/jss.h +++ b/src/ripple/protocol/jss.h @@ -368,6 +368,7 @@ JSS(invalid_API_version); // out: Many, when a request has an invalid // version JSS(io_latency_ms); // out: NetworkOPs JSS(ip); // in: Connect, out: OverlayImpl +JSS(is_burned); // out: nft_info (clio) JSS(issuer); // in: RipplePathFind, Subscribe, // Unsubscribe, BookOffers // out: STPathSet, STAmount @@ -483,6 +484,9 @@ JSS(nft_offer); // in: LedgerEntry JSS(nft_offer_index); // out nft_buy_offers, nft_sell_offers JSS(nft_page); // in: LedgerEntry JSS(nft_serial); // out: account_nfts +JSS(nft_taxon); // out: nft_info (clio) +JSS(nftoken_id); // out: insertNFTokenID +JSS(nftoken_ids); // out: insertNFTokenID JSS(no_ripple); // out: AccountLines JSS(no_ripple_peer); // out: AccountLines JSS(node); // out: LedgerEntry @@ -504,6 +508,7 @@ JSS(nth); // out: RPC server_definitions JSS(obligations); // out: GatewayBalances JSS(offer); // in: LedgerEntry JSS(offers); // out: NetworkOPs, AccountOffers, Subscribe +JSS(offer_id); // out: insertNFTokenOfferID JSS(offline); // in: TransactionSign JSS(offset); // in/out: AccountTxOld JSS(open); // out: handlers/Ledger @@ -660,6 +665,7 @@ JSS(transaction); // in: Tx JSS(transaction_hash); // out: RCLCxPeerPos, LedgerToJson JSS(transactions); // out: LedgerToJson, // in: AccountTx*, Unsubscribe +JSS(transfer_rate); // out: nft_info (clio) JSS(transitions); // out: NetworkOPs JSS(treenode_cache_size); // out: GetCounts JSS(treenode_track_size); // out: GetCounts diff --git a/src/ripple/rpc/NFTSyntheticSerializer.h b/src/ripple/rpc/NFTSyntheticSerializer.h new file mode 100644 index 0000000000..090e893786 --- /dev/null +++ b/src/ripple/rpc/NFTSyntheticSerializer.h @@ -0,0 +1,58 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_RPC_NFTSYNTHETICSERIALIZER_H_INCLUDED +#define RIPPLE_RPC_NFTSYNTHETICSERIALIZER_H_INCLUDED + +#include +#include + +#include +#include + +namespace Json { +class Value; +} + +namespace ripple { + +class TxMeta; +class STTx; + +namespace RPC { + +struct JsonContext; + +/** + Adds common synthetic fields to transaction-related JSON responses + + @{ + */ +void +insertNFTSyntheticInJson( + Json::Value&, + RPC::JsonContext const&, + std::shared_ptr const&, + TxMeta const&); +/** @} */ + +} // namespace RPC +} // namespace ripple + +#endif diff --git a/src/ripple/rpc/NFTokenID.h b/src/ripple/rpc/NFTokenID.h new file mode 100644 index 0000000000..cb218966fd --- /dev/null +++ b/src/ripple/rpc/NFTokenID.h @@ -0,0 +1,68 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_RPC_NFTOKENID_H_INCLUDED +#define RIPPLE_RPC_NFTOKENID_H_INCLUDED + +#include + +#include +#include + +namespace Json { +class Value; +} + +namespace ripple { + +class TxMeta; +class STTx; + +namespace RPC { + +/** + Add a `nftoken_ids` field to the `meta` output parameter. + The field is only added to successful NFTokenMint, NFTokenAcceptOffer, + and NFTokenCancelOffer transactions. + + Helper functions are not static because they can be used by Clio. + @{ + */ +bool +canHaveNFTokenID( + std::shared_ptr const& serializedTx, + TxMeta const& transactionMeta); + +std::optional +getNFTokenIDFromPage(TxMeta const& transactionMeta); + +std::vector +getNFTokenIDFromDeletedOffer(TxMeta const& transactionMeta); + +void +insertNFTokenID( + Json::Value& response, + std::shared_ptr const& transaction, + TxMeta const& transactionMeta); +/** @} */ + +} // namespace RPC +} // namespace ripple + +#endif diff --git a/src/ripple/rpc/NFTokenOfferID.h b/src/ripple/rpc/NFTokenOfferID.h new file mode 100644 index 0000000000..6c1bef3d12 --- /dev/null +++ b/src/ripple/rpc/NFTokenOfferID.h @@ -0,0 +1,64 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_RPC_NFTOKENOFFERID_H_INCLUDED +#define RIPPLE_RPC_NFTOKENOFFERID_H_INCLUDED + +#include + +#include +#include + +namespace Json { +class Value; +} + +namespace ripple { + +class TxMeta; +class STTx; + +namespace RPC { + +/** + Add an `offer_id` field to the `meta` output parameter. + The field is only added to successful NFTokenCreateOffer transactions. + + Helper functions are not static because they can be used by Clio. + @{ + */ +bool +canHaveNFTokenOfferID( + std::shared_ptr const& serializedTx, + TxMeta const& transactionMeta); + +std::optional +getOfferIDFromCreatedOffer(TxMeta const& transactionMeta); + +void +insertNFTokenOfferID( + Json::Value& response, + std::shared_ptr const& transaction, + TxMeta const& transactionMeta); +/** @} */ + +} // namespace RPC +} // namespace ripple + +#endif diff --git a/src/ripple/rpc/handlers/AccountChannels.cpp b/src/ripple/rpc/handlers/AccountChannels.cpp index 4476d4a3a0..229222b6a7 100644 --- a/src/ripple/rpc/handlers/AccountChannels.cpp +++ b/src/ripple/rpc/handlers/AccountChannels.cpp @@ -58,7 +58,7 @@ addChannel(Json::Value& jsonLines, SLE const& line) } // { -// account: | +// account: // ledger_hash : // ledger_index : // limit: integer // optional @@ -76,11 +76,12 @@ doAccountChannels(RPC::JsonContext& context) if (!ledger) return result; - std::string strIdent(params[jss::account].asString()); - AccountID accountID; - - if (auto const err = RPC::accountFromString(accountID, strIdent)) - return err; + auto id = parseBase58(params[jss::account].asString()); + if (!id) + { + return rpcError(rpcACT_MALFORMED); + } + AccountID const accountID{std::move(id.value())}; if (!ledger->exists(keylet::account(accountID))) return rpcError(rpcACT_NOT_FOUND); @@ -88,14 +89,12 @@ doAccountChannels(RPC::JsonContext& context) std::string strDst; if (params.isMember(jss::destination_account)) strDst = params[jss::destination_account].asString(); - auto hasDst = !strDst.empty(); - AccountID raDstAccount; - if (hasDst) - { - if (auto const err = RPC::accountFromString(raDstAccount, strDst)) - return err; - } + auto const raDstAccount = [&]() -> std::optional { + return strDst.empty() ? std::nullopt : parseBase58(strDst); + }(); + if (!strDst.empty() && !raDstAccount) + return rpcError(rpcACT_MALFORMED); unsigned int limit; if (auto err = readLimitField(limit, RPC::Tuning::accountChannels, context)) @@ -109,10 +108,9 @@ doAccountChannels(RPC::JsonContext& context) { std::vector> items; AccountID const& accountID; - bool hasDst; - AccountID const& raDstAccount; + std::optional const& raDstAccount; }; - VisitData visitData = {{}, accountID, hasDst, raDstAccount}; + VisitData visitData = {{}, accountID, raDstAccount}; visitData.items.reserve(limit); uint256 startAfter = beast::zero; std::uint64_t startHint = 0; @@ -163,7 +161,7 @@ doAccountChannels(RPC::JsonContext& context) accountID, startAfter, startHint, - limit, + limit + 1, [&visitData, &accountID, &count, &limit, &marker, &nextHint]( std::shared_ptr const& sleCur) { if (!sleCur) @@ -172,10 +170,9 @@ doAccountChannels(RPC::JsonContext& context) return false; } + // Filter out ripple state objects. if (sleCur->getType() != ltPAYCHAN) - { return false; - } if (++count == limit) { @@ -183,9 +180,10 @@ doAccountChannels(RPC::JsonContext& context) nextHint = RPC::getStartHint(sleCur, visitData.accountID); } - if (count <= limit && (*sleCur)[sfAccount] == accountID && - (!visitData.hasDst || - visitData.raDstAccount == (*sleCur)[sfDestination])) + if (count <= limit && sleCur->getType() == ltPAYCHAN && + (*sleCur)[sfAccount] == accountID && + (!visitData.raDstAccount || + *visitData.raDstAccount == (*sleCur)[sfDestination])) { visitData.items.emplace_back(sleCur); } @@ -199,7 +197,7 @@ doAccountChannels(RPC::JsonContext& context) // Both conditions need to be checked because marker is set on the limit-th // item, but if there is no item on the limit + 1 iteration, then there is // no need to return a marker. - if (count == limit && marker) + if (count == limit + 1 && marker) { result[jss::limit] = limit; result[jss::marker] = diff --git a/src/ripple/rpc/handlers/AccountCurrenciesHandler.cpp b/src/ripple/rpc/handlers/AccountCurrenciesHandler.cpp index d735e5976f..64956a7d0a 100644 --- a/src/ripple/rpc/handlers/AccountCurrenciesHandler.cpp +++ b/src/ripple/rpc/handlers/AccountCurrenciesHandler.cpp @@ -46,13 +46,14 @@ doAccountCurrencies(RPC::JsonContext& context) params.isMember(jss::account) ? params[jss::account].asString() : params[jss::ident].asString()); - bool const bStrict = - params.isMember(jss::strict) && params[jss::strict].asBool(); - // Get info on account. - AccountID accountID; // out param - if (auto jvAccepted = RPC::accountFromString(accountID, strIdent, bStrict)) - return jvAccepted; + auto id = parseBase58(strIdent); + if (!id) + { + RPC::inject_error(rpcACT_MALFORMED, result); + return result; + } + auto const accountID{std::move(id.value())}; if (!ledger->exists(keylet::account(accountID))) return rpcError(rpcACT_NOT_FOUND); diff --git a/src/ripple/rpc/handlers/AccountInfo.cpp b/src/ripple/rpc/handlers/AccountInfo.cpp index 5b85fc2575..b71fd48654 100644 --- a/src/ripple/rpc/handlers/AccountInfo.cpp +++ b/src/ripple/rpc/handlers/AccountInfo.cpp @@ -34,8 +34,6 @@ namespace ripple { // { // account: , -// strict: // optional (default false) -// // if true only allow public keys and addresses. // ledger_hash : // ledger_index : // signer_lists : // optional (default false) @@ -67,15 +65,14 @@ doAccountInfo(RPC::JsonContext& context) if (!ledger) return result; - bool bStrict = params.isMember(jss::strict) && params[jss::strict].asBool(); - AccountID accountID; - // Get info on account. - - auto jvAccepted = RPC::accountFromString(accountID, strIdent, bStrict); - - if (jvAccepted) - return jvAccepted; + auto id = parseBase58(strIdent); + if (!id) + { + RPC::inject_error(rpcACT_MALFORMED, result); + return result; + } + auto const accountID{std::move(id.value())}; static constexpr std:: array, 10> @@ -115,6 +112,7 @@ doAccountInfo(RPC::JsonContext& context) return result; } + Json::Value jvAccepted(Json::objectValue); RPC::injectSLE(jvAccepted, *sleAccepted); result[jss::account_data] = jvAccepted; diff --git a/src/ripple/rpc/handlers/AccountLines.cpp b/src/ripple/rpc/handlers/AccountLines.cpp index 10d6e7255f..ace5b3898f 100644 --- a/src/ripple/rpc/handlers/AccountLines.cpp +++ b/src/ripple/rpc/handlers/AccountLines.cpp @@ -30,17 +30,6 @@ namespace ripple { -struct VisitData -{ - std::vector items; - AccountID const& accountID; - bool hasPeer; - AccountID const& raPeerAccount; - - bool ignoreDefault; - uint32_t foundCount; -}; - void addLine(Json::Value& jsonLines, RPCTrustLine const& line) { @@ -88,7 +77,7 @@ addLine(Json::Value& jsonLines, RPCTrustLine const& line) } // { -// account: | +// account: // ledger_hash : // ledger_index : // limit: integer // optional @@ -108,15 +97,13 @@ doAccountLines(RPC::JsonContext& context) if (!ledger) return result; - std::string strIdent(params[jss::account].asString()); - AccountID accountID; - - if (auto jv = RPC::accountFromString(accountID, strIdent)) + auto id = parseBase58(params[jss::account].asString()); + if (!id) { - for (auto it = jv.begin(); it != jv.end(); ++it) - result[it.memberName()] = *it; + RPC::inject_error(rpcACT_MALFORMED, result); return result; } + auto const accountID{std::move(id.value())}; if (!ledger->exists(keylet::account(accountID))) return rpcError(rpcACT_NOT_FOUND); @@ -124,17 +111,14 @@ doAccountLines(RPC::JsonContext& context) std::string strPeer; if (params.isMember(jss::peer)) strPeer = params[jss::peer].asString(); - auto hasPeer = !strPeer.empty(); - AccountID raPeerAccount; - if (hasPeer) + auto const raPeerAccount = [&]() -> std::optional { + return strPeer.empty() ? std::nullopt : parseBase58(strPeer); + }(); + if (!strPeer.empty() && !raPeerAccount) { - if (auto jv = RPC::accountFromString(raPeerAccount, strPeer)) - { - for (auto it = jv.begin(); it != jv.end(); ++it) - result[it.memberName()] = *it; - return result; - } + RPC::inject_error(rpcACT_MALFORMED, result); + return result; } unsigned int limit; @@ -150,8 +134,15 @@ doAccountLines(RPC::JsonContext& context) params[jss::ignore_default].asBool(); Json::Value& jsonLines(result[jss::lines] = Json::arrayValue); - VisitData visitData = { - {}, accountID, hasPeer, raPeerAccount, ignoreDefault, 0}; + struct VisitData + { + std::vector items; + AccountID const& accountID; + std::optional const& raPeerAccount; + bool ignoreDefault; + uint32_t foundCount; + }; + VisitData visitData = {{}, accountID, raPeerAccount, ignoreDefault, 0}; uint256 startAfter = beast::zero; std::uint64_t startHint = 0; @@ -239,8 +230,8 @@ doAccountLines(RPC::JsonContext& context) RPCTrustLine::makeItem(visitData.accountID, sleCur); if (line && - (!visitData.hasPeer || - visitData.raPeerAccount == + (!visitData.raPeerAccount || + *visitData.raPeerAccount == line->getAccountIDPeer())) { visitData.items.emplace_back(*line); diff --git a/src/ripple/rpc/handlers/AccountObjects.cpp b/src/ripple/rpc/handlers/AccountObjects.cpp index 26611ea87e..1238456869 100644 --- a/src/ripple/rpc/handlers/AccountObjects.cpp +++ b/src/ripple/rpc/handlers/AccountObjects.cpp @@ -19,7 +19,6 @@ #include #include -#include #include #include #include @@ -39,7 +38,7 @@ namespace ripple { /** General RPC command that can retrieve objects in the account root. { - account: | + account: ledger_hash: // optional ledger_index: // optional type: // optional, defaults to all account objects types @@ -60,17 +59,13 @@ doAccountNFTs(RPC::JsonContext& context) if (ledger == nullptr) return result; - AccountID accountID; + auto id = parseBase58(params[jss::account].asString()); + if (!id) { - auto const strIdent = params[jss::account].asString(); - if (auto jv = RPC::accountFromString(accountID, strIdent)) - { - for (auto it = jv.begin(); it != jv.end(); ++it) - result[it.memberName()] = *it; - - return result; - } + RPC::inject_error(rpcACT_MALFORMED, result); + return result; } + auto const accountID{id.value()}; if (!ledger->exists(keylet::account(accountID))) return rpcError(rpcACT_NOT_FOUND); @@ -177,17 +172,13 @@ doAccountObjects(RPC::JsonContext& context) if (ledger == nullptr) return result; - AccountID accountID; + auto const id = parseBase58(params[jss::account].asString()); + if (!id) { - auto const strIdent = params[jss::account].asString(); - if (auto jv = RPC::accountFromString(accountID, strIdent)) - { - for (auto it = jv.begin(); it != jv.end(); ++it) - result[it.memberName()] = *it; - - return result; - } + RPC::inject_error(rpcACT_MALFORMED, result); + return result; } + auto const accountID{id.value()}; if (!ledger->exists(keylet::account(accountID))) return rpcError(rpcACT_NOT_FOUND); diff --git a/src/ripple/rpc/handlers/AccountOffers.cpp b/src/ripple/rpc/handlers/AccountOffers.cpp index 409d071fb0..1afd273255 100644 --- a/src/ripple/rpc/handlers/AccountOffers.cpp +++ b/src/ripple/rpc/handlers/AccountOffers.cpp @@ -47,7 +47,7 @@ appendOfferJson(std::shared_ptr const& offer, Json::Value& offers) }; // { -// account: | +// account: // ledger_hash : // ledger_index : // limit: integer // optional @@ -65,16 +65,13 @@ doAccountOffers(RPC::JsonContext& context) if (!ledger) return result; - std::string strIdent(params[jss::account].asString()); - AccountID accountID; - - if (auto jv = RPC::accountFromString(accountID, strIdent)) + auto id = parseBase58(params[jss::account].asString()); + if (!id) { - for (auto it = jv.begin(); it != jv.end(); ++it) - result[it.memberName()] = (*it); - + RPC::inject_error(rpcACT_MALFORMED, result); return result; } + auto const accountID{std::move(id.value())}; // Get info on account. result[jss::account] = toBase58(accountID); diff --git a/src/ripple/rpc/handlers/AccountTx.cpp b/src/ripple/rpc/handlers/AccountTx.cpp index 67c80ad9bd..f65657d92e 100644 --- a/src/ripple/rpc/handlers/AccountTx.cpp +++ b/src/ripple/rpc/handlers/AccountTx.cpp @@ -34,6 +34,7 @@ #include #include #include +#include #include #include @@ -307,6 +308,8 @@ populateJsonResponse( jvObj[jss::validated] = true; insertDeliveredAmount( jvObj[jss::meta], context, txn, *txnMeta); + insertNFTSyntheticInJson( + jvObj, context, txn->getSTransaction(), *txnMeta); } } } diff --git a/src/ripple/rpc/handlers/DepositAuthorized.cpp b/src/ripple/rpc/handlers/DepositAuthorized.cpp index a74db92437..a5c9c9a21f 100644 --- a/src/ripple/rpc/handlers/DepositAuthorized.cpp +++ b/src/ripple/rpc/handlers/DepositAuthorized.cpp @@ -18,6 +18,7 @@ //============================================================================== #include +#include #include #include #include @@ -46,13 +47,10 @@ doDepositAuthorized(RPC::JsonContext& context) rpcINVALID_PARAMS, RPC::expected_field_message(jss::source_account, "a string")); - AccountID srcAcct; - { - Json::Value const jvAccepted = RPC::accountFromString( - srcAcct, params[jss::source_account].asString(), true); - if (jvAccepted) - return jvAccepted; - } + auto srcID = parseBase58(params[jss::source_account].asString()); + if (!srcID) + return rpcError(rpcACT_MALFORMED); + auto const srcAcct{std::move(srcID.value())}; // Validate destination_account. if (!params.isMember(jss::destination_account)) @@ -62,13 +60,11 @@ doDepositAuthorized(RPC::JsonContext& context) rpcINVALID_PARAMS, RPC::expected_field_message(jss::destination_account, "a string")); - AccountID dstAcct; - { - Json::Value const jvAccepted = RPC::accountFromString( - dstAcct, params[jss::destination_account].asString(), true); - if (jvAccepted) - return jvAccepted; - } + auto dstID = + parseBase58(params[jss::destination_account].asString()); + if (!dstID) + return rpcError(rpcACT_MALFORMED); + auto const dstAcct{std::move(dstID.value())}; // Validate ledger. std::shared_ptr ledger; diff --git a/src/ripple/rpc/handlers/GatewayBalances.cpp b/src/ripple/rpc/handlers/GatewayBalances.cpp index 3a422c6e96..77cec496ed 100644 --- a/src/ripple/rpc/handlers/GatewayBalances.cpp +++ b/src/ripple/rpc/handlers/GatewayBalances.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -68,16 +69,11 @@ doGatewayBalances(RPC::JsonContext& context) params.isMember(jss::account) ? params[jss::account].asString() : params[jss::ident].asString()); - bool const bStrict = - params.isMember(jss::strict) && params[jss::strict].asBool(); - // Get info on account. - AccountID accountID; - auto jvAccepted = RPC::accountFromString(accountID, strIdent, bStrict); - - if (jvAccepted) - return jvAccepted; - + auto id = parseBase58(strIdent); + if (!id) + return rpcError(rpcACT_MALFORMED); + auto const accountID{std::move(id.value())}; context.loadType = Resource::feeHighBurdenRPC; result[jss::account] = toBase58(accountID); @@ -90,19 +86,9 @@ doGatewayBalances(RPC::JsonContext& context) auto addHotWallet = [&hotWallets](Json::Value const& j) { if (j.isString()) { - auto const pk = parseBase58( - TokenType::AccountPublic, j.asString()); - if (pk) - { - hotWallets.insert(calcAccountID(*pk)); - return true; - } - - auto const id = parseBase58(j.asString()); - - if (id) + if (auto id = parseBase58(j.asString()); id) { - hotWallets.insert(*id); + hotWallets.insert(std::move(id.value())); return true; } } diff --git a/src/ripple/rpc/handlers/NoRippleCheck.cpp b/src/ripple/rpc/handlers/NoRippleCheck.cpp index 18156ea424..20137c985c 100644 --- a/src/ripple/rpc/handlers/NoRippleCheck.cpp +++ b/src/ripple/rpc/handlers/NoRippleCheck.cpp @@ -50,7 +50,7 @@ fillTransaction( } // { -// account: | +// account: // ledger_hash : // ledger_index : // limit: integer // optional, number of problems @@ -92,17 +92,13 @@ doNoRippleCheck(RPC::JsonContext& context) Json::Value& jvTransactions = transactions ? (result[jss::transactions] = Json::arrayValue) : dummy; - std::string strIdent(params[jss::account].asString()); - AccountID accountID; - - if (auto jv = RPC::accountFromString(accountID, strIdent)) + auto id = parseBase58(params[jss::account].asString()); + if (!id) { - for (auto it(jv.begin()); it != jv.end(); ++it) - result[it.memberName()] = *it; - + RPC::inject_error(rpcACT_MALFORMED, result); return result; } - + auto const accountID{std::move(id.value())}; auto const sle = ledger->read(keylet::account(accountID)); if (!sle) return rpcError(rpcACT_NOT_FOUND); diff --git a/src/ripple/rpc/handlers/OwnerInfo.cpp b/src/ripple/rpc/handlers/OwnerInfo.cpp index b336107ec2..2bd9f258da 100644 --- a/src/ripple/rpc/handlers/OwnerInfo.cpp +++ b/src/ripple/rpc/handlers/OwnerInfo.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -45,21 +46,16 @@ doOwnerInfo(RPC::JsonContext& context) Json::Value ret; // Get info on account. - auto const& closedLedger = context.ledgerMaster.getClosedLedger(); - AccountID accountID; - auto jAccepted = RPC::accountFromString(accountID, strIdent); - - ret[jss::accepted] = !jAccepted - ? context.netOps.getOwnerInfo(closedLedger, accountID) - : jAccepted; + std::optional const accountID = parseBase58(strIdent); + ret[jss::accepted] = accountID.has_value() + ? context.netOps.getOwnerInfo(closedLedger, accountID.value()) + : rpcError(rpcACT_MALFORMED); auto const& currentLedger = context.ledgerMaster.getCurrentLedger(); - auto jCurrent = RPC::accountFromString(accountID, strIdent); - - ret[jss::current] = !jCurrent - ? context.netOps.getOwnerInfo(currentLedger, accountID) - : jCurrent; + ret[jss::current] = accountID.has_value() + ? context.netOps.getOwnerInfo(currentLedger, *accountID) + : rpcError(rpcACT_MALFORMED); return ret; } diff --git a/src/ripple/rpc/handlers/Tx.cpp b/src/ripple/rpc/handlers/Tx.cpp index 4574d7541d..d4bdf9a570 100644 --- a/src/ripple/rpc/handlers/Tx.cpp +++ b/src/ripple/rpc/handlers/Tx.cpp @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -330,6 +331,8 @@ populateJsonResponse( response[jss::meta] = meta->getJson(JsonOptions::none); insertDeliveredAmount( response[jss::meta], context, result.txn, *meta); + insertNFTSyntheticInJson( + response, context, result.txn->getSTransaction(), *meta); } } response[jss::validated] = result.validated; diff --git a/src/ripple/rpc/impl/NFTSyntheticSerializer.cpp b/src/ripple/rpc/impl/NFTSyntheticSerializer.cpp new file mode 100644 index 0000000000..f4692cfd4f --- /dev/null +++ b/src/ripple/rpc/impl/NFTSyntheticSerializer.cpp @@ -0,0 +1,50 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ripple { +namespace RPC { + +void +insertNFTSyntheticInJson( + Json::Value& response, + RPC::JsonContext const& context, + std::shared_ptr const& transaction, + TxMeta const& transactionMeta) +{ + insertNFTokenID(response[jss::meta], transaction, transactionMeta); + insertNFTokenOfferID(response[jss::meta], transaction, transactionMeta); +} + +} // namespace RPC +} // namespace ripple diff --git a/src/ripple/rpc/impl/NFTokenID.cpp b/src/ripple/rpc/impl/NFTokenID.cpp new file mode 100644 index 0000000000..d0be439ec6 --- /dev/null +++ b/src/ripple/rpc/impl/NFTokenID.cpp @@ -0,0 +1,202 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ripple { +namespace RPC { + +bool +canHaveNFTokenID( + std::shared_ptr const& serializedTx, + TxMeta const& transactionMeta) +{ + if (!serializedTx) + return false; + + TxType const tt = serializedTx->getTxnType(); + if (tt != ttNFTOKEN_MINT && tt != ttNFTOKEN_ACCEPT_OFFER && + tt != ttNFTOKEN_CANCEL_OFFER) + return false; + + // if the transaction failed nothing could have been delivered. + if (transactionMeta.getResultTER() != tesSUCCESS) + return false; + + return true; +} + +std::optional +getNFTokenIDFromPage(TxMeta const& transactionMeta) +{ + // The metadata does not make it obvious which NFT was added. To figure + // that out we gather up all of the previous NFT IDs and all of the final + // NFT IDs and compare them to find what changed. + std::vector prevIDs; + std::vector finalIDs; + + for (STObject const& node : transactionMeta.getNodes()) + { + if (node.getFieldU16(sfLedgerEntryType) != ltNFTOKEN_PAGE) + continue; + + SField const& fName = node.getFName(); + if (fName == sfCreatedNode) + { + STArray const& toAddPrevNFTs = node.peekAtField(sfNewFields) + .downcast() + .getFieldArray(sfNFTokens); + std::transform( + toAddPrevNFTs.begin(), + toAddPrevNFTs.end(), + std::back_inserter(finalIDs), + [](STObject const& nft) { + return nft.getFieldH256(sfNFTokenID); + }); + } + else if (fName == sfModifiedNode) + { + // When a mint results in splitting an existing page, + // it results in a created page and a modified node. Sometimes, + // the created node needs to be linked to a third page, resulting + // in modifying that third page's PreviousPageMin or NextPageMin + // field changing, but no NFTs within that page changing. In this + // case, there will be no previous NFTs and we need to skip. + // However, there will always be NFTs listed in the final fields, + // as rippled outputs all fields in final fields even if they were + // not changed. + STObject const& previousFields = + node.peekAtField(sfPreviousFields).downcast(); + if (!previousFields.isFieldPresent(sfNFTokens)) + continue; + + STArray const& toAddPrevNFTs = + previousFields.getFieldArray(sfNFTokens); + std::transform( + toAddPrevNFTs.begin(), + toAddPrevNFTs.end(), + std::back_inserter(prevIDs), + [](STObject const& nft) { + return nft.getFieldH256(sfNFTokenID); + }); + + STArray const& toAddFinalNFTs = node.peekAtField(sfFinalFields) + .downcast() + .getFieldArray(sfNFTokens); + std::transform( + toAddFinalNFTs.begin(), + toAddFinalNFTs.end(), + std::back_inserter(finalIDs), + [](STObject const& nft) { + return nft.getFieldH256(sfNFTokenID); + }); + } + } + + // We expect NFTs to be added one at a time. So finalIDs should be one + // longer than prevIDs. If that's not the case something is messed up. + if (finalIDs.size() != prevIDs.size() + 1) + return std::nullopt; + + // Find the first NFT ID that doesn't match. We're looking for an + // added NFT, so the one we want will be the mismatch in finalIDs. + auto const diff = std::mismatch( + finalIDs.begin(), finalIDs.end(), prevIDs.begin(), prevIDs.end()); + + // There should always be a difference so the returned finalIDs + // iterator should never be end(). But better safe than sorry. + if (diff.first == finalIDs.end()) + return std::nullopt; + + return *diff.first; +} + +std::vector +getNFTokenIDFromDeletedOffer(TxMeta const& transactionMeta) +{ + std::vector tokenIDResult; + for (STObject const& node : transactionMeta.getNodes()) + { + if (node.getFieldU16(sfLedgerEntryType) != ltNFTOKEN_OFFER || + node.getFName() != sfDeletedNode) + continue; + + auto const& toAddNFT = node.peekAtField(sfFinalFields) + .downcast() + .getFieldH256(sfNFTokenID); + tokenIDResult.push_back(toAddNFT); + } + + // Deduplicate the NFT IDs because multiple offers could affect the same NFT + // and hence we would get duplicate NFT IDs + sort(tokenIDResult.begin(), tokenIDResult.end()); + tokenIDResult.erase( + unique(tokenIDResult.begin(), tokenIDResult.end()), + tokenIDResult.end()); + return tokenIDResult; +} + +void +insertNFTokenID( + Json::Value& response, + std::shared_ptr const& transaction, + TxMeta const& transactionMeta) +{ + if (!canHaveNFTokenID(transaction, transactionMeta)) + return; + + // We extract the NFTokenID from metadata by comparing affected nodes + if (auto const type = transaction->getTxnType(); type == ttNFTOKEN_MINT) + { + std::optional result = getNFTokenIDFromPage(transactionMeta); + if (result.has_value()) + response[jss::nftoken_id] = to_string(result.value()); + } + else if (type == ttNFTOKEN_ACCEPT_OFFER) + { + std::vector result = + getNFTokenIDFromDeletedOffer(transactionMeta); + + if (result.size() > 0) + response[jss::nftoken_id] = to_string(result.front()); + } + else if (type == ttNFTOKEN_CANCEL_OFFER) + { + std::vector result = + getNFTokenIDFromDeletedOffer(transactionMeta); + + response[jss::nftoken_ids] = Json::Value(Json::arrayValue); + for (auto const& nftID : result) + response[jss::nftoken_ids].append(to_string(nftID)); + } +} + +} // namespace RPC +} // namespace ripple diff --git a/src/ripple/rpc/impl/NFTokenOfferID.cpp b/src/ripple/rpc/impl/NFTokenOfferID.cpp new file mode 100644 index 0000000000..05d110aac5 --- /dev/null +++ b/src/ripple/rpc/impl/NFTokenOfferID.cpp @@ -0,0 +1,85 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2023 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ripple { +namespace RPC { + +bool +canHaveNFTokenOfferID( + std::shared_ptr const& serializedTx, + TxMeta const& transactionMeta) +{ + if (!serializedTx) + return false; + + TxType const tt = serializedTx->getTxnType(); + if (tt != ttNFTOKEN_CREATE_OFFER) + return false; + + // if the transaction failed nothing could have been delivered. + if (transactionMeta.getResultTER() != tesSUCCESS) + return false; + + return true; +} + +std::optional +getOfferIDFromCreatedOffer(TxMeta const& transactionMeta) +{ + for (STObject const& node : transactionMeta.getNodes()) + { + if (node.getFieldU16(sfLedgerEntryType) != ltNFTOKEN_OFFER || + node.getFName() != sfCreatedNode) + continue; + + return node.getFieldH256(sfLedgerIndex); + } + return std::nullopt; +} + +void +insertNFTokenOfferID( + Json::Value& response, + std::shared_ptr const& transaction, + TxMeta const& transactionMeta) +{ + if (!canHaveNFTokenOfferID(transaction, transactionMeta)) + return; + + std::optional result = getOfferIDFromCreatedOffer(transactionMeta); + + if (result.has_value()) + response[jss::offer_id] = to_string(result.value()); +} + +} // namespace RPC +} // namespace ripple diff --git a/src/ripple/shamap/SHAMap.h b/src/ripple/shamap/SHAMap.h index 2f0a677f97..2d1aa192fc 100644 --- a/src/ripple/shamap/SHAMap.h +++ b/src/ripple/shamap/SHAMap.h @@ -120,14 +120,18 @@ class SHAMap static inline constexpr unsigned int leafDepth = 64; using DeltaItem = std::pair< - std::shared_ptr, - std::shared_ptr>; + boost::intrusive_ptr, + boost::intrusive_ptr>; using Delta = std::map; + SHAMap() = delete; SHAMap(SHAMap const&) = delete; SHAMap& operator=(SHAMap const&) = delete; + // Take a snapshot of the given map: + SHAMap(SHAMap const& other, bool isMutable); + // build new map SHAMap(SHAMapType t, Family& f); @@ -190,23 +194,27 @@ class SHAMap delItem(uint256 const& id); bool - addItem(SHAMapNodeType type, SHAMapItem&& i); + addItem(SHAMapNodeType type, boost::intrusive_ptr item); SHAMapHash getHash() const; // save a copy if you have a temporary anyway bool - updateGiveItem(SHAMapNodeType type, std::shared_ptr); + updateGiveItem( + SHAMapNodeType type, + boost::intrusive_ptr item); bool - addGiveItem(SHAMapNodeType type, std::shared_ptr item); + addGiveItem( + SHAMapNodeType type, + boost::intrusive_ptr item); // Save a copy if you need to extend the life // of the SHAMapItem beyond this SHAMap - std::shared_ptr const& + boost::intrusive_ptr const& peekItem(uint256 const& id) const; - std::shared_ptr const& + boost::intrusive_ptr const& peekItem(uint256 const& id, SHAMapHash& hash) const; // traverse functions @@ -253,8 +261,8 @@ class SHAMap */ void visitLeaves( - std::function const&)> const&) - const; + std::function< + void(boost::intrusive_ptr const&)> const&) const; // comparison/sync functions @@ -361,8 +369,8 @@ class SHAMap using SharedPtrNodeStack = std::stack, SHAMapNodeID>>; using DeltaRef = std::pair< - std::shared_ptr const&, - std::shared_ptr const&>; + boost::intrusive_ptr, + boost::intrusive_ptr>; // tree node cache operations std::shared_ptr @@ -475,7 +483,7 @@ class SHAMap descendNoStore(std::shared_ptr const&, int branch) const; /** If there is only one leaf below this node, get its contents */ - std::shared_ptr const& + boost::intrusive_ptr const& onlyBelow(SHAMapTreeNode*) const; bool @@ -490,7 +498,7 @@ class SHAMap bool walkBranch( SHAMapTreeNode* node, - std::shared_ptr const& otherMapItem, + boost::intrusive_ptr const& otherMapItem, bool isFirstMap, Delta& differences, int& maxCount) const; diff --git a/src/ripple/shamap/SHAMapAccountStateLeafNode.h b/src/ripple/shamap/SHAMapAccountStateLeafNode.h index 8daaf24953..45f0c50807 100644 --- a/src/ripple/shamap/SHAMapAccountStateLeafNode.h +++ b/src/ripple/shamap/SHAMapAccountStateLeafNode.h @@ -36,7 +36,7 @@ class SHAMapAccountStateLeafNode final { public: SHAMapAccountStateLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid) : SHAMapLeafNode(std::move(item), cowid) { @@ -44,7 +44,7 @@ class SHAMapAccountStateLeafNode final } SHAMapAccountStateLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid, SHAMapHash const& hash) : SHAMapLeafNode(std::move(item), cowid, hash) diff --git a/src/ripple/shamap/SHAMapItem.h b/src/ripple/shamap/SHAMapItem.h index 24e6c08c2d..160cc3cb49 100644 --- a/src/ripple/shamap/SHAMapItem.h +++ b/src/ripple/shamap/SHAMapItem.h @@ -20,52 +20,170 @@ #ifndef RIPPLE_SHAMAP_SHAMAPITEM_H_INCLUDED #define RIPPLE_SHAMAP_SHAMAPITEM_H_INCLUDED -#include +#include #include +#include #include #include +#include +#include namespace ripple { // an item stored in a SHAMap class SHAMapItem : public CountedObject { + // These are used to support boost::intrusive_ptr reference counting + // These functions are used internally by boost::intrusive_ptr to handle + // lifetime management. + friend void + intrusive_ptr_add_ref(SHAMapItem const* x); + + friend void + intrusive_ptr_release(SHAMapItem const* x); + + // This is the interface for creating new instances of this class. + friend boost::intrusive_ptr + make_shamapitem(uint256 const& tag, Slice data); + private: - uint256 tag_; - Buffer data_; + uint256 const tag_; -public: - SHAMapItem() = delete; + // We use std::uint32_t to minimize the size; there's no SHAMapItem whose + // size exceeds 4GB and there won't ever be (famous last words?), so this + // is safe. + std::uint32_t const size_; + + // This is the reference count used to support boost::intrusive_ptr + mutable std::atomic refcount_ = 1; - SHAMapItem(uint256 const& tag, Slice data) : tag_(tag), data_(data) + // Because of the unusual way in which SHAMapItem objects are constructed + // the only way to properly create one is to first allocate enough memory + // so we limit this constructor to codepaths that do this right and limit + // arbitrary construction. + SHAMapItem(uint256 const& tag, Slice data) + : tag_(tag), size_(static_cast(data.size())) { + std::memcpy( + reinterpret_cast(this) + sizeof(*this), + data.data(), + data.size()); } +public: + SHAMapItem() = delete; + + SHAMapItem(SHAMapItem const& other) = delete; + + SHAMapItem& + operator=(SHAMapItem const& other) = delete; + + SHAMapItem(SHAMapItem&& other) = delete; + + SHAMapItem& + operator=(SHAMapItem&&) = delete; + uint256 const& key() const { return tag_; } - Slice - slice() const - { - return static_cast(data_); - } - std::size_t size() const { - return data_.size(); + return size_; } void const* data() const { - return data_.data(); + return reinterpret_cast(this) + sizeof(*this); + } + + Slice + slice() const + { + return {data(), size()}; } }; +namespace detail { + +// clang-format off +// The slab cutoffs and the number of megabytes per allocation are customized +// based on the number of objects of each size we expect to need at any point +// in time and with an eye to minimize the number of slack bytes in a block. +inline SlabAllocatorSet slabber({ + { 128, megabytes(std::size_t(60)) }, + { 192, megabytes(std::size_t(46)) }, + { 272, megabytes(std::size_t(60)) }, + { 384, megabytes(std::size_t(56)) }, + { 564, megabytes(std::size_t(40)) }, + { 772, megabytes(std::size_t(46)) }, + { 1052, megabytes(std::size_t(60)) }, +}); +// clang-format on + +} // namespace detail + +inline void +intrusive_ptr_add_ref(SHAMapItem const* x) +{ + // This can only happen if someone releases the last reference to the + // item while we were trying to increment the refcount. + if (x->refcount_++ == 0) + LogicError("SHAMapItem: the reference count is 0!"); +} + +inline void +intrusive_ptr_release(SHAMapItem const* x) +{ + if (--x->refcount_ == 0) + { + auto p = reinterpret_cast(x); + + // The SHAMapItem constuctor isn't trivial (because the destructor + // for CountedObject isn't) so we can't avoid calling it here, but + // plan for a future where we might not need to. + if constexpr (!std::is_trivially_destructible_v) + std::destroy_at(x); + + // If the slabber doens't claim this pointer, it was allocated + // manually, so we free it manually. + if (!detail::slabber.deallocate(const_cast(p))) + delete[] p; + } +} + +inline boost::intrusive_ptr +make_shamapitem(uint256 const& tag, Slice data) +{ + assert(data.size() <= megabytes(16)); + + std::uint8_t* raw = detail::slabber.allocate(data.size()); + + // If we can't grab memory from the slab allocators, we fall back to + // the standard library and try to grab a precisely-sized memory block: + if (raw == nullptr) + raw = new std::uint8_t[sizeof(SHAMapItem) + data.size()]; + + // We do not increment the reference count here on purpose: the + // constructor of SHAMapItem explicitly sets it to 1. We use the fact + // that the refcount can never be zero before incrementing as an + // invariant. + return {new (raw) SHAMapItem{tag, data}, false}; +} + +static_assert(alignof(SHAMapItem) != 40); +static_assert(alignof(SHAMapItem) == 8 || alignof(SHAMapItem) == 4); + +inline boost::intrusive_ptr +make_shamapitem(SHAMapItem const& other) +{ + return make_shamapitem(other.key(), other.slice()); +} + } // namespace ripple #endif diff --git a/src/ripple/shamap/SHAMapLeafNode.h b/src/ripple/shamap/SHAMapLeafNode.h index 776aca76db..f24d7053cb 100644 --- a/src/ripple/shamap/SHAMapLeafNode.h +++ b/src/ripple/shamap/SHAMapLeafNode.h @@ -32,11 +32,14 @@ namespace ripple { class SHAMapLeafNode : public SHAMapTreeNode { protected: - std::shared_ptr item_; + boost::intrusive_ptr item_; - SHAMapLeafNode(std::shared_ptr item, std::uint32_t cowid); SHAMapLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, + std::uint32_t cowid); + + SHAMapLeafNode( + boost::intrusive_ptr item, std::uint32_t cowid, SHAMapHash const& hash); @@ -61,7 +64,7 @@ class SHAMapLeafNode : public SHAMapTreeNode invariants(bool is_root = false) const final override; public: - std::shared_ptr const& + boost::intrusive_ptr const& peekItem() const; /** Set the item that this node points to and update the node's hash. @@ -71,7 +74,7 @@ class SHAMapLeafNode : public SHAMapTreeNode hash was unchanged); true otherwise. */ bool - setItem(std::shared_ptr i); + setItem(boost::intrusive_ptr i); std::string getString(SHAMapNodeID const&) const final override; diff --git a/src/ripple/shamap/SHAMapTxLeafNode.h b/src/ripple/shamap/SHAMapTxLeafNode.h index d9a2e01e9a..e794a1a8f3 100644 --- a/src/ripple/shamap/SHAMapTxLeafNode.h +++ b/src/ripple/shamap/SHAMapTxLeafNode.h @@ -35,7 +35,7 @@ class SHAMapTxLeafNode final : public SHAMapLeafNode, { public: SHAMapTxLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid) : SHAMapLeafNode(std::move(item), cowid) { @@ -43,7 +43,7 @@ class SHAMapTxLeafNode final : public SHAMapLeafNode, } SHAMapTxLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid, SHAMapHash const& hash) : SHAMapLeafNode(std::move(item), cowid, hash) diff --git a/src/ripple/shamap/SHAMapTxPlusMetaLeafNode.h b/src/ripple/shamap/SHAMapTxPlusMetaLeafNode.h index be2c239039..ff32c64e09 100644 --- a/src/ripple/shamap/SHAMapTxPlusMetaLeafNode.h +++ b/src/ripple/shamap/SHAMapTxPlusMetaLeafNode.h @@ -36,7 +36,7 @@ class SHAMapTxPlusMetaLeafNode final { public: SHAMapTxPlusMetaLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid) : SHAMapLeafNode(std::move(item), cowid) { @@ -44,7 +44,7 @@ class SHAMapTxPlusMetaLeafNode final } SHAMapTxPlusMetaLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid, SHAMapHash const& hash) : SHAMapLeafNode(std::move(item), cowid, hash) diff --git a/src/ripple/shamap/impl/SHAMap.cpp b/src/ripple/shamap/impl/SHAMap.cpp index 7adee6f13c..1aab436c84 100644 --- a/src/ripple/shamap/impl/SHAMap.cpp +++ b/src/ripple/shamap/impl/SHAMap.cpp @@ -30,7 +30,7 @@ namespace ripple { [[nodiscard]] std::shared_ptr makeTypedLeaf( SHAMapNodeType type, - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t owner) { if (type == SHAMapNodeType::tnTRANSACTION_NM) @@ -66,28 +66,28 @@ SHAMap::SHAMap(SHAMapType t, uint256 const& hash, Family& f) root_ = std::make_shared(cowid_); } -std::shared_ptr -SHAMap::snapShot(bool isMutable) const +SHAMap::SHAMap(SHAMap const& other, bool isMutable) + : f_(other.f_) + , journal_(other.f_.journal()) + , cowid_(other.cowid_ + 1) + , ledgerSeq_(other.ledgerSeq_) + , root_(other.root_) + , state_(isMutable ? SHAMapState::Modifying : SHAMapState::Immutable) + , type_(other.type_) + , backed_(other.backed_) { - auto ret = std::make_shared(type_, f_); - SHAMap& newMap = *ret; - - if (!isMutable) - newMap.state_ = SHAMapState::Immutable; - - newMap.cowid_ = cowid_ + 1; - newMap.ledgerSeq_ = ledgerSeq_; - newMap.root_ = root_; - newMap.backed_ = backed_; - + // If either map may change, they cannot share nodes if ((state_ != SHAMapState::Immutable) || - (newMap.state_ != SHAMapState::Immutable)) + (other.state_ != SHAMapState::Immutable)) { - // If either map may change, they cannot share nodes - newMap.unshare(); + unshare(); } +} - return ret; +std::shared_ptr +SHAMap::snapShot(bool isMutable) const +{ + return std::make_shared(*this, isMutable); } void @@ -174,7 +174,6 @@ SHAMap::finishFetch( { assert(backed_); - std::shared_ptr node; try { if (!object) @@ -187,27 +186,23 @@ SHAMap::finishFetch( return {}; } - node = + auto node = SHAMapTreeNode::makeFromPrefix(makeSlice(object->getData()), hash); if (node) canonicalize(hash, node); return node; } - - catch (SHAMapMissingNode const& e) - { - JLOG(journal_.warn()) << "Missing node: " << hash << " : " << e.what(); - } catch (std::runtime_error const& e) { - JLOG(journal_.warn()) << __func__ << " : " << e.what(); + JLOG(journal_.warn()) << "finishFetch exception: " << e.what(); } catch (...) { - JLOG(journal_.warn()) << "Invalid DB node " << hash; + JLOG(journal_.warn()) + << "finishFetch exception: unknonw exception: " << hash; } - return std::shared_ptr(); + return {}; } // See if a sync filter has a node @@ -556,9 +551,9 @@ SHAMap::firstBelow( return belowHelper(node, stack, branch, {init, cmp, incr}); } -static const std::shared_ptr no_item; +static const boost::intrusive_ptr no_item; -std::shared_ptr const& +boost::intrusive_ptr const& SHAMap::onlyBelow(SHAMapTreeNode* node) const { // If there is only one item below this node, return it @@ -637,7 +632,7 @@ SHAMap::peekNextItem(uint256 const& id, SharedPtrNodeStack& stack) const return nullptr; } -std::shared_ptr const& +boost::intrusive_ptr const& SHAMap::peekItem(uint256 const& id) const { SHAMapLeafNode* leaf = findKey(id); @@ -648,7 +643,7 @@ SHAMap::peekItem(uint256 const& id) const return leaf->peekItem(); } -std::shared_ptr const& +boost::intrusive_ptr const& SHAMap::peekItem(uint256 const& id, SHAMapHash& hash) const { SHAMapLeafNode* leaf = findKey(id); @@ -820,7 +815,9 @@ SHAMap::delItem(uint256 const& id) } bool -SHAMap::addGiveItem(SHAMapNodeType type, std::shared_ptr item) +SHAMap::addGiveItem( + SHAMapNodeType type, + boost::intrusive_ptr item) { assert(state_ != SHAMapState::Immutable); assert(type != SHAMapNodeType::tnINNER); @@ -857,7 +854,7 @@ SHAMap::addGiveItem(SHAMapNodeType type, std::shared_ptr item) // this is a leaf node that has to be made an inner node holding two // items auto leaf = std::static_pointer_cast(node); - std::shared_ptr otherItem = leaf->peekItem(); + auto otherItem = leaf->peekItem(); assert(otherItem && (tag != otherItem->key())); node = std::make_shared(node->cowid()); @@ -888,9 +885,11 @@ SHAMap::addGiveItem(SHAMapNodeType type, std::shared_ptr item) } bool -SHAMap::addItem(SHAMapNodeType type, SHAMapItem&& i) +SHAMap::addItem( + SHAMapNodeType type, + boost::intrusive_ptr item) { - return addGiveItem(type, std::make_shared(std::move(i))); + return addGiveItem(type, std::move(item)); } SHAMapHash @@ -908,7 +907,7 @@ SHAMap::getHash() const bool SHAMap::updateGiveItem( SHAMapNodeType type, - std::shared_ptr item) + boost::intrusive_ptr item) { // can't change the tag but can change the hash uint256 tag = item->key(); @@ -933,13 +932,13 @@ SHAMap::updateGiveItem( if (node->getType() != type) { - JLOG(journal_.fatal()) << "SHAMap::setItem: cross-type change!"; + JLOG(journal_.fatal()) << "SHAMap::updateGiveItem: cross-type change!"; return false; } node = unshareNode(std::move(node), nodeID); - if (node->setItem(std::move(item))) + if (node->setItem(item)) dirtyUp(stack, tag, node); return true; diff --git a/src/ripple/shamap/impl/SHAMapDelta.cpp b/src/ripple/shamap/impl/SHAMapDelta.cpp index 896678ef93..ab9e329eb3 100644 --- a/src/ripple/shamap/impl/SHAMapDelta.cpp +++ b/src/ripple/shamap/impl/SHAMapDelta.cpp @@ -37,7 +37,7 @@ namespace ripple { bool SHAMap::walkBranch( SHAMapTreeNode* node, - std::shared_ptr const& otherMapItem, + boost::intrusive_ptr const& otherMapItem, bool isFirstMap, Delta& differences, int& maxCount) const @@ -71,13 +71,11 @@ SHAMap::walkBranch( { // unmatched if (isFirstMap) - differences.insert(std::make_pair( - item->key(), - DeltaRef(item, std::shared_ptr()))); + differences.insert( + std::make_pair(item->key(), DeltaRef(item, nullptr))); else - differences.insert(std::make_pair( - item->key(), - DeltaRef(std::shared_ptr(), item))); + differences.insert( + std::make_pair(item->key(), DeltaRef(nullptr, item))); if (--maxCount <= 0) return false; @@ -110,12 +108,10 @@ SHAMap::walkBranch( // otherMapItem was unmatched, must add if (isFirstMap) // this is first map, so other item is from second differences.insert(std::make_pair( - otherMapItem->key(), - DeltaRef(std::shared_ptr(), otherMapItem))); + otherMapItem->key(), DeltaRef(nullptr, otherMapItem))); else differences.insert(std::make_pair( - otherMapItem->key(), - DeltaRef(otherMapItem, std::shared_ptr()))); + otherMapItem->key(), DeltaRef(otherMapItem, nullptr))); if (--maxCount <= 0) return false; @@ -173,17 +169,13 @@ SHAMap::compare(SHAMap const& otherMap, Delta& differences, int maxCount) const { differences.insert(std::make_pair( ours->peekItem()->key(), - DeltaRef( - ours->peekItem(), - std::shared_ptr()))); + DeltaRef(ours->peekItem(), nullptr))); if (--maxCount <= 0) return false; differences.insert(std::make_pair( other->peekItem()->key(), - DeltaRef( - std::shared_ptr(), - other->peekItem()))); + DeltaRef(nullptr, other->peekItem()))); if (--maxCount <= 0) return false; } @@ -216,11 +208,7 @@ SHAMap::compare(SHAMap const& otherMap, Delta& differences, int maxCount) const // We have a branch, the other tree does not SHAMapTreeNode* iNode = descendThrow(ours, i); if (!walkBranch( - iNode, - std::shared_ptr(), - true, - differences, - maxCount)) + iNode, nullptr, true, differences, maxCount)) return false; } else if (ours->isEmptyBranch(i)) @@ -228,11 +216,7 @@ SHAMap::compare(SHAMap const& otherMap, Delta& differences, int maxCount) const // The other tree has a branch, we do not SHAMapTreeNode* iNode = otherMap.descendThrow(other, i); if (!otherMap.walkBranch( - iNode, - std::shared_ptr(), - false, - differences, - maxCount)) + iNode, nullptr, false, differences, maxCount)) return false; } else // The two trees have different non-empty branches diff --git a/src/ripple/shamap/impl/SHAMapInnerNode.cpp b/src/ripple/shamap/impl/SHAMapInnerNode.cpp index 6ea6f47eb3..1cac616b00 100644 --- a/src/ripple/shamap/impl/SHAMapInnerNode.cpp +++ b/src/ripple/shamap/impl/SHAMapInnerNode.cpp @@ -398,7 +398,7 @@ SHAMapInnerNode::canonicalizeChild( void SHAMapInnerNode::invariants(bool is_root) const { - unsigned count = 0; + [[maybe_unused]] unsigned count = 0; auto [numAllocated, hashes, children] = hashesAndChildren_.getHashesAndChildren(); diff --git a/src/ripple/shamap/impl/SHAMapLeafNode.cpp b/src/ripple/shamap/impl/SHAMapLeafNode.cpp index 1f1f3c7ff3..8f634cfad8 100644 --- a/src/ripple/shamap/impl/SHAMapLeafNode.cpp +++ b/src/ripple/shamap/impl/SHAMapLeafNode.cpp @@ -24,7 +24,7 @@ namespace ripple { SHAMapLeafNode::SHAMapLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid) : SHAMapTreeNode(cowid), item_(std::move(item)) { @@ -32,7 +32,7 @@ SHAMapLeafNode::SHAMapLeafNode( } SHAMapLeafNode::SHAMapLeafNode( - std::shared_ptr item, + boost::intrusive_ptr item, std::uint32_t cowid, SHAMapHash const& hash) : SHAMapTreeNode(cowid, hash), item_(std::move(item)) @@ -40,17 +40,17 @@ SHAMapLeafNode::SHAMapLeafNode( assert(item_->size() >= 12); } -std::shared_ptr const& +boost::intrusive_ptr const& SHAMapLeafNode::peekItem() const { return item_; } bool -SHAMapLeafNode::setItem(std::shared_ptr i) +SHAMapLeafNode::setItem(boost::intrusive_ptr item) { assert(cowid_ != 0); - item_ = std::move(i); + item_ = std::move(item); auto const oldHash = hash_; diff --git a/src/ripple/shamap/impl/SHAMapSync.cpp b/src/ripple/shamap/impl/SHAMapSync.cpp index 1bada85133..3f24047eb7 100644 --- a/src/ripple/shamap/impl/SHAMapSync.cpp +++ b/src/ripple/shamap/impl/SHAMapSync.cpp @@ -25,8 +25,8 @@ namespace ripple { void SHAMap::visitLeaves( - std::function const& item)> const& - leafFunction) const + std::function const& + item)> const& leafFunction) const { visitNodes([&leafFunction](SHAMapTreeNode& node) { if (!node.isInner()) diff --git a/src/ripple/shamap/impl/SHAMapTreeNode.cpp b/src/ripple/shamap/impl/SHAMapTreeNode.cpp index 480a560a2a..e7645a16a4 100644 --- a/src/ripple/shamap/impl/SHAMapTreeNode.cpp +++ b/src/ripple/shamap/impl/SHAMapTreeNode.cpp @@ -42,8 +42,8 @@ SHAMapTreeNode::makeTransaction( SHAMapHash const& hash, bool hashValid) { - auto item = std::make_shared( - sha512Half(HashPrefix::transactionID, data), data); + auto item = + make_shamapitem(sha512Half(HashPrefix::transactionID, data), data); if (hashValid) return std::make_shared(std::move(item), 0, hash); @@ -71,7 +71,7 @@ SHAMapTreeNode::makeTransactionWithMeta( s.chop(tag.bytes); - auto item = std::make_shared(tag, s.slice()); + auto item = make_shamapitem(tag, s.slice()); if (hashValid) return std::make_shared( @@ -103,7 +103,7 @@ SHAMapTreeNode::makeAccountState( if (tag.isZero()) Throw("Invalid AS node"); - auto item = std::make_shared(tag, s.slice()); + auto item = make_shamapitem(tag, s.slice()); if (hashValid) return std::make_shared( diff --git a/src/test/app/LedgerReplay_test.cpp b/src/test/app/LedgerReplay_test.cpp index ab10ed5be3..f1558d5db3 100644 --- a/src/test/app/LedgerReplay_test.cpp +++ b/src/test/app/LedgerReplay_test.cpp @@ -1306,8 +1306,8 @@ struct LedgerReplayer_test : public beast::unit_test::suite std::uint8_t payload[55] = { 0x6A, 0x09, 0xE6, 0x67, 0xF3, 0xBC, 0xC9, 0x08, 0xB2}; - auto item = std::make_shared( - uint256(12345), Slice(payload, sizeof(payload))); + auto item = + make_shamapitem(uint256(12345), Slice(payload, sizeof(payload))); skipList->processData(l->seq(), item); std::vector deltaStatuses; diff --git a/src/test/app/NFToken_test.cpp b/src/test/app/NFToken_test.cpp index 6669d9f5df..63f621a7c8 100644 --- a/src/test/app/NFToken_test.cpp +++ b/src/test/app/NFToken_test.cpp @@ -6574,6 +6574,238 @@ class NFToken_test : public beast::unit_test::suite } } + void + testTxJsonMetaFields(FeatureBitset features) + { + // `nftoken_id` is added in the `tx` response for NFTokenMint and + // NFTokenAcceptOffer. + // + // `nftoken_ids` is added in the `tx` response for NFTokenCancelOffer + // + // `offer_id` is added in the `tx` response for NFTokenCreateOffer + // + // The values of these fields are dependent on the NFTokenID/OfferID + // changed in its corresponding transaction. We want to validate each + // transaction to make sure the synethic fields hold the right values. + + testcase("Test synthetic fields from JSON response"); + + using namespace test::jtx; + + Account const alice{"alice"}; + Account const bob{"bob"}; + Account const broker{"broker"}; + + Env env{*this, features}; + env.fund(XRP(10000), alice, bob, broker); + env.close(); + + // Verify `nftoken_id` value equals to the NFTokenID that was + // changed in the most recent NFTokenMint or NFTokenAcceptOffer + // transaction + auto verifyNFTokenID = [&](uint256 const& actualNftID) { + // Get the hash for the most recent transaction. + std::string const txHash{ + env.tx()->getJson(JsonOptions::none)[jss::hash].asString()}; + + env.close(); + Json::Value const meta = + env.rpc("tx", txHash)[jss::result][jss::meta]; + + // Expect nftokens_id field + if (!BEAST_EXPECT(meta.isMember(jss::nftoken_id))) + return; + + // Check the value of NFT ID in the meta with the + // actual value + uint256 nftID; + BEAST_EXPECT(nftID.parseHex(meta[jss::nftoken_id].asString())); + BEAST_EXPECT(nftID == actualNftID); + }; + + // Verify `nftoken_ids` value equals to the NFTokenIDs that were + // changed in the most recent NFTokenCancelOffer transaction + auto verifyNFTokenIDsInCancelOffer = + [&](std::vector actualNftIDs) { + // Get the hash for the most recent transaction. + std::string const txHash{ + env.tx()->getJson(JsonOptions::none)[jss::hash].asString()}; + + env.close(); + Json::Value const meta = + env.rpc("tx", txHash)[jss::result][jss::meta]; + + // Expect nftokens_ids field and verify the values + if (!BEAST_EXPECT(meta.isMember(jss::nftoken_ids))) + return; + + // Convert NFT IDs from Json::Value to uint256 + std::vector metaIDs; + std::transform( + meta[jss::nftoken_ids].begin(), + meta[jss::nftoken_ids].end(), + std::back_inserter(metaIDs), + [this](Json::Value id) { + uint256 nftID; + BEAST_EXPECT(nftID.parseHex(id.asString())); + return nftID; + }); + + // Sort both array to prepare for comparison + std::sort(metaIDs.begin(), metaIDs.end()); + std::sort(actualNftIDs.begin(), actualNftIDs.end()); + + // Make sure the expect number of NFTs is correct + BEAST_EXPECT(metaIDs.size() == actualNftIDs.size()); + + // Check the value of NFT ID in the meta with the + // actual values + for (size_t i = 0; i < metaIDs.size(); ++i) + BEAST_EXPECT(metaIDs[i] == actualNftIDs[i]); + }; + + // Verify `offer_id` value equals to the offerID that was + // changed in the most recent NFTokenCreateOffer tx + auto verifyNFTokenOfferID = [&](uint256 const& offerID) { + // Get the hash for the most recent transaction. + std::string const txHash{ + env.tx()->getJson(JsonOptions::none)[jss::hash].asString()}; + + env.close(); + Json::Value const meta = + env.rpc("tx", txHash)[jss::result][jss::meta]; + + // Expect offer_id field and verify the value + if (!BEAST_EXPECT(meta.isMember(jss::offer_id))) + return; + + uint256 metaOfferID; + BEAST_EXPECT(metaOfferID.parseHex(meta[jss::offer_id].asString())); + BEAST_EXPECT(metaOfferID == offerID); + }; + + // Check new fields in tx meta when for all NFTtransactions + { + // Alice mints 2 NFTs + // Verify the NFTokenIDs are correct in the NFTokenMint tx meta + uint256 const nftId1{ + token::getNextID(env, alice, 0u, tfTransferable)}; + env(token::mint(alice, 0u), txflags(tfTransferable)); + env.close(); + verifyNFTokenID(nftId1); + + uint256 const nftId2{ + token::getNextID(env, alice, 0u, tfTransferable)}; + env(token::mint(alice, 0u), txflags(tfTransferable)); + env.close(); + verifyNFTokenID(nftId2); + + // Alice creates one sell offer for each NFT + // Verify the offer indexes are correct in the NFTokenCreateOffer tx + // meta + uint256 const aliceOfferIndex1 = + keylet::nftoffer(alice, env.seq(alice)).key; + env(token::createOffer(alice, nftId1, drops(1)), + txflags(tfSellNFToken)); + env.close(); + verifyNFTokenOfferID(aliceOfferIndex1); + + uint256 const aliceOfferIndex2 = + keylet::nftoffer(alice, env.seq(alice)).key; + env(token::createOffer(alice, nftId2, drops(1)), + txflags(tfSellNFToken)); + env.close(); + verifyNFTokenOfferID(aliceOfferIndex2); + + // Alice cancels two offers she created + // Verify the NFTokenIDs are correct in the NFTokenCancelOffer tx + // meta + env(token::cancelOffer( + alice, {aliceOfferIndex1, aliceOfferIndex2})); + env.close(); + verifyNFTokenIDsInCancelOffer({nftId1, nftId2}); + + // Bobs creates a buy offer for nftId1 + // Verify the offer id is correct in the NFTokenCreateOffer tx meta + auto const bobBuyOfferIndex = + keylet::nftoffer(bob, env.seq(bob)).key; + env(token::createOffer(bob, nftId1, drops(1)), token::owner(alice)); + env.close(); + verifyNFTokenOfferID(bobBuyOfferIndex); + + // Alice accepts bob's buy offer + // Verify the NFTokenID is correct in the NFTokenAcceptOffer tx meta + env(token::acceptBuyOffer(alice, bobBuyOfferIndex)); + env.close(); + verifyNFTokenID(nftId1); + } + + // Check `nftoken_ids` in brokered mode + { + // Alice mints a NFT + uint256 const nftId{ + token::getNextID(env, alice, 0u, tfTransferable)}; + env(token::mint(alice, 0u), txflags(tfTransferable)); + env.close(); + verifyNFTokenID(nftId); + + // Alice creates sell offer and set broker as destination + uint256 const offerAliceToBroker = + keylet::nftoffer(alice, env.seq(alice)).key; + env(token::createOffer(alice, nftId, drops(1)), + token::destination(broker), + txflags(tfSellNFToken)); + env.close(); + verifyNFTokenOfferID(offerAliceToBroker); + + // Bob creates buy offer + uint256 const offerBobToBroker = + keylet::nftoffer(bob, env.seq(bob)).key; + env(token::createOffer(bob, nftId, drops(1)), token::owner(alice)); + env.close(); + verifyNFTokenOfferID(offerBobToBroker); + + // Check NFTokenID meta for NFTokenAcceptOffer in brokered mode + env(token::brokerOffers( + broker, offerBobToBroker, offerAliceToBroker)); + env.close(); + verifyNFTokenID(nftId); + } + + // Check if there are no duplicate nft id in Cancel transactions where + // multiple offers are cancelled for the same NFT + { + // Alice mints a NFT + uint256 const nftId{ + token::getNextID(env, alice, 0u, tfTransferable)}; + env(token::mint(alice, 0u), txflags(tfTransferable)); + env.close(); + verifyNFTokenID(nftId); + + // Alice creates 2 sell offers for the same NFT + uint256 const aliceOfferIndex1 = + keylet::nftoffer(alice, env.seq(alice)).key; + env(token::createOffer(alice, nftId, drops(1)), + txflags(tfSellNFToken)); + env.close(); + verifyNFTokenOfferID(aliceOfferIndex1); + + uint256 const aliceOfferIndex2 = + keylet::nftoffer(alice, env.seq(alice)).key; + env(token::createOffer(alice, nftId, drops(1)), + txflags(tfSellNFToken)); + env.close(); + verifyNFTokenOfferID(aliceOfferIndex2); + + // Make sure the metadata only has 1 nft id, since both offers are + // for the same nft + env(token::cancelOffer( + alice, {aliceOfferIndex1, aliceOfferIndex2})); + env.close(); + verifyNFTokenIDsInCancelOffer({nftId}); + } + } + void testWithFeats(FeatureBitset features) { @@ -6606,6 +6838,7 @@ class NFToken_test : public beast::unit_test::suite testIOUWithTransferFee(features); testBrokeredSaleToSelf(features); testFixNFTokenRemint(features); + testTxJsonMetaFields(features); } public: diff --git a/src/test/app/NetworkID_test.cpp b/src/test/app/NetworkID_test.cpp index c5488b3316..51c4595daf 100644 --- a/src/test/app/NetworkID_test.cpp +++ b/src/test/app/NetworkID_test.cpp @@ -48,7 +48,9 @@ class NetworkID_test : public beast::unit_test::suite void testNetworkID() { - testcase("network_id"); + testcase( + "Require txn NetworkID to be specified (or not) depending on the " + "network ID of the node"); using namespace jtx; auto const alice = Account{"alice"}; @@ -66,9 +68,14 @@ class NetworkID_test : public beast::unit_test::suite jv[jss::Destination] = alice.human(); jv[jss::TransactionType] = "Payment"; jv[jss::Amount] = "10000000000"; + if (env.app().config().NETWORK_ID > 1024) + jv[jss::NetworkID] = + std::to_string(env.app().config().NETWORK_ID); + env(jv, fee(1000), sig(env.master)); } + // run tx env(jv, fee(1000), ter(expectedOutcome)); env.close(); }; @@ -121,9 +128,9 @@ class NetworkID_test : public beast::unit_test::suite test::jtx::Env env{*this, makeNetworkConfig(1025)}; BEAST_EXPECT(env.app().config().NETWORK_ID == 1025); + // try to submit a txn without network id, this should not work { env.fund(XRP(200), alice); - // try to submit a txn without network id, this should not work Json::Value jvn; jvn[jss::Account] = alice.human(); jvn[jss::TransactionType] = jss::AccountSet; diff --git a/src/test/app/ValidatorList_test.cpp b/src/test/app/ValidatorList_test.cpp index 379571ccd6..ff9b57f3ce 100644 --- a/src/test/app/ValidatorList_test.cpp +++ b/src/test/app/ValidatorList_test.cpp @@ -1318,8 +1318,7 @@ class ValidatorList_test : public beast::unit_test::suite BEAST_EXPECT(changes.added == expectedTrusted); BEAST_EXPECT(trustedKeys->quorum() == minQuorum); - /* - // Use normal quorum when seen validators >= quorum + // Use configured quorum even when seen validators >= quorum activeValidators.emplace(toBeSeen); changes = trustedKeys->updateTrusted( activeValidators, @@ -1329,8 +1328,7 @@ class ValidatorList_test : public beast::unit_test::suite env.app().getHashRouter()); BEAST_EXPECT(changes.removed.empty()); BEAST_EXPECT(changes.added.empty()); - BEAST_EXPECT(trustedKeys->quorum() == std::ceil(n * 0.8f)); - */ + BEAST_EXPECT(trustedKeys->quorum() == minQuorum); } { // Remove expired published list @@ -1830,8 +1828,8 @@ class ValidatorList_test : public beast::unit_test::suite env.app().getOPs(), env.app().overlay(), env.app().getHashRouter()); - if (trustedKeys->quorum() == std::ceil(cfgKeys.size() * 0.8f) || - (minimumQuorum && trustedKeys->quorum() == *minimumQuorum)) + if (minimumQuorum == trustedKeys->quorum() || + trustedKeys->quorum() == std::ceil(cfgKeys.size() * 0.8f)) return trustedKeys; } return nullptr; diff --git a/src/test/basics/FileUtilities_test.cpp b/src/test/basics/FileUtilities_test.cpp index 2a9710f8e9..b9371d214c 100644 --- a/src/test/basics/FileUtilities_test.cpp +++ b/src/test/basics/FileUtilities_test.cpp @@ -20,6 +20,7 @@ #include #include #include +#include #include namespace ripple { diff --git a/src/test/consensus/ByzantineFailureSim_test.cpp b/src/test/consensus/ByzantineFailureSim_test.cpp index 73104ab5e8..a907b7c224 100644 --- a/src/test/consensus/ByzantineFailureSim_test.cpp +++ b/src/test/consensus/ByzantineFailureSim_test.cpp @@ -78,7 +78,7 @@ class ByzantineFailureSim_test : public beast::unit_test::suite // All peers see some TX 0 for (Peer* peer : network) { - peer->submit(Tx(0)); + peer->submit(Tx{0}); // Peers 0,1,2,6 will close the next ledger differently by injecting // a non-consensus approved transaciton if (byzantineNodes.contains(peer)) diff --git a/src/test/csf/Tx.h b/src/test/csf/Tx.h index 5ccd910b80..d271338141 100644 --- a/src/test/csf/Tx.h +++ b/src/test/csf/Tx.h @@ -25,6 +25,7 @@ #include #include #include +#include namespace ripple { namespace test { @@ -40,6 +41,11 @@ class Tx { } + template >> + Tx(T const* t) : id_{t->id_} + { + } + ID id() const { diff --git a/src/test/csf/ledgers.h b/src/test/csf/ledgers.h index 635fbec211..987c5e706c 100644 --- a/src/test/csf/ledgers.h +++ b/src/test/csf/ledgers.h @@ -348,7 +348,7 @@ struct LedgerHistoryHelper assert(seen.emplace(s.back()).second); Ledger const& parent = (*this)[s.substr(0, s.size() - 1)]; - return ledgers.emplace(s, oracle.accept(parent, ++nextTx)) + return ledgers.emplace(s, oracle.accept(parent, Tx{++nextTx})) .first->second; } }; diff --git a/src/test/rpc/AccountCurrencies_test.cpp b/src/test/rpc/AccountCurrencies_test.cpp index ac4adcf516..c3e46a3e66 100644 --- a/src/test/rpc/AccountCurrencies_test.cpp +++ b/src/test/rpc/AccountCurrencies_test.cpp @@ -56,11 +56,22 @@ class AccountCurrencies_test : public beast::unit_test::suite result[jss::error_message] == "Missing field 'account'."); } - { // strict mode, invalid bitcoin token + { Json::Value params; params[jss::account] = "llIIOO"; // these are invalid in bitcoin alphabet - params[jss::strict] = true; + auto const result = env.rpc( + "json", + "account_currencies", + boost::lexical_cast(params))[jss::result]; + BEAST_EXPECT(result[jss::error] == "actMalformed"); + BEAST_EXPECT(result[jss::error_message] == "Account malformed."); + } + + { + // Cannot use a seed as account + Json::Value params; + params[jss::account] = "Bob"; auto const result = env.rpc( "json", "account_currencies", diff --git a/src/test/rpc/AccountInfo_test.cpp b/src/test/rpc/AccountInfo_test.cpp index 5c9e323285..ed6527b55f 100644 --- a/src/test/rpc/AccountInfo_test.cpp +++ b/src/test/rpc/AccountInfo_test.cpp @@ -53,7 +53,9 @@ class AccountInfo_test : public beast::unit_test::suite "{\"account\": " "\"n94JNrQYkDrpt62bbSR7nVEhdyAvcJXRAsjEkFYyqRkh9SUTYEqV\"}"); BEAST_EXPECT( - info[jss::result][jss::error_message] == "Disallowed seed."); + info[jss::result][jss::error_code] == rpcACT_MALFORMED); + BEAST_EXPECT( + info[jss::result][jss::error_message] == "Account malformed."); } { // account_info with an account that's not in the ledger. @@ -61,10 +63,21 @@ class AccountInfo_test : public beast::unit_test::suite auto const info = env.rpc( "json", "account_info", - std::string("{ ") + "\"account\": \"" + bogie.human() + "\"}"); + R"({ "account": ")" + bogie.human() + R"("})"); + BEAST_EXPECT( + info[jss::result][jss::error_code] == rpcACT_NOT_FOUND); BEAST_EXPECT( info[jss::result][jss::error_message] == "Account not found."); } + { + // Cannot use a seed as account + auto const info = + env.rpc("json", "account_info", R"({"account": "foo"})"); + BEAST_EXPECT( + info[jss::result][jss::error_code] == rpcACT_MALFORMED); + BEAST_EXPECT( + info[jss::result][jss::error_message] == "Account malformed."); + } } // Test the "signer_lists" argument in account_info. diff --git a/src/test/rpc/AccountLinesRPC_test.cpp b/src/test/rpc/AccountLinesRPC_test.cpp index 1b099f7b7b..04688156d1 100644 --- a/src/test/rpc/AccountLinesRPC_test.cpp +++ b/src/test/rpc/AccountLinesRPC_test.cpp @@ -53,7 +53,7 @@ class AccountLinesRPC_test : public beast::unit_test::suite R"("n9MJkEKHDhy5eTLuHUQeAAjo382frHNbFK4C8hcwN4nwM2SrLdBj"})"); BEAST_EXPECT( lines[jss::result][jss::error_message] == - RPC::make_error(rpcBAD_SEED)[jss::error_message]); + RPC::make_error(rpcACT_MALFORMED)[jss::error_message]); } Account const alice{"alice"}; { @@ -239,7 +239,7 @@ class AccountLinesRPC_test : public beast::unit_test::suite R"("n9MJkEKHDhy5eTLuHUQeAAjo382frHNbFK4C8hcwN4nwM2SrLdBj"})"); BEAST_EXPECT( lines[jss::result][jss::error_message] == - RPC::make_error(rpcBAD_SEED)[jss::error_message]); + RPC::make_error(rpcACT_MALFORMED)[jss::error_message]); } { // A negative limit should fail. @@ -815,7 +815,7 @@ class AccountLinesRPC_test : public beast::unit_test::suite R"("n9MJkEKHDhy5eTLuHUQeAAjo382frHNbFK4C8hcwN4nwM2SrLdBj"}})"); BEAST_EXPECT( lines[jss::error][jss::message] == - RPC::make_error(rpcBAD_SEED)[jss::error_message]); + RPC::make_error(rpcACT_MALFORMED)[jss::error_message]); BEAST_EXPECT( lines.isMember(jss::jsonrpc) && lines[jss::jsonrpc] == "2.0"); BEAST_EXPECT( @@ -1122,7 +1122,7 @@ class AccountLinesRPC_test : public beast::unit_test::suite R"("n9MJkEKHDhy5eTLuHUQeAAjo382frHNbFK4C8hcwN4nwM2SrLdBj"}})"); BEAST_EXPECT( lines[jss::error][jss::message] == - RPC::make_error(rpcBAD_SEED)[jss::error_message]); + RPC::make_error(rpcACT_MALFORMED)[jss::error_message]); BEAST_EXPECT( lines.isMember(jss::jsonrpc) && lines[jss::jsonrpc] == "2.0"); BEAST_EXPECT( diff --git a/src/test/rpc/AccountObjects_test.cpp b/src/test/rpc/AccountObjects_test.cpp index 8a784b8840..e20720ca7c 100644 --- a/src/test/rpc/AccountObjects_test.cpp +++ b/src/test/rpc/AccountObjects_test.cpp @@ -133,7 +133,7 @@ class AccountObjects_test : public beast::unit_test::suite "n94JNrQYkDrpt62bbSR7nVEhdyAvcJXRAsjEkFYyqRkh9SUTYEqV"; auto resp = env.rpc("json", "account_objects", to_string(params)); BEAST_EXPECT( - resp[jss::result][jss::error_message] == "Disallowed seed."); + resp[jss::result][jss::error_message] == "Account malformed."); } // test error on account that's not in the ledger. { diff --git a/src/test/rpc/AccountOffers_test.cpp b/src/test/rpc/AccountOffers_test.cpp index 79e0ddd728..7d944c8def 100644 --- a/src/test/rpc/AccountOffers_test.cpp +++ b/src/test/rpc/AccountOffers_test.cpp @@ -248,15 +248,15 @@ class AccountOffers_test : public beast::unit_test::suite "json", "account_offers", jvParams.toStyledString())[jss::result]; - BEAST_EXPECT(jrr[jss::error] == "badSeed"); + BEAST_EXPECT(jrr[jss::error] == "actMalformed"); BEAST_EXPECT(jrr[jss::status] == "error"); - BEAST_EXPECT(jrr[jss::error_message] == "Disallowed seed."); + BEAST_EXPECT(jrr[jss::error_message] == "Account malformed."); } { // bogus account value - auto const jrr = - env.rpc("account_offers", "rNOT_AN_ACCOUNT")[jss::result]; + auto const jrr = env.rpc( + "account_offers", Account("bogus").human())[jss::result]; BEAST_EXPECT(jrr[jss::error] == "actNotFound"); BEAST_EXPECT(jrr[jss::status] == "error"); BEAST_EXPECT(jrr[jss::error_message] == "Account not found."); diff --git a/src/test/rpc/NoRippleCheck_test.cpp b/src/test/rpc/NoRippleCheck_test.cpp index 73934899e0..3d34f55c90 100644 --- a/src/test/rpc/NoRippleCheck_test.cpp +++ b/src/test/rpc/NoRippleCheck_test.cpp @@ -127,8 +127,8 @@ class NoRippleCheck_test : public beast::unit_test::suite "json", "noripple_check", boost::lexical_cast(params))[jss::result]; - BEAST_EXPECT(result[jss::error] == "badSeed"); - BEAST_EXPECT(result[jss::error_message] == "Disallowed seed."); + BEAST_EXPECT(result[jss::error] == "actMalformed"); + BEAST_EXPECT(result[jss::error_message] == "Account malformed."); } } diff --git a/src/test/rpc/OwnerInfo_test.cpp b/src/test/rpc/OwnerInfo_test.cpp index 1dbfdb4ae9..af4b4b3bd1 100644 --- a/src/test/rpc/OwnerInfo_test.cpp +++ b/src/test/rpc/OwnerInfo_test.cpp @@ -56,14 +56,16 @@ class OwnerInfo_test : public beast::unit_test::suite result.isMember(jss::accepted) && result.isMember(jss::current))) { - BEAST_EXPECT(result[jss::accepted][jss::error] == "badSeed"); + BEAST_EXPECT( + result[jss::accepted][jss::error] == "actMalformed"); BEAST_EXPECT( result[jss::accepted][jss::error_message] == - "Disallowed seed."); - BEAST_EXPECT(result[jss::current][jss::error] == "badSeed"); + "Account malformed."); + BEAST_EXPECT( + result[jss::current][jss::error] == "actMalformed"); BEAST_EXPECT( result[jss::current][jss::error_message] == - "Disallowed seed."); + "Account malformed."); } } diff --git a/src/test/rpc/RPCCall_test.cpp b/src/test/rpc/RPCCall_test.cpp index 0d4702799d..82428779a5 100644 --- a/src/test/rpc/RPCCall_test.cpp +++ b/src/test/rpc/RPCCall_test.cpp @@ -86,7 +86,7 @@ static RPCCallTestData const rpcCallTestArray[] = { __LINE__, {"account_channels", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "FEDCBA9876543210FEDCBA9876543210FEDCBA9876543210FEDCBA9876543210"}, + "rD5MbavGfiSC5m7mkxy1FANuT7s3HxqpoF"}, RPCCallTestData::no_exception, R"({ "method" : "account_channels", @@ -94,13 +94,15 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "destination_account" : "FEDCBA9876543210FEDCBA9876543210FEDCBA9876543210FEDCBA9876543210" + "destination_account" : "rD5MbavGfiSC5m7mkxy1FANuT7s3HxqpoF" } ] })"}, {"account_channels: account and ledger index.", __LINE__, - {"account_channels", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "closed"}, + {"account_channels", + "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", + "r9emE59aTWb85t64dAebKrxYMBTpzK5yR7"}, RPCCallTestData::no_exception, R"({ "method" : "account_channels", @@ -108,7 +110,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "destination_account" : "closed" + "destination_account" : "r9emE59aTWb85t64dAebKrxYMBTpzK5yR7" } ] })"}, @@ -186,7 +188,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "rnUy2SHTrB9DubsPmkJZUXTf5FcNDGrYEA", "current", - "strict"}, + "extra"}, RPCCallTestData::no_exception, R"({ "method" : "account_channels", @@ -218,7 +220,7 @@ static RPCCallTestData const rpcCallTestArray[] = { // account_currencies // ---------------------------------------------------------- - {"account_currencies: minimal.", + {"account_currencies: minimal 1.", __LINE__, {"account_currencies", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}, RPCCallTestData::no_exception, @@ -231,17 +233,16 @@ static RPCCallTestData const rpcCallTestArray[] = { } ] })"}, - {"account_currencies: strict.", + {"account_currencies: minimal 2.", __LINE__, - {"account_currencies", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "strict"}, + {"account_currencies", "racb4o3DrdYxuCfyVa6vsLb7vgju9RFbBr"}, RPCCallTestData::no_exception, R"({ "method" : "account_currencies", "params" : [ { "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 + "account" : "racb4o3DrdYxuCfyVa6vsLb7vgju9RFbBr" } ] })"}, @@ -275,10 +276,7 @@ static RPCCallTestData const rpcCallTestArray[] = { })"}, {"account_currencies: current ledger.", __LINE__, - {"account_currencies", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "current", - "strict"}, + {"account_currencies", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "current"}, RPCCallTestData::no_exception, R"({ "method" : "account_currencies", @@ -286,8 +284,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "api_version" : %MAX_API_VER%, - "ledger_index" : "current", - "strict" : 1 + "ledger_index" : "current" } ] })"}, @@ -312,8 +309,8 @@ static RPCCallTestData const rpcCallTestArray[] = { {"account_currencies", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "current", - "strict", - "spare"}, + "spare1", + "spare2"}, RPCCallTestData::no_exception, R"({ "method" : "account_currencies", @@ -358,20 +355,6 @@ static RPCCallTestData const rpcCallTestArray[] = { ] })", }, - {"account_currencies: floating point first argument.", - __LINE__, - {"account_currencies", "3.14159", "strict"}, - RPCCallTestData::no_exception, - R"({ - "method" : "account_currencies", - "params" : [ - { - "api_version" : %MAX_API_VER%, - "account" : "3.14159", - "strict" : 1 - } - ] - })"}, // account_info // ---------------------------------------------------------------- @@ -432,26 +415,9 @@ static RPCCallTestData const rpcCallTestArray[] = { } ] })"}, - {"account_info: strict.", + {"account_info: with ledger index.", __LINE__, - {"account_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "strict"}, - RPCCallTestData::no_exception, - R"({ - "method" : "account_info", - "params" : [ - { - "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 - } - ] - })"}, - {"account_info: with ledger index and strict.", - __LINE__, - {"account_info", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "validated", - "strict"}, + {"account_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "validated"}, RPCCallTestData::no_exception, R"({ "method" : "account_info", @@ -459,8 +425,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "api_version" : %MAX_API_VER%, - "ledger_index" : "validated", - "strict" : 1 + "ledger_index" : "validated" } ] })"}, @@ -485,8 +450,8 @@ static RPCCallTestData const rpcCallTestArray[] = { {"account_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "current", - "strict", - "extra"}, + "extra1", + "extra2"}, RPCCallTestData::no_exception, R"({ "method" : "account_info", @@ -798,26 +763,9 @@ static RPCCallTestData const rpcCallTestArray[] = { } ] })"}, - {"account_objects: strict.", - __LINE__, - {"account_objects", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "strict"}, - RPCCallTestData::no_exception, - R"({ - "method" : "account_objects", - "params" : [ - { - "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 - } - ] - })"}, - {"account_objects: with ledger index and strict.", + {"account_objects: with ledger index.", __LINE__, - {"account_objects", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "validated", - "strict"}, + {"account_objects", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "validated"}, RPCCallTestData::no_exception, R"({ "method" : "account_objects", @@ -825,8 +773,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "ledger_index" : "validated", - "strict" : 1 + "ledger_index" : "validated" } ] })"}, @@ -853,8 +800,8 @@ static RPCCallTestData const rpcCallTestArray[] = { "account_objects", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "current", - "extra", - "strict", + "extra1", + "extra2", }, RPCCallTestData::no_exception, R"({ @@ -862,8 +809,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "params" : [ { "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 + "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh" } ] })"}, @@ -876,7 +822,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "current", "extra1", "extra2", - "strict", + "extra3", }, RPCCallTestData::no_exception, R"({ @@ -884,8 +830,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "params" : [ { "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 + "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh" } ] })"}, @@ -898,7 +843,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "extra1", "extra2", "extra3", - "strict", + "extra4", }, RPCCallTestData::no_exception, R"({ @@ -953,10 +898,7 @@ static RPCCallTestData const rpcCallTestArray[] = { // cannot currently occur because jvParseLedger() always returns true. "account_objects: invalid ledger selection 2.", __LINE__, - {"account_objects", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "no_ledger", - "strict"}, + {"account_objects", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "no_ledger"}, RPCCallTestData::no_exception, R"({ "method" : "account_objects", @@ -964,8 +906,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "ledger_index" : 0, - "strict" : 1 + "ledger_index" : 0 } ] })", @@ -1030,26 +971,9 @@ static RPCCallTestData const rpcCallTestArray[] = { } ] })"}, - {"account_offers: strict.", + {"account_offers: with ledger index.", __LINE__, - {"account_offers", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "strict"}, - RPCCallTestData::no_exception, - R"({ - "method" : "account_offers", - "params" : [ - { - "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 - } - ] - })"}, - {"account_offers: with ledger index and strict.", - __LINE__, - {"account_offers", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "validated", - "strict"}, + {"account_offers", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "validated"}, RPCCallTestData::no_exception, R"({ "method" : "account_offers", @@ -1057,8 +981,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "ledger_index" : "validated", - "strict" : 1 + "ledger_index" : "validated" } ] })"}, @@ -1081,21 +1004,17 @@ static RPCCallTestData const rpcCallTestArray[] = { {// Note: I believe this _ought_ to be detected as too many arguments. "account_offers: four arguments.", __LINE__, - { - "account_offers", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "current", - "extra", - "strict", - }, + {"account_offers", + "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", + "current", + "extra"}, RPCCallTestData::no_exception, R"({ "method" : "account_offers", "params" : [ { "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 + "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh" } ] })"}, @@ -1107,7 +1026,7 @@ static RPCCallTestData const rpcCallTestArray[] = { "current", "extra1", "extra2", - "strict", + "extra3", }, RPCCallTestData::no_exception, R"({ @@ -1162,10 +1081,7 @@ static RPCCallTestData const rpcCallTestArray[] = { // cannot currently occur because jvParseLedger() always returns true. "account_offers: invalid ledger selection 2.", __LINE__, - {"account_offers", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "no_ledger", - "strict"}, + {"account_offers", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "no_ledger"}, RPCCallTestData::no_exception, R"({ "method" : "account_offers", @@ -1173,8 +1089,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "api_version" : %MAX_API_VER%, "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "ledger_index" : 0, - "strict" : 1 + "ledger_index" : 0 } ] })", @@ -4475,26 +4390,9 @@ static RPCCallTestData const rpcCallTestArray[] = { } ] })"}, - {"owner_info: strict.", - __LINE__, - {"owner_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "strict"}, - RPCCallTestData::no_exception, - R"({ - "method" : "owner_info", - "params" : [ - { - "api_version" : %MAX_API_VER%, - "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "strict" : 1 - } - ] - })"}, - {"owner_info: with ledger index and strict.", + {"owner_info: with ledger index.", __LINE__, - {"owner_info", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "validated", - "strict"}, + {"owner_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "validated"}, RPCCallTestData::no_exception, R"({ "method" : "owner_info", @@ -4502,8 +4400,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "api_version" : %MAX_API_VER%, - "ledger_index" : "validated", - "strict" : 1 + "ledger_index" : "validated" } ] })"}, @@ -4529,8 +4426,8 @@ static RPCCallTestData const rpcCallTestArray[] = { "owner_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "current", - "extra", - "strict", + "extra1", + "extra2", }, RPCCallTestData::no_exception, R"({ @@ -4583,12 +4480,9 @@ static RPCCallTestData const rpcCallTestArray[] = { { // Note: there is code in place to return rpcLGR_IDX_MALFORMED. That // cannot currently occur because jvParseLedger() always returns true. - "owner_info: invalid ledger selection and strict.", + "owner_info: invalid ledger selection.", __LINE__, - {"owner_info", - "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - "no_ledger", - "strict"}, + {"owner_info", "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "no_ledger"}, RPCCallTestData::no_exception, R"({ "method" : "owner_info", @@ -4596,8 +4490,7 @@ static RPCCallTestData const rpcCallTestArray[] = { { "account" : "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "api_version" : %MAX_API_VER%, - "ledger_index" : 0, - "strict" : 1 + "ledger_index" : 0 } ] })", diff --git a/src/test/rpc/ReportingETL_test.cpp b/src/test/rpc/ReportingETL_test.cpp index 3c1d00ad96..c44678d0a8 100644 --- a/src/test/rpc/ReportingETL_test.cpp +++ b/src/test/rpc/ReportingETL_test.cpp @@ -18,7 +18,6 @@ */ //============================================================================== -#include #include #include #include diff --git a/src/test/shamap/FetchPack_test.cpp b/src/test/shamap/FetchPack_test.cpp index ae1174b4d5..f15f3163e5 100644 --- a/src/test/shamap/FetchPack_test.cpp +++ b/src/test/shamap/FetchPack_test.cpp @@ -85,13 +85,13 @@ class FetchPack_test : public beast::unit_test::suite beast::Journal mJournal; }; - std::shared_ptr + boost::intrusive_ptr make_random_item(beast::xor_shift_engine& r) { Serializer s; for (int d = 0; d < 3; ++d) s.add32(ripple::rand_int(r)); - return std::make_shared(s.getSHA512Half(), s.slice()); + return make_shamapitem(s.getSHA512Half(), s.slice()); } void @@ -99,9 +99,8 @@ class FetchPack_test : public beast::unit_test::suite { while (n--) { - std::shared_ptr item(make_random_item(r)); - auto const result( - t.addItem(SHAMapNodeType::tnACCOUNT_STATE, std::move(*item))); + auto const result(t.addItem( + SHAMapNodeType::tnACCOUNT_STATE, make_random_item(r))); assert(result); (void)result; } diff --git a/src/test/shamap/SHAMapSync_test.cpp b/src/test/shamap/SHAMapSync_test.cpp index 6b2648a96c..70e578b5fb 100644 --- a/src/test/shamap/SHAMapSync_test.cpp +++ b/src/test/shamap/SHAMapSync_test.cpp @@ -34,14 +34,14 @@ class SHAMapSync_test : public beast::unit_test::suite public: beast::xor_shift_engine eng_; - std::shared_ptr + boost::intrusive_ptr makeRandomAS() { Serializer s; for (int d = 0; d < 3; ++d) s.add32(rand_int(eng_)); - return std::make_shared(s.getSHA512Half(), s.slice()); + return make_shamapitem(s.getSHA512Half(), s.slice()); } bool @@ -55,10 +55,10 @@ class SHAMapSync_test : public beast::unit_test::suite for (int i = 0; i < count; ++i) { - std::shared_ptr item = makeRandomAS(); + auto item = makeRandomAS(); items.push_back(item->key()); - if (!map.addItem(SHAMapNodeType::tnACCOUNT_STATE, std::move(*item))) + if (!map.addItem(SHAMapNodeType::tnACCOUNT_STATE, item)) { log << "Unable to add item to map\n"; return false; @@ -97,8 +97,7 @@ class SHAMapSync_test : public beast::unit_test::suite int items = 10000; for (int i = 0; i < items; ++i) { - source.addItem( - SHAMapNodeType::tnACCOUNT_STATE, std::move(*makeRandomAS())); + source.addItem(SHAMapNodeType::tnACCOUNT_STATE, makeRandomAS()); if (i % 100 == 0) source.invariants(); } diff --git a/src/test/shamap/SHAMap_test.cpp b/src/test/shamap/SHAMap_test.cpp index 182b443ce0..83bbc13253 100644 --- a/src/test/shamap/SHAMap_test.cpp +++ b/src/test/shamap/SHAMap_test.cpp @@ -22,7 +22,6 @@ #include #include #include -#include #include #include @@ -45,10 +44,7 @@ static_assert(std::is_move_assignable{}, ""); static_assert(std::is_nothrow_destructible{}, ""); static_assert(!std::is_default_constructible{}, ""); -static_assert(std::is_copy_constructible{}, ""); -static_assert(std::is_copy_assignable{}, ""); -static_assert(std::is_move_constructible{}, ""); -static_assert(std::is_move_assignable{}, ""); +static_assert(!std::is_copy_constructible{}, ""); static_assert(std::is_nothrow_destructible{}, ""); static_assert(std::is_default_constructible{}, ""); @@ -155,37 +151,43 @@ class SHAMap_test : public beast::unit_test::suite if (!backed) sMap.setUnbacked(); - SHAMapItem i1(h1, IntToVUC(1)), i2(h2, IntToVUC(2)), - i3(h3, IntToVUC(3)), i4(h4, IntToVUC(4)), i5(h5, IntToVUC(5)); + auto i1 = make_shamapitem(h1, IntToVUC(1)); + auto i2 = make_shamapitem(h2, IntToVUC(2)); + auto i3 = make_shamapitem(h3, IntToVUC(3)); + auto i4 = make_shamapitem(h4, IntToVUC(4)); + auto i5 = make_shamapitem(h5, IntToVUC(5)); + unexpected( - !sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, SHAMapItem{i2}), + !sMap.addItem( + SHAMapNodeType::tnTRANSACTION_NM, make_shamapitem(*i2)), "no add"); sMap.invariants(); unexpected( - !sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, SHAMapItem{i1}), + !sMap.addItem( + SHAMapNodeType::tnTRANSACTION_NM, make_shamapitem(*i1)), "no add"); sMap.invariants(); auto i = sMap.begin(); auto e = sMap.end(); - unexpected(i == e || (*i != i1), "bad traverse"); + unexpected(i == e || (*i != *i1), "bad traverse"); ++i; - unexpected(i == e || (*i != i2), "bad traverse"); + unexpected(i == e || (*i != *i2), "bad traverse"); ++i; unexpected(i != e, "bad traverse"); - sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, SHAMapItem{i4}); + sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, make_shamapitem(*i4)); sMap.invariants(); - sMap.delItem(i2.key()); + sMap.delItem(i2->key()); sMap.invariants(); - sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, SHAMapItem{i3}); + sMap.addItem(SHAMapNodeType::tnTRANSACTION_NM, make_shamapitem(*i3)); sMap.invariants(); i = sMap.begin(); e = sMap.end(); - unexpected(i == e || (*i != i1), "bad traverse"); + unexpected(i == e || (*i != *i1), "bad traverse"); ++i; - unexpected(i == e || (*i != i3), "bad traverse"); + unexpected(i == e || (*i != *i3), "bad traverse"); ++i; - unexpected(i == e || (*i != i4), "bad traverse"); + unexpected(i == e || (*i != *i4), "bad traverse"); ++i; unexpected(i != e, "bad traverse"); @@ -265,9 +267,9 @@ class SHAMap_test : public beast::unit_test::suite BEAST_EXPECT(map.getHash() == beast::zero); for (int k = 0; k < keys.size(); ++k) { - SHAMapItem item(keys[k], IntToVUC(k)); BEAST_EXPECT(map.addItem( - SHAMapNodeType::tnTRANSACTION_NM, std::move(item))); + SHAMapNodeType::tnTRANSACTION_NM, + make_shamapitem(keys[k], IntToVUC(k)))); BEAST_EXPECT(map.getHash().as_uint256() == hashes[k]); map.invariants(); } @@ -312,7 +314,7 @@ class SHAMap_test : public beast::unit_test::suite { map.addItem( SHAMapNodeType::tnTRANSACTION_NM, - SHAMapItem{k, IntToVUC(0)}); + make_shamapitem(k, IntToVUC(0))); map.invariants(); } @@ -346,7 +348,7 @@ class SHAMapPathProof_test : public beast::unit_test::suite uint256 k(c); map.addItem( SHAMapNodeType::tnACCOUNT_STATE, - SHAMapItem{k, Slice{k.data(), k.size()}}); + make_shamapitem(k, Slice{k.data(), k.size()})); map.invariants(); auto root = map.getHash().as_uint256(); From 21863b05f3dbb601a8df2eba376e8fd9496f5890 Mon Sep 17 00:00:00 2001 From: RichardAH Date: Sat, 23 Nov 2024 21:19:09 +1000 Subject: [PATCH 02/17] Limit xahau genesis to networks starting with 2133X (#395) --- src/ripple/app/tx/impl/Change.cpp | 7 ++ src/test/app/XahauGenesis_test.cpp | 147 ++++++++++++++++++++++++----- 2 files changed, 128 insertions(+), 26 deletions(-) diff --git a/src/ripple/app/tx/impl/Change.cpp b/src/ripple/app/tx/impl/Change.cpp index 1c087ff660..c91b794039 100644 --- a/src/ripple/app/tx/impl/Change.cpp +++ b/src/ripple/app/tx/impl/Change.cpp @@ -458,6 +458,13 @@ Change::activateXahauGenesis() bool const isTest = (ctx_.tx.getFlags() & tfTestSuite) && ctx_.app.config().standalone(); + // RH NOTE: we'll only configure xahau governance structure on networks that + // begin with 2133... so production xahau: 21337 and its testnet 21338 + // with 21330-21336 and 21339 also valid and reserved for dev nets etc. + // all other Network IDs will be conventionally configured. + if ((ctx_.app.config().NETWORK_ID / 10) != 2133 && !isTest) + return; + auto [ng_entries, l1_entries, l2_entries, gov_params] = normalizeXahauGenesis( isTest ? TestNonGovernanceDistribution : NonGovernanceDistribution, diff --git a/src/test/app/XahauGenesis_test.cpp b/src/test/app/XahauGenesis_test.cpp index aa546a116e..d143591e06 100644 --- a/src/test/app/XahauGenesis_test.cpp +++ b/src/test/app/XahauGenesis_test.cpp @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -27,6 +28,7 @@ #include #include #include +#include #include #define BEAST_REQUIRE(x) \ @@ -59,7 +61,18 @@ maybe_to_string(T val, std::enable_if_t, int> = 0) using namespace XahauGenesis; namespace ripple { + +inline std::unique_ptr +makeNetworkConfig(uint32_t networkID) +{ + using namespace test::jtx; + return envconfig([&](std::unique_ptr cfg) { + cfg->NETWORK_ID = networkID; + return cfg; + }); +} namespace test { + /* Accounts used in this test suite: alice: AE123A8556F3CF91154711376AFB0F894F832B3D, @@ -125,7 +138,8 @@ struct XahauGenesis_test : public beast::unit_test::suite bool burnedViaTest = false, // means the calling test already burned some of the genesis bool skipTests = false, - bool const testFlag = false) + bool const testFlag = false, + bool const badNetID = false) { using namespace jtx; @@ -183,6 +197,20 @@ struct XahauGenesis_test : public beast::unit_test::suite if (skipTests) return; + if (badNetID) + { + BEAST_EXPECT( + 100000000000000000ULL == + env.app().getLedgerMaster().getClosedLedger()->info().drops); + + auto genesisAccRoot = env.le(keylet::account(genesisAccID)); + BEAST_REQUIRE(!!genesisAccRoot); + BEAST_EXPECT( + genesisAccRoot->getFieldAmount(sfBalance) == + XRPAmount(100000000000000000ULL)); + return; + } + // sum the initial distribution balances, these should equal total coins // in the closed ledger std::vector> const& l1membership = @@ -442,17 +470,59 @@ struct XahauGenesis_test : public beast::unit_test::suite { testcase("Test activation"); using namespace jtx; - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; activate(__LINE__, env, false, false, false); } + void + testBadNetworkIDActivation(FeatureBitset features) + { + testcase("Test Bad Network ID activation"); + using namespace jtx; + std::vector badNetIDs{ + 0, + 1, + 2, + 10, + 100, + 1000, + 10000, + 20000, + 21000, + 21328, + 21329, + 21340, + 21341, + 65535}; + + for (int netid : badNetIDs) + { + Env env{ + *this, + makeNetworkConfig(netid), + features - featureXahauGenesis}; + activate(__LINE__, env, false, false, false, true); + } + + for (int netid = 21330; netid <= 21339; ++netid) + { + Env env{ + *this, + makeNetworkConfig(netid), + features - featureXahauGenesis}; + activate(__LINE__, env, false, false, false, false); + } + } + void testWithSignerList(FeatureBitset features) { using namespace jtx; testcase("Test signerlist"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; Account const alice{"alice", KeyType::ed25519}; env.fund(XRP(1000), alice); @@ -468,7 +538,8 @@ struct XahauGenesis_test : public beast::unit_test::suite { using namespace jtx; testcase("Test regkey"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; env.memoize(env.master); Account const alice("alice"); @@ -667,7 +738,11 @@ struct XahauGenesis_test : public beast::unit_test::suite { using namespace jtx; testcase("Test governance membership voting L1"); - Env env{*this, envconfig(), features - featureXahauGenesis, nullptr}; + Env env{ + *this, + makeNetworkConfig(21337), + features - featureXahauGenesis, + nullptr}; auto const alice = Account("alice"); auto const bob = Account("bob"); @@ -2111,7 +2186,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("Test governance membership voting L2"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; auto const alice = Account("alice"); auto const bob = Account("bob"); @@ -3708,7 +3784,7 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test last close time"); - Env env{*this, envconfig(), features}; + Env env{*this, makeNetworkConfig(21337), features}; validateTime(lastClose(env), 0); // last close = 0 @@ -3738,7 +3814,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("test claim reward rate is == 0"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -3783,7 +3860,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("test claim reward rate is > 1"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -3828,7 +3906,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("test claim reward delay is == 0"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -3873,7 +3952,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("test claim reward delay is < 0"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -3918,7 +3998,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace jtx; testcase("test claim reward before time"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -3968,7 +4049,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward valid without unl report"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; bool const has240819 = env.current()->rules().enabled(fix240819); double const rateDrops = 0.00333333333 * 1'000'000; @@ -4115,7 +4197,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward valid with unl report"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -4250,7 +4333,7 @@ struct XahauGenesis_test : public beast::unit_test::suite { FeatureBitset _features = features - featureXahauGenesis; auto const amend = withXahauV1 ? _features : _features - fixXahauV1; - Env env{*this, envconfig(), amend}; + Env env{*this, makeNetworkConfig(21337), amend}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -4387,7 +4470,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward optin optout"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; bool const has240819 = env.current()->rules().enabled(fix240819); double const rateDrops = 0.00333333333 * 1'000'000; @@ -4499,7 +4583,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward bal == 1"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -4587,7 +4672,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward elapsed_since_last == 1"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -4668,7 +4754,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test claim reward elapsed_since_last == 0"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; STAmount const feesXRP = XRP(1); @@ -4929,7 +5016,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test compound interest over 12 claims"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5027,7 +5115,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test deposit"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5117,7 +5206,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test deposit withdraw"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5209,7 +5299,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test deposit late"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5299,7 +5390,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test deposit late withdraw"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5392,7 +5484,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test no claim"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5480,7 +5573,8 @@ struct XahauGenesis_test : public beast::unit_test::suite using namespace std::chrono_literals; testcase("test no claim late"); - Env env{*this, envconfig(), features - featureXahauGenesis}; + Env env{ + *this, makeNetworkConfig(21337), features - featureXahauGenesis}; double const rateDrops = 0.00333333333 * 1'000'000; STAmount const feesXRP = XRP(1); @@ -5594,6 +5688,7 @@ struct XahauGenesis_test : public beast::unit_test::suite testGovernHookWithFeats(FeatureBitset features) { testPlainActivation(features); + testBadNetworkIDActivation(features); testWithSignerList(features); testWithRegularKey(features); testGovernanceL1(features); From e086724772b68a96732d65cf4996277e164aa267 Mon Sep 17 00:00:00 2001 From: Richard Holland Date: Wed, 13 Nov 2024 11:35:59 +1100 Subject: [PATCH 03/17] UDP RPC (admin) support (#390) --- src/ripple/app/misc/NetworkOPs.cpp | 3 +- src/ripple/rpc/handlers/Subscribe.cpp | 10 +- src/ripple/rpc/handlers/Unsubscribe.cpp | 7 + src/ripple/rpc/impl/ServerHandlerImp.cpp | 200 ++++++++++++++++ src/ripple/rpc/impl/ServerHandlerImp.h | 15 ++ src/ripple/rpc/impl/UDPInfoSub.h | 140 +++++++++++ src/ripple/server/Port.h | 9 + src/ripple/server/impl/Port.cpp | 7 + src/ripple/server/impl/ServerImpl.h | 28 ++- src/ripple/server/impl/UDPDoor.h | 284 +++++++++++++++++++++++ src/test/server/Server_test.cpp | 16 ++ 11 files changed, 712 insertions(+), 7 deletions(-) create mode 100644 src/ripple/rpc/impl/UDPInfoSub.h create mode 100644 src/ripple/server/impl/UDPDoor.h diff --git a/src/ripple/app/misc/NetworkOPs.cpp b/src/ripple/app/misc/NetworkOPs.cpp index 6f6bfcc1c9..637f7725ef 100644 --- a/src/ripple/app/misc/NetworkOPs.cpp +++ b/src/ripple/app/misc/NetworkOPs.cpp @@ -67,9 +67,10 @@ #include #include #include +#include #include #include - +#include #include #include #include diff --git a/src/ripple/rpc/handlers/Subscribe.cpp b/src/ripple/rpc/handlers/Subscribe.cpp index f17aa62b62..90cf682e1a 100644 --- a/src/ripple/rpc/handlers/Subscribe.cpp +++ b/src/ripple/rpc/handlers/Subscribe.cpp @@ -30,6 +30,7 @@ #include #include #include +#include namespace ripple { @@ -42,7 +43,7 @@ doSubscribe(RPC::JsonContext& context) if (!context.infoSub && !context.params.isMember(jss::url)) { // Must be a JSON-RPC call. - JLOG(context.j.info()) << "doSubscribe: RPC subscribe requires a url"; + JLOG(context.j.warn()) << "doSubscribe: RPC subscribe requires a url"; return rpcError(rpcINVALID_PARAMS); } @@ -373,6 +374,13 @@ doSubscribe(RPC::JsonContext& context) } } + if (ispSub) + { + if (std::shared_ptr udp = + std::dynamic_pointer_cast(ispSub)) + udp->increment(); + } + return jvResult; } diff --git a/src/ripple/rpc/handlers/Unsubscribe.cpp b/src/ripple/rpc/handlers/Unsubscribe.cpp index 8a606a26dc..4df234cd28 100644 --- a/src/ripple/rpc/handlers/Unsubscribe.cpp +++ b/src/ripple/rpc/handlers/Unsubscribe.cpp @@ -25,6 +25,7 @@ #include #include #include +#include namespace ripple { @@ -245,6 +246,12 @@ doUnsubscribe(RPC::JsonContext& context) context.netOps.tryRemoveRpcSub(context.params[jss::url].asString()); } + if (ispSub) + { + if (auto udp = std::dynamic_pointer_cast(ispSub)) + udp->destroy(); + } + return jvResult; } diff --git a/src/ripple/rpc/impl/ServerHandlerImp.cpp b/src/ripple/rpc/impl/ServerHandlerImp.cpp index 81075a5c02..c4e41fa27b 100644 --- a/src/ripple/rpc/impl/ServerHandlerImp.cpp +++ b/src/ripple/rpc/impl/ServerHandlerImp.cpp @@ -361,6 +361,67 @@ ServerHandlerImp::onWSMessage( } } +void +ServerHandlerImp::onUDPMessage( + std::string const& message, + boost::asio::ip::tcp::endpoint const& remoteEndpoint, + std::function sendResponse) +{ + Json::Value jv; + if (message.size() > RPC::Tuning::maxRequestSize || + !Json::Reader{}.parse(message, jv) || !jv.isObject()) + { + Json::Value jvResult(Json::objectValue); + jvResult[jss::type] = jss::error; + jvResult[jss::error] = "jsonInvalid"; + jvResult[jss::value] = message; + + std::string const response = to_string(jvResult); + JLOG(m_journal.trace()) + << "UDP sending error response: '" << jvResult << "'"; + sendResponse(response); + return; + } + + JLOG(m_journal.trace()) + << "UDP received '" << jv << "' from " << remoteEndpoint; + + auto const postResult = m_jobQueue.postCoro( + jtCLIENT_RPC, // Using RPC job type since this is admin RPC + "UDP-RPC", + [this, + remoteEndpoint, + jv = std::move(jv), + sendResponse = std::move(sendResponse)]( + std::shared_ptr const& coro) { + // Process the request similar to WebSocket but with UDP context + Role const role = Role::ADMIN; // UDP-RPC is admin-only + auto const jr = + this->processUDP(jv, role, coro, sendResponse, remoteEndpoint); + + std::string const response = to_string(jr); + JLOG(m_journal.trace()) + << "UDP sending '" << jr << "' to " << remoteEndpoint; + + // Send response back via UDP + sendResponse(response); + }); + + if (postResult == nullptr) + { + // Request rejected, probably shutting down + Json::Value jvResult(Json::objectValue); + jvResult[jss::type] = jss::error; + jvResult[jss::error] = "serverShuttingDown"; + jvResult[jss::value] = "Server is shutting down"; + + std::string const response = to_string(jvResult); + JLOG(m_journal.trace()) + << "UDP sending shutdown response to " << remoteEndpoint; + sendResponse(response); + } +} + void ServerHandlerImp::onClose(Session& session, boost::system::error_code const&) { @@ -397,6 +458,145 @@ logDuration( << " microseconds. request = " << request; } +Json::Value +ServerHandlerImp::processUDP( + Json::Value const& jv, + Role const& role, + std::shared_ptr const& coro, + std::optional> + sendResponse /* used for subscriptions */, + boost::asio::ip::tcp::endpoint const& remoteEndpoint) +{ + std::shared_ptr is; + // Requests without "command" are invalid. + Json::Value jr(Json::objectValue); + try + { + auto apiVersion = + RPC::getAPIVersionNumber(jv, app_.config().BETA_RPC_API); + if (apiVersion == RPC::apiInvalidVersion || + (!jv.isMember(jss::command) && !jv.isMember(jss::method)) || + (jv.isMember(jss::command) && !jv[jss::command].isString()) || + (jv.isMember(jss::method) && !jv[jss::method].isString()) || + (jv.isMember(jss::command) && jv.isMember(jss::method) && + jv[jss::command].asString() != jv[jss::method].asString())) + { + jr[jss::type] = jss::response; + jr[jss::status] = jss::error; + jr[jss::error] = apiVersion == RPC::apiInvalidVersion + ? jss::invalid_API_version + : jss::missingCommand; + jr[jss::request] = jv; + if (jv.isMember(jss::id)) + jr[jss::id] = jv[jss::id]; + if (jv.isMember(jss::jsonrpc)) + jr[jss::jsonrpc] = jv[jss::jsonrpc]; + if (jv.isMember(jss::ripplerpc)) + jr[jss::ripplerpc] = jv[jss::ripplerpc]; + if (jv.isMember(jss::api_version)) + jr[jss::api_version] = jv[jss::api_version]; + + return jr; + } + + auto required = RPC::roleRequired( + apiVersion, + app_.config().BETA_RPC_API, + jv.isMember(jss::command) ? jv[jss::command].asString() + : jv[jss::method].asString()); + if (Role::FORBID == role) + { + jr[jss::result] = rpcError(rpcFORBIDDEN); + } + else + { + Resource::Consumer c; + Resource::Charge loadType = Resource::feeReferenceRPC; + + if (sendResponse.has_value()) + is = UDPInfoSub::getInfoSub( + m_networkOPs, *sendResponse, remoteEndpoint); + + RPC::JsonContext context{ + {app_.journal("RPCHandler"), + app_, + loadType, + app_.getOPs(), + app_.getLedgerMaster(), + c, + role, + coro, + is, + apiVersion}, + jv}; + + auto start = std::chrono::system_clock::now(); + RPC::doCommand(context, jr[jss::result]); + auto end = std::chrono::system_clock::now(); + logDuration(jv, end - start, m_journal); + } + } + catch (std::exception const& ex) + { + jr[jss::result] = RPC::make_error(rpcINTERNAL); + JLOG(m_journal.error()) + << "Exception while processing WS: " << ex.what() << "\n" + << "Input JSON: " << Json::Compact{Json::Value{jv}}; + } + + if (is) + { + if (auto udp = std::dynamic_pointer_cast(is)) + udp->destroy(); + } + + // Currently we will simply unwrap errors returned by the RPC + // API, in the future maybe we can make the responses + // consistent. + // + // Regularize result. This is duplicate code. + if (jr[jss::result].isMember(jss::error)) + { + jr = jr[jss::result]; + jr[jss::status] = jss::error; + + auto rq = jv; + + if (rq.isObject()) + { + if (rq.isMember(jss::passphrase.c_str())) + rq[jss::passphrase.c_str()] = ""; + if (rq.isMember(jss::secret.c_str())) + rq[jss::secret.c_str()] = ""; + if (rq.isMember(jss::seed.c_str())) + rq[jss::seed.c_str()] = ""; + if (rq.isMember(jss::seed_hex.c_str())) + rq[jss::seed_hex.c_str()] = ""; + } + + jr[jss::request] = rq; + } + else + { + if (jr[jss::result].isMember("forwarded") && + jr[jss::result]["forwarded"]) + jr = jr[jss::result]; + jr[jss::status] = jss::success; + } + + if (jv.isMember(jss::id)) + jr[jss::id] = jv[jss::id]; + if (jv.isMember(jss::jsonrpc)) + jr[jss::jsonrpc] = jv[jss::jsonrpc]; + if (jv.isMember(jss::ripplerpc)) + jr[jss::ripplerpc] = jv[jss::ripplerpc]; + if (jv.isMember(jss::api_version)) + jr[jss::api_version] = jv[jss::api_version]; + + jr[jss::type] = jss::response; + return jr; +} + Json::Value ServerHandlerImp::processSession( std::shared_ptr const& session, diff --git a/src/ripple/rpc/impl/ServerHandlerImp.h b/src/ripple/rpc/impl/ServerHandlerImp.h index 7c0bf9c9ae..36ee6f5e29 100644 --- a/src/ripple/rpc/impl/ServerHandlerImp.h +++ b/src/ripple/rpc/impl/ServerHandlerImp.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -164,6 +165,12 @@ class ServerHandlerImp std::shared_ptr session, std::vector const& buffers); + void + onUDPMessage( + std::string const& message, + boost::asio::ip::tcp::endpoint const& remoteEndpoint, + std::function sendResponse); + void onClose(Session& session, boost::system::error_code const&); @@ -177,6 +184,14 @@ class ServerHandlerImp std::shared_ptr const& coro, Json::Value const& jv); + Json::Value + processUDP( + Json::Value const& jv, + Role const& role, + std::shared_ptr const& coro, + std::optional> sendResponse, + boost::asio::ip::tcp::endpoint const& remoteEndpoint); + void processSession( std::shared_ptr const&, diff --git a/src/ripple/rpc/impl/UDPInfoSub.h b/src/ripple/rpc/impl/UDPInfoSub.h new file mode 100644 index 0000000000..4766b7b071 --- /dev/null +++ b/src/ripple/rpc/impl/UDPInfoSub.h @@ -0,0 +1,140 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2012, 2013 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_RPC_UDPINFOSUB_H +#define RIPPLE_RPC_UDPINFOSUB_H + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ripple { +class UDPInfoSub : public InfoSub +{ + std::function send_; + boost::asio::ip::tcp::endpoint endpoint_; + + UDPInfoSub( + Source& source, + std::function& sendResponse, + boost::asio::ip::tcp::endpoint const& remoteEndpoint) + : InfoSub(source), send_(sendResponse), endpoint_(remoteEndpoint) + { + } + + struct RefCountedSub + { + std::shared_ptr sub; + size_t refCount; + + RefCountedSub(std::shared_ptr s) + : sub(std::move(s)), refCount(1) + { + } + }; + + static inline std::mutex mtx_; + static inline std::map map_; + +public: + static std::shared_ptr + getInfoSub( + Source& source, + std::function& sendResponse, + boost::asio::ip::tcp::endpoint const& remoteEndpoint) + { + std::lock_guard lock(mtx_); + + auto it = map_.find(remoteEndpoint); + if (it != map_.end()) + { + it->second.refCount++; + return it->second.sub; + } + + auto sub = std::shared_ptr( + new UDPInfoSub(source, sendResponse, remoteEndpoint)); + map_.emplace(remoteEndpoint, RefCountedSub(sub)); + return sub; + } + + static bool + increment(boost::asio::ip::tcp::endpoint const& remoteEndpoint) + { + std::lock_guard lock(mtx_); + + auto it = map_.find(remoteEndpoint); + if (it != map_.end()) + { + it->second.refCount++; + return true; + } + return false; + } + + bool + increment() + { + return increment(endpoint_); + } + + static bool + destroy(boost::asio::ip::tcp::endpoint const& remoteEndpoint) + { + std::lock_guard lock(mtx_); + + auto it = map_.find(remoteEndpoint); + if (it != map_.end()) + { + if (--it->second.refCount == 0) + { + map_.erase(it); + return true; + } + } + return false; + } + + bool + destroy() + { + return destroy(endpoint_); + } + + void + send(Json::Value const& jv, bool) override + { + std::string const str = to_string(jv); + send_(str); + } + + boost::asio::ip::tcp::endpoint const& + endpoint() const + { + return endpoint_; + } +}; +} // namespace ripple +#endif diff --git a/src/ripple/server/Port.h b/src/ripple/server/Port.h index 9dccfdf9c0..438d521eab 100644 --- a/src/ripple/server/Port.h +++ b/src/ripple/server/Port.h @@ -86,6 +86,15 @@ struct Port // Returns a string containing the list of protocols std::string protocols() const; + + bool + has_udp() const + { + return protocol.count("udp") > 0; + } + + // Maximum UDP packet size (default 64KB) + std::size_t udp_packet_size = 65536; }; std::ostream& diff --git a/src/ripple/server/impl/Port.cpp b/src/ripple/server/impl/Port.cpp index 1b869f6a5d..a3e88d5cd6 100644 --- a/src/ripple/server/impl/Port.cpp +++ b/src/ripple/server/impl/Port.cpp @@ -244,6 +244,13 @@ parse_Port(ParsedPort& port, Section const& section, std::ostream& log) optResult->begin(), optResult->end())) port.protocol.insert(s); } + + if (port.protocol.count("udp") > 0 && port.protocol.size() > 1) + { + log << "Port " << section.name() + << " cannot mix UDP with other protocols"; + Throw(); + } } { diff --git a/src/ripple/server/impl/ServerImpl.h b/src/ripple/server/impl/ServerImpl.h index a3abf78914..8c06223962 100644 --- a/src/ripple/server/impl/ServerImpl.h +++ b/src/ripple/server/impl/ServerImpl.h @@ -24,6 +24,7 @@ #include #include #include +#include #include #include #include @@ -162,18 +163,35 @@ ServerImpl::ports(std::vector const& ports) { if (closed()) Throw("ports() on closed Server"); + ports_.reserve(ports.size()); Endpoints eps; eps.reserve(ports.size()); + for (auto const& port : ports) { ports_.push_back(port); - if (auto sp = ios_.emplace>( - handler_, io_service_, ports_.back(), j_)) + + if (port.has_udp()) + { + // UDP-RPC door + if (auto sp = ios_.emplace>( + handler_, io_service_, ports_.back(), j_)) + { + eps.push_back(sp->get_endpoint()); + sp->run(); + } + } + else { - list_.push_back(sp); - eps.push_back(sp->get_endpoint()); - sp->run(); + // Standard TCP door + if (auto sp = ios_.emplace>( + handler_, io_service_, ports_.back(), j_)) + { + list_.push_back(sp); + eps.push_back(sp->get_endpoint()); + sp->run(); + } } } return eps; diff --git a/src/ripple/server/impl/UDPDoor.h b/src/ripple/server/impl/UDPDoor.h new file mode 100644 index 0000000000..60e417d2cf --- /dev/null +++ b/src/ripple/server/impl/UDPDoor.h @@ -0,0 +1,284 @@ +//------------------------------------------------------------------------------ +/* + This file is part of rippled: https://github.com/ripple/rippled + Copyright (c) 2012, 2013 Ripple Labs Inc. + + Permission to use, copy, modify, and/or distribute this software for any + purpose with or without fee is hereby granted, provided that the above + copyright notice and this permission notice appear in all copies. + + THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + ANY SPECIAL , DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +*/ +//============================================================================== + +#ifndef RIPPLE_SERVER_UDPDOOR_H_INCLUDED +#define RIPPLE_SERVER_UDPDOOR_H_INCLUDED + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace ripple { + +template +class UDPDoor : public io_list::work, + public std::enable_shared_from_this> +{ +private: + using error_code = boost::system::error_code; + using endpoint_type = boost::asio::ip::tcp::endpoint; + using udp_socket = boost::asio::ip::udp::socket; + + beast::Journal const j_; + Port const& port_; + Handler& handler_; + boost::asio::io_context& ioc_; + boost::asio::strand strand_; + udp_socket socket_; + std::vector recv_buffer_; + endpoint_type local_endpoint_; // Store TCP-style endpoint + +public: + UDPDoor( + Handler& handler, + boost::asio::io_context& io_context, + Port const& port, + beast::Journal j) + : j_(j) + , port_(port) + , handler_(handler) + , ioc_(io_context) + , strand_(io_context.get_executor()) + , socket_(io_context) + , recv_buffer_(port.udp_packet_size) + , local_endpoint_(port.ip, port.port) // Store as TCP endpoint + { + error_code ec; + + // Create UDP endpoint from port configuration + auto const addr = port_.ip.to_v4(); + boost::asio::ip::udp::endpoint udp_endpoint(addr, port_.port); + + socket_.open(boost::asio::ip::udp::v4(), ec); + if (ec) + { + JLOG(j_.error()) << "UDP socket open failed: " << ec.message(); + return; + } + + // Set socket options + socket_.set_option(boost::asio::socket_base::reuse_address(true), ec); + if (ec) + { + JLOG(j_.error()) + << "UDP set reuse_address failed: " << ec.message(); + return; + } + + socket_.bind(udp_endpoint, ec); + if (ec) + { + JLOG(j_.error()) << "UDP socket bind failed: " << ec.message(); + return; + } + + JLOG(j_.info()) << "UDP-RPC listening on " << udp_endpoint; + } + + endpoint_type + get_endpoint() const + { + return local_endpoint_; + } + + void + run() + { + if (!socket_.is_open()) + return; + + do_receive(); + } + + void + close() override + { + error_code ec; + socket_.close(ec); + } + +private: + void + do_receive() + { + if (!socket_.is_open()) + return; + + socket_.async_receive_from( + boost::asio::buffer(recv_buffer_), + sender_endpoint_, + boost::asio::bind_executor( + strand_, + std::bind( + &UDPDoor::on_receive, + this->shared_from_this(), + std::placeholders::_1, + std::placeholders::_2))); + } + + void + on_receive(error_code ec, std::size_t bytes_transferred) + { + if (ec) + { + if (ec != boost::asio::error::operation_aborted) + { + JLOG(j_.error()) << "UDP receive failed: " << ec.message(); + do_receive(); + } + return; + } + + // Convert UDP endpoint to TCP endpoint for compatibility + endpoint_type tcp_endpoint( + sender_endpoint_.address(), sender_endpoint_.port()); + + // Handle the received UDP message + handler_.onUDPMessage( + std::string(recv_buffer_.data(), bytes_transferred), + tcp_endpoint, + [this, tcp_endpoint](std::string const& response) { + do_send(response, tcp_endpoint); + }); + + do_receive(); + } + + void + do_send(std::string const& response, endpoint_type const& tcp_endpoint) + { + if (!socket_.is_open()) + { + std::cout << "UDP SOCKET NOT OPEN WHEN SENDING\n\n"; + return; + } + + const size_t HEADER_SIZE = 16; + const size_t MAX_DATAGRAM_SIZE = + 65487; // Allow for ipv6 header 40 bytes + 8 bytes of udp header + const size_t MAX_PAYLOAD_SIZE = MAX_DATAGRAM_SIZE - HEADER_SIZE; + + // Convert TCP endpoint back to UDP for sending + boost::asio::ip::udp::endpoint udp_endpoint( + tcp_endpoint.address(), tcp_endpoint.port()); + + // If message fits in single datagram, send normally + if (response.length() <= MAX_DATAGRAM_SIZE) + { + socket_.async_send_to( + boost::asio::buffer(response), + udp_endpoint, + boost::asio::bind_executor( + strand_, + [this, self = this->shared_from_this()]( + error_code ec, std::size_t bytes_transferred) { + if (ec && ec != boost::asio::error::operation_aborted) + { + JLOG(j_.error()) + << "UDP send failed: " << ec.message(); + } + })); + return; + } + + // Calculate number of packets needed + const size_t payload_size = MAX_PAYLOAD_SIZE; + const uint16_t total_packets = + (response.length() + payload_size - 1) / payload_size; + + // Get current timestamp in microseconds + auto now = std::chrono::system_clock::now(); + auto micros = std::chrono::duration_cast( + now.time_since_epoch()) + .count(); + uint64_t timestamp = static_cast(micros); + + // Send fragmented packets + for (uint16_t packet_num = 0; packet_num < total_packets; packet_num++) + { + std::string fragment; + fragment.reserve(MAX_DATAGRAM_SIZE); + + // Add header - 4 bytes of zeros + fragment.push_back(0); + fragment.push_back(0); + fragment.push_back(0); + fragment.push_back(0); + + // Add packet number (little endian) + fragment.push_back(packet_num & 0xFF); + fragment.push_back((packet_num >> 8) & 0xFF); + + // Add total packets (little endian) + fragment.push_back(total_packets & 0xFF); + fragment.push_back((total_packets >> 8) & 0xFF); + + // Add timestamp (8 bytes, little endian) + fragment.push_back(timestamp & 0xFF); + fragment.push_back((timestamp >> 8) & 0xFF); + fragment.push_back((timestamp >> 16) & 0xFF); + fragment.push_back((timestamp >> 24) & 0xFF); + fragment.push_back((timestamp >> 32) & 0xFF); + fragment.push_back((timestamp >> 40) & 0xFF); + fragment.push_back((timestamp >> 48) & 0xFF); + fragment.push_back((timestamp >> 56) & 0xFF); + + // Calculate payload slice + size_t start = packet_num * payload_size; + size_t length = std::min(payload_size, response.length() - start); + fragment.append(response.substr(start, length)); + + socket_.async_send_to( + boost::asio::buffer(fragment), + udp_endpoint, + boost::asio::bind_executor( + strand_, + [this, self = this->shared_from_this()]( + error_code ec, std::size_t bytes_transferred) { + if (ec && ec != boost::asio::error::operation_aborted) + { + JLOG(j_.error()) + << "UDP send failed: " << ec.message(); + } + })); + } + } + + boost::asio::ip::udp::endpoint sender_endpoint_; +}; + +} // namespace ripple + +#endif diff --git a/src/test/server/Server_test.cpp b/src/test/server/Server_test.cpp index b5eb71f361..d141b9ebc1 100644 --- a/src/test/server/Server_test.cpp +++ b/src/test/server/Server_test.cpp @@ -144,6 +144,14 @@ class Server_test : public beast::unit_test::suite { } + void + onUDPMessage( + std::string const& message, + boost::asio::ip::tcp::endpoint const& remoteEndpoint, + std::function sendResponse) + { + } + void onClose(Session& session, boost::system::error_code const&) { @@ -349,6 +357,14 @@ class Server_test : public beast::unit_test::suite { } + void + onUDPMessage( + std::string const& message, + boost::asio::ip::tcp::endpoint const& remoteEndpoint, + std::function sendResponse) + { + } + void onClose(Session& session, boost::system::error_code const&) { From 542172f0a1ae21cca09444891f4bf8d3cd985148 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ekiserrep=C3=A9?= <126416117+Ekiserrepe@users.noreply.github.com> Date: Mon, 25 Nov 2024 23:52:49 +0100 Subject: [PATCH 04/17] Update README.md (#396) Updated Xaman link. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e73c034613..c9335cfc10 100644 --- a/README.md +++ b/README.md @@ -67,5 +67,5 @@ git-subtree. See those directories' README files for more details. - [explorer.xahau.network](https://explorer.xahau.network) - **Testnet & Faucet**: Test applications and obtain test XAH at [xahau-test.net](https://xahau-test.net) and use the testnet explorer at [explorer.xahau.network](https://explorer.xahau.network). - **Supporting Wallets**: A list of wallets that support XAH and Xahau-based assets. - - [Xumm](https://xumm.app) + - [Xaman](https://xaman.app) - [Crossmark](https://crossmark.io) From 9d54da3880b3c256823b80c88a01ba185f6c3733 Mon Sep 17 00:00:00 2001 From: Denis Angell Date: Thu, 28 Nov 2024 10:20:44 +0100 Subject: [PATCH 05/17] Fix: failing assert (#397) --- src/ripple/nodestore/impl/Shard.cpp | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/src/ripple/nodestore/impl/Shard.cpp b/src/ripple/nodestore/impl/Shard.cpp index 8d0eab8115..ac4af4782b 100644 --- a/src/ripple/nodestore/impl/Shard.cpp +++ b/src/ripple/nodestore/impl/Shard.cpp @@ -710,10 +710,7 @@ Shard::finalize(bool writeSQLite, std::optional const& referenceHash) if (writeSQLite && !storeSQLite(ledger)) return fail("failed storing to SQLite databases"); - assert( - ledger->info().seq == ledgerSeq && - (ledger->info().seq < XRP_LEDGER_EARLIEST_FEES || - ledger->read(keylet::fees()))); + assert(ledger->info().seq == ledgerSeq && ledger->read(keylet::fees())); hash = ledger->info().parentHash; next = std::move(ledger); From e9468d8b4a9af85ec4a17f1361af46ecb5dafd9e Mon Sep 17 00:00:00 2001 From: RichardAH Date: Wed, 11 Dec 2024 09:38:16 +1000 Subject: [PATCH 06/17] Datagram monitor (#400) Co-authored-by: Denis Angell --- Builds/CMake/RippledCore.cmake | 1 + src/ripple/app/ledger/LedgerMaster.h | 3 + src/ripple/app/ledger/impl/LedgerMaster.cpp | 7 + src/ripple/app/main/Application.cpp | 11 + src/ripple/app/misc/DatagramMonitor.h | 1052 +++++++++++++++++++ src/ripple/app/misc/NetworkOPs.cpp | 166 +-- src/ripple/app/misc/NetworkOPs.h | 41 +- src/ripple/app/misc/StateAccounting.cpp | 49 + src/ripple/app/misc/StateAccounting.h | 99 ++ src/ripple/core/Config.h | 2 + src/ripple/core/ConfigSections.h | 1 + src/ripple/core/impl/Config.cpp | 46 +- 12 files changed, 1295 insertions(+), 183 deletions(-) create mode 100644 src/ripple/app/misc/DatagramMonitor.h create mode 100644 src/ripple/app/misc/StateAccounting.cpp create mode 100644 src/ripple/app/misc/StateAccounting.h diff --git a/Builds/CMake/RippledCore.cmake b/Builds/CMake/RippledCore.cmake index 5a25d2741e..78843991f1 100644 --- a/Builds/CMake/RippledCore.cmake +++ b/Builds/CMake/RippledCore.cmake @@ -392,6 +392,7 @@ target_sources (rippled PRIVATE src/ripple/app/misc/NegativeUNLVote.cpp src/ripple/app/misc/NetworkOPs.cpp src/ripple/app/misc/SHAMapStoreImp.cpp + src/ripple/app/misc/StateAccounting.cpp src/ripple/app/misc/detail/impl/WorkSSL.cpp src/ripple/app/misc/impl/AccountTxPaging.cpp src/ripple/app/misc/impl/AmendmentTable.cpp diff --git a/src/ripple/app/ledger/LedgerMaster.h b/src/ripple/app/ledger/LedgerMaster.h index 3d7adc8622..040ef3bf6c 100644 --- a/src/ripple/app/ledger/LedgerMaster.h +++ b/src/ripple/app/ledger/LedgerMaster.h @@ -152,6 +152,9 @@ class LedgerMaster : public AbstractFetchPackContainer std::string getCompleteLedgers(); + RangeSet + getCompleteLedgersRangeSet(); + /** Apply held transactions to the open ledger This is normally called as we close the ledger. The open ledger remains open to handle new transactions diff --git a/src/ripple/app/ledger/impl/LedgerMaster.cpp b/src/ripple/app/ledger/impl/LedgerMaster.cpp index 844e9da48e..4a3301a9c3 100644 --- a/src/ripple/app/ledger/impl/LedgerMaster.cpp +++ b/src/ripple/app/ledger/impl/LedgerMaster.cpp @@ -1714,6 +1714,13 @@ LedgerMaster::getCompleteLedgers() return to_string(mCompleteLedgers); } +RangeSet +LedgerMaster::getCompleteLedgersRangeSet() +{ + std::lock_guard sl(mCompleteLock); + return mCompleteLedgers; +} + std::optional LedgerMaster::getCloseTimeBySeq(LedgerIndex ledgerIndex) { diff --git a/src/ripple/app/main/Application.cpp b/src/ripple/app/main/Application.cpp index 9134df0355..be0a7b46a6 100644 --- a/src/ripple/app/main/Application.cpp +++ b/src/ripple/app/main/Application.cpp @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #include @@ -167,6 +168,8 @@ class ApplicationImp : public Application, public BasicApp std::unique_ptr logs_; std::unique_ptr timeKeeper_; + std::unique_ptr datagram_monitor_; + std::uint64_t const instanceCookie_; beast::Journal m_journal; @@ -1523,6 +1526,14 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline) if (reportingETL_) reportingETL_->start(); + // Datagram monitor if applicable + if (!config_->standalone() && config_->DATAGRAM_MONITOR != "") + { + datagram_monitor_ = std::make_unique(*this); + if (datagram_monitor_) + datagram_monitor_->start(); + } + return true; } diff --git a/src/ripple/app/misc/DatagramMonitor.h b/src/ripple/app/misc/DatagramMonitor.h new file mode 100644 index 0000000000..ba6ce02133 --- /dev/null +++ b/src/ripple/app/misc/DatagramMonitor.h @@ -0,0 +1,1052 @@ +// +#ifndef RIPPLE_APP_MAIN_DATAGRAMMONITOR_H_INCLUDED +#define RIPPLE_APP_MAIN_DATAGRAMMONITOR_H_INCLUDED + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#if defined(__linux__) +#include +#include +#elif defined(__APPLE__) +#include +#include +#include +#include +#include +#include +#include +#include +#endif +#include +#include + +namespace ripple { + +// Magic number for server info packets: 'XDGM' (le) Xahau DataGram Monitor +constexpr uint32_t SERVER_INFO_MAGIC = 0x4D474458; +constexpr uint32_t SERVER_INFO_VERSION = 1; + +// Warning flag bits +constexpr uint32_t WARNING_AMENDMENT_BLOCKED = 1 << 0; +constexpr uint32_t WARNING_UNL_BLOCKED = 1 << 1; +constexpr uint32_t WARNING_AMENDMENT_WARNED = 1 << 2; +constexpr uint32_t WARNING_NOT_SYNCED = 1 << 3; + +// Time window statistics for rates +struct [[gnu::packed]] MetricRates +{ + double rate_1m; // Average rate over last minute + double rate_5m; // Average rate over last 5 minutes + double rate_1h; // Average rate over last hour + double rate_24h; // Average rate over last 24 hours +}; + +struct AllRates +{ + MetricRates network_in; + MetricRates network_out; + MetricRates disk_read; + MetricRates disk_write; +}; + +// Structure to represent a ledger sequence range +struct [[gnu::packed]] LgrRange +{ + uint32_t start; + uint32_t end; +}; + +// Map is returned separately since variable-length data +// shouldn't be included in network structures +using ObjectCountMap = std::vector, int>>; + +struct [[gnu::packed]] DebugCounters +{ + // Database metrics + std::uint64_t dbKBTotal{0}; + std::uint64_t dbKBLedger{0}; + std::uint64_t dbKBTransaction{0}; + std::uint64_t localTxCount{0}; + + // Basic metrics + std::uint32_t writeLoad{0}; + std::int32_t historicalPerMinute{0}; + + // Cache metrics + std::uint32_t sleHitRate{0}; // Stored as fixed point, multiplied by 1000 + std::uint32_t ledgerHitRate{ + 0}; // Stored as fixed point, multiplied by 1000 + std::uint32_t alSize{0}; + std::uint32_t alHitRate{0}; // Stored as fixed point, multiplied by 1000 + std::int32_t fullbelowSize{0}; + std::uint32_t treenodeCacheSize{0}; + std::uint32_t treenodeTrackSize{0}; + + // Shard metrics + std::int32_t shardFullbelowSize{0}; + std::uint32_t shardTreenodeCacheSize{0}; + std::uint32_t shardTreenodeTrackSize{0}; + std::uint32_t shardWriteLoad{0}; + std::uint64_t shardNodeWrites{0}; + std::uint64_t shardNodeReadsTotal{0}; + std::uint64_t shardNodeReadsHit{0}; + std::uint64_t shardNodeWrittenBytes{0}; + std::uint64_t shardNodeReadBytes{0}; + + // Node store metrics (when not using shards) + std::uint64_t nodeWriteCount{0}; + std::uint64_t nodeWriteSize{0}; + std::uint64_t nodeFetchCount{0}; + std::uint64_t nodeFetchHitCount{0}; + std::uint64_t nodeFetchSize{0}; +}; + +// Core server metrics in the fixed header +struct [[gnu::packed]] ServerInfoHeader +{ + // Fixed header fields come first + uint32_t magic; // Magic number to identify packet type + uint32_t version; // Protocol version number + uint32_t network_id; // Network ID from config + uint32_t server_state; // Operating mode as enum + uint32_t peer_count; // Number of connected peers + uint32_t node_size; // Size category (0=tiny through 4=huge) + uint32_t cpu_cores; // CPU core count + uint32_t ledger_range_count; // Number of range entries + uint32_t warning_flags; // Warning flags (reduced size) + + uint32_t padding_1; // padding for alignment + + // 64-bit metrics + uint64_t timestamp; // System time in microseconds + uint64_t uptime; // Server uptime in seconds + uint64_t io_latency_us; // IO latency in microseconds + uint64_t validation_quorum; // Validation quorum count + uint64_t fetch_pack_size; // Size of fetch pack cache + uint64_t proposer_count; // Number of proposers in last close + uint64_t converge_time_ms; // Last convergence time in ms + uint64_t load_factor; // Load factor (scaled by 1M) + uint64_t load_base; // Load base value + uint64_t reserve_base; // Reserve base amount + uint64_t reserve_inc; // Reserve increment amount + uint64_t ledger_seq; // Latest ledger sequence + + // Fixed-size byte arrays + uint8_t ledger_hash[32]; // Latest ledger hash + uint8_t node_public_key[33]; // Node's public key + uint8_t padding2[7]; // Padding to maintain 8-byte alignment + uint8_t version_string[32]; + + // System metrics + uint64_t process_memory_pages; // Process memory usage in bytes + uint64_t system_memory_total; // Total system memory in bytes + uint64_t system_memory_free; // Free system memory in bytes + uint64_t system_memory_used; // Used system memory in bytes + uint64_t system_disk_total; // Total disk space in bytes + uint64_t system_disk_free; // Free disk space in bytes + uint64_t system_disk_used; // Used disk space in bytes + uint64_t io_wait_time; // IO wait time in milliseconds + double load_avg_1min; // 1 minute load average + double load_avg_5min; // 5 minute load average + double load_avg_15min; // 15 minute load average + + // State transition metrics + uint64_t state_transitions[5]; // Count for each operating mode + uint64_t state_durations[5]; // Duration in each mode + uint64_t initial_sync_us; // Initial sync duration + + // Network and disk rates remain unchanged + struct + { + MetricRates network_in; + MetricRates network_out; + MetricRates disk_read; + MetricRates disk_write; + } rates; + + DebugCounters dbg_counters; +}; + +// System metrics collected for rate calculations +struct SystemMetrics +{ + uint64_t timestamp; // When metrics were collected + uint64_t network_bytes_in; // Current total bytes in + uint64_t network_bytes_out; // Current total bytes out + uint64_t disk_bytes_read; // Current total bytes read + uint64_t disk_bytes_written; // Current total bytes written +}; + +class MetricsTracker +{ +private: + static constexpr size_t SAMPLES_1M = 60; // 1 sample/second for 1 minute + static constexpr size_t SAMPLES_5M = 300; // 1 sample/second for 5 minutes + static constexpr size_t SAMPLES_1H = 3600; // 1 sample/second for 1 hour + static constexpr size_t SAMPLES_24H = 1440; // 1 sample/minute for 24 hours + + std::vector samples_1m{SAMPLES_1M}; + std::vector samples_5m{SAMPLES_5M}; + std::vector samples_1h{SAMPLES_1H}; + std::vector samples_24h{SAMPLES_24H}; + + size_t index_1m{0}, index_5m{0}, index_1h{0}, index_24h{0}; + std::chrono::system_clock::time_point last_24h_sample{}; + + double + calculateRate( + const SystemMetrics& current, + const std::vector& samples, + size_t current_index, + size_t max_samples, + bool is_24h_window, + std::function metric_getter) + { + // If we don't have at least 2 samples, the rate is 0 + if (current_index < 2) + { + return 0.0; + } + + // Calculate time window based on the window type + uint64_t expected_window_micros; + if (is_24h_window) + { + expected_window_micros = + 24ULL * 60ULL * 60ULL * 1000000ULL; // 24 hours in microseconds + } + else + { + expected_window_micros = max_samples * + 1000000ULL; // window in seconds * 1,000,000 for microseconds + } + + // For any window where we don't have full data, we should scale the + // rate based on the actual time we have data for + uint64_t actual_window_micros = + current.timestamp - samples[0].timestamp; + double window_scale = std::min( + 1.0, + static_cast(actual_window_micros) / expected_window_micros); + + // Get the oldest valid sample + size_t oldest_index = (current_index >= max_samples) + ? ((current_index + 1) % max_samples) + : 0; + const auto& oldest = samples[oldest_index]; + + double elapsed = actual_window_micros / + 1000000.0; // Convert microseconds to seconds + + // Ensure we have a meaningful time difference + if (elapsed < 0.001) + { // Less than 1ms difference + return 0.0; + } + + uint64_t current_value = metric_getter(current); + uint64_t oldest_value = metric_getter(oldest); + + // Handle counter wraparound + uint64_t diff = (current_value >= oldest_value) + ? (current_value - oldest_value) + : (std::numeric_limits::max() - oldest_value + + current_value + 1); + + // Calculate the rate and scale it based on our window coverage + return (static_cast(diff) / elapsed) * window_scale; + } + + MetricRates + calculateMetricRates( + const SystemMetrics& current, + std::function metric_getter) + { + MetricRates rates; + rates.rate_1m = calculateRate( + current, samples_1m, index_1m, SAMPLES_1M, false, metric_getter); + rates.rate_5m = calculateRate( + current, samples_5m, index_5m, SAMPLES_5M, false, metric_getter); + rates.rate_1h = calculateRate( + current, samples_1h, index_1h, SAMPLES_1H, false, metric_getter); + rates.rate_24h = calculateRate( + current, samples_24h, index_24h, SAMPLES_24H, true, metric_getter); + return rates; + } + +public: + void + addSample(const SystemMetrics& metrics) + { + auto now = std::chrono::system_clock::now(); + + // Update 1-minute window (every second) + samples_1m[index_1m++ % SAMPLES_1M] = metrics; + + // Update 5-minute window (every second) + samples_5m[index_5m++ % SAMPLES_5M] = metrics; + + // Update 1-hour window (every second) + samples_1h[index_1h++ % SAMPLES_1H] = metrics; + + // Update 24-hour window (every minute) + if (last_24h_sample + std::chrono::minutes(1) <= now) + { + samples_24h[index_24h++ % SAMPLES_24H] = metrics; + last_24h_sample = now; + } + } + + AllRates + getRates(const SystemMetrics& current) + { + AllRates rates; + rates.network_in = calculateMetricRates( + current, [](const SystemMetrics& m) { return m.network_bytes_in; }); + rates.network_out = calculateMetricRates( + current, + [](const SystemMetrics& m) { return m.network_bytes_out; }); + rates.disk_read = calculateMetricRates( + current, [](const SystemMetrics& m) { return m.disk_bytes_read; }); + rates.disk_write = calculateMetricRates( + current, + [](const SystemMetrics& m) { return m.disk_bytes_written; }); + return rates; + } +}; + +class DatagramMonitor +{ +private: + Application& app_; + std::atomic running_{false}; + std::thread monitor_thread_; + MetricsTracker metrics_tracker_; + + struct EndpointInfo + { + std::string ip; + uint16_t port; + bool is_ipv6; + }; + EndpointInfo + parseEndpoint(std::string const& endpoint) + { + auto space_pos = endpoint.find(' '); + if (space_pos == std::string::npos) + throw std::runtime_error("Invalid endpoint format"); + + EndpointInfo info; + info.ip = endpoint.substr(0, space_pos); + info.port = std::stoi(endpoint.substr(space_pos + 1)); + info.is_ipv6 = info.ip.find(':') != std::string::npos; + return info; + } + + int + createSocket(EndpointInfo const& endpoint) + { + int sock = socket(endpoint.is_ipv6 ? AF_INET6 : AF_INET, SOCK_DGRAM, 0); + if (sock < 0) + throw std::runtime_error("Failed to create socket"); + return sock; + } + + void + sendPacket( + int sock, + EndpointInfo const& endpoint, + std::vector const& buffer) + { + struct sockaddr_storage addr; + socklen_t addr_len; + + if (endpoint.is_ipv6) + { + struct sockaddr_in6* addr6 = + reinterpret_cast(&addr); + addr6->sin6_family = AF_INET6; + addr6->sin6_port = htons(endpoint.port); + inet_pton(AF_INET6, endpoint.ip.c_str(), &addr6->sin6_addr); + addr_len = sizeof(struct sockaddr_in6); + } + else + { + struct sockaddr_in* addr4 = + reinterpret_cast(&addr); + addr4->sin_family = AF_INET; + addr4->sin_port = htons(endpoint.port); + inet_pton(AF_INET, endpoint.ip.c_str(), &addr4->sin_addr); + addr_len = sizeof(struct sockaddr_in); + } + + sendto( + sock, + buffer.data(), + buffer.size(), + 0, + reinterpret_cast(&addr), + addr_len); + } + + // Returns both the counters and object count map separately + std::pair + getDebugCounters() + { + DebugCounters counters; + ObjectCountMap objectCounts = + CountedObjects::getInstance().getCounts(1); + + // Database metrics if applicable + if (!app_.config().reporting() && app_.config().useTxTables()) + { + auto const db = + dynamic_cast(&app_.getRelationalDatabase()); + if (!db) + Throw("Failed to get relational database"); + + if (auto dbKB = db->getKBUsedAll()) + counters.dbKBTotal = dbKB; + if (auto dbKB = db->getKBUsedLedger()) + counters.dbKBLedger = dbKB; + if (auto dbKB = db->getKBUsedTransaction()) + counters.dbKBTransaction = dbKB; + if (auto count = app_.getOPs().getLocalTxCount()) + counters.localTxCount = count; + } + + // Basic metrics + counters.writeLoad = app_.getNodeStore().getWriteLoad(); + counters.historicalPerMinute = + static_cast(app_.getInboundLedgers().fetchRate()); + + // Cache metrics - convert floating point rates to fixed point + counters.sleHitRate = + static_cast(app_.cachedSLEs().rate() * 1000); + counters.ledgerHitRate = static_cast( + app_.getLedgerMaster().getCacheHitRate() * 1000); + counters.alSize = app_.getAcceptedLedgerCache().size(); + counters.alHitRate = static_cast( + app_.getAcceptedLedgerCache().getHitRate() * 1000); + counters.fullbelowSize = static_cast( + app_.getNodeFamily().getFullBelowCache(0)->size()); + counters.treenodeCacheSize = + app_.getNodeFamily().getTreeNodeCache(0)->getCacheSize(); + counters.treenodeTrackSize = + app_.getNodeFamily().getTreeNodeCache(0)->getTrackSize(); + + // Handle shard metrics if available + if (auto shardStore = app_.getShardStore()) + { + auto shardFamily = + dynamic_cast(app_.getShardFamily()); + auto const [cacheSz, trackSz] = shardFamily->getTreeNodeCacheSize(); + + counters.shardFullbelowSize = shardFamily->getFullBelowCacheSize(); + counters.shardTreenodeCacheSize = cacheSz; + counters.shardTreenodeTrackSize = trackSz; + counters.shardWriteLoad = shardStore->getWriteLoad(); + counters.shardNodeWrites = shardStore->getStoreCount(); + counters.shardNodeReadsTotal = shardStore->getFetchTotalCount(); + counters.shardNodeReadsHit = shardStore->getFetchHitCount(); + counters.shardNodeWrittenBytes = shardStore->getStoreSize(); + counters.shardNodeReadBytes = shardStore->getFetchSize(); + } + else + { + // Get regular node store metrics + counters.nodeWriteCount = app_.getNodeStore().getStoreCount(); + counters.nodeWriteSize = app_.getNodeStore().getStoreSize(); + counters.nodeFetchCount = app_.getNodeStore().getFetchTotalCount(); + counters.nodeFetchHitCount = app_.getNodeStore().getFetchHitCount(); + counters.nodeFetchSize = app_.getNodeStore().getFetchSize(); + } + + return {counters, objectCounts}; + } + + uint32_t + getPhysicalCPUCount() + { + static uint32_t count = 0; + if (count > 0) + return count; + +#if defined(__linux__) + try + { + std::ifstream cpuinfo("/proc/cpuinfo"); + if (!cpuinfo) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Unable to open file: /proc/cpuinfo"; + return count; + } + std::string line; + std::set physical_ids; + std::string current_physical_id; + + while (std::getline(cpuinfo, line)) + { + if (line.find("core id") != std::string::npos) + { + current_physical_id = line.substr(line.find(":") + 1); + // Trim whitespace + current_physical_id.erase( + 0, current_physical_id.find_first_not_of(" \t")); + current_physical_id.erase( + current_physical_id.find_last_not_of(" \t") + 1); + physical_ids.insert(current_physical_id); + } + } + + count = physical_ids.size(); + } + catch (const std::exception& e) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Error getting CPU count: " << e.what(); + } + + // Return at least 1 if we couldn't determine the count + return count > 0 ? count : (count = 1); +#elif defined(__APPLE__) + int value = 0; + size_t size = sizeof(value); + if (sysctlbyname("hw.physicalcpu", &value, &size, NULL, 0) == 0) + count = value; + return count > 0 ? count : (count = 1); +#endif + } + + SystemMetrics + collectSystemMetrics() + { + SystemMetrics metrics{}; + metrics.timestamp = + std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()) + .count(); + +#if defined(__linux__) + // Network stats collection + try + { + std::ifstream net_file("/proc/net/dev"); + if (!net_file) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Unable to open file /proc/net/dev"; + return metrics; + } + + std::string line; + uint64_t total_bytes_in = 0, total_bytes_out = 0; + + // Skip header lines + std::getline(net_file, line); // Inter-| Receive... + std::getline(net_file, line); // face |bytes... + + while (std::getline(net_file, line)) + { + if (line.find(':') != std::string::npos) + { + std::string interface = line.substr(0, line.find(':')); + interface = + interface.substr(interface.find_first_not_of(" \t")); + interface = interface.substr( + 0, interface.find_last_not_of(" \t") + 1); + + // Skip loopback interface + if (interface == "lo") + continue; + + uint64_t bytes_in, bytes_out; + std::istringstream iss(line.substr(line.find(':') + 1)); + iss >> bytes_in; // First field after : is bytes_in + for (int i = 0; i < 8; ++i) + iss >> std::ws; // Skip 8 fields + iss >> bytes_out; // 9th field is bytes_out + + total_bytes_in += bytes_in; + total_bytes_out += bytes_out; + } + } + metrics.network_bytes_in = total_bytes_in; + metrics.network_bytes_out = total_bytes_out; + } + catch (const std::exception& e) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Error collecting network stats: " << e.what(); + } + + // Disk stats collection + try + { + std::ifstream disk_file("/proc/diskstats"); + if (!disk_file) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Unable to open file: /proc/diskstats"; + return metrics; + } + std::string line; + uint64_t total_bytes_read = 0, total_bytes_written = 0; + + while (std::getline(disk_file, line)) + { + unsigned int major, minor; + char dev_name[32]; + uint64_t reads, read_sectors, writes, write_sectors; + + if (sscanf( + line.c_str(), + "%u %u %31s %lu %*u %lu %*u %lu %*u %lu", + &major, + &minor, + dev_name, + &reads, + &read_sectors, + &writes, + &write_sectors) == 7) + { + // Only process physical devices + std::string device_name(dev_name); + if (device_name.substr(0, 3) == "dm-" || + device_name.substr(0, 4) == "loop" || + device_name.substr(0, 3) == "ram") + { + continue; + } + + // Skip partitions (usually have a number at the end) + if (std::isdigit(device_name.back())) + { + continue; + } + + uint64_t bytes_read = read_sectors * 512; + uint64_t bytes_written = write_sectors * 512; + + total_bytes_read += bytes_read; + total_bytes_written += bytes_written; + } + } + metrics.disk_bytes_read = total_bytes_read; + metrics.disk_bytes_written = total_bytes_written; + } + catch (const std::exception& e) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Error collecting disk stats: " << e.what(); + } +#elif defined(__APPLE__) + // Network stats collection + try + { + struct ifaddrs* ifap; + if (getifaddrs(&ifap) == 0) + { + uint64_t total_bytes_in = 0, total_bytes_out = 0; + for (struct ifaddrs* ifa = ifap; ifa; ifa = ifa->ifa_next) + { + if (ifa->ifa_addr != NULL && + ifa->ifa_addr->sa_family == AF_LINK) + { + struct if_data* ifd = (struct if_data*)ifa->ifa_data; + if (ifd != NULL) + { + // Skip loopback interface + if (strcmp(ifa->ifa_name, "lo0") == 0) + continue; + + total_bytes_in += ifd->ifi_ibytes; + total_bytes_out += ifd->ifi_obytes; + } + } + } + freeifaddrs(ifap); + + metrics.network_bytes_in = total_bytes_in; + metrics.network_bytes_out = total_bytes_out; + } + } + catch (const std::exception& e) + { + JLOG(app_.journal("DatagramMonitor").error()) + << "Error collecting network stats: " << e.what(); + } + + // Disk stats collection + // Disk IO stats are not easily accessible in macOS. + // We'll set these values to zero for now. + metrics.disk_bytes_read = 0; + metrics.disk_bytes_written = 0; +#endif + return metrics; + } + + std::vector + generateServerInfo() + { + auto& ops = app_.getOPs(); + + auto [dbg_counters, obj_count_map] = getDebugCounters(); + + // Get the RangeSet directly + auto rangeSet = app_.getLedgerMaster().getCompleteLedgersRangeSet(); + auto currentMetrics = collectSystemMetrics(); + metrics_tracker_.addSample(currentMetrics); + + // Count only non-zero intervals and calculate total size needed + size_t validRangeCount = 0; + for (auto const& interval : rangeSet) + { + // Skip intervals where both lower and upper are 0 + if (interval.lower() != 0 || interval.upper() != 0) + { + validRangeCount++; + } + } + + size_t totalSize = sizeof(ServerInfoHeader) + + (validRangeCount * sizeof(LgrRange)) + (64 * obj_count_map.size()); + + // Allocate buffer and initialize header + std::vector buffer(totalSize); + auto* header = reinterpret_cast(buffer.data()); + memset(header, 0, sizeof(ServerInfoHeader)); + + // Set magic number and version + header->magic = SERVER_INFO_MAGIC; + header->version = SERVER_INFO_VERSION; + header->network_id = app_.config().NETWORK_ID; + header->timestamp = + std::chrono::duration_cast( + std::chrono::system_clock::now().time_since_epoch()) + .count(); + header->uptime = UptimeClock::now().time_since_epoch().count(); + header->io_latency_us = app_.getIOLatency().count(); + header->validation_quorum = app_.validators().quorum(); + + if (!app_.config().reporting()) + header->peer_count = app_.overlay().size(); + + header->node_size = app_.config().NODE_SIZE; + + // Get state accounting data + auto const [counters, mode, start, initialSync] = + app_.getOPs().getStateAccountingData(); + + // Pack state metrics into header + for (size_t i = 0; i < 5; ++i) + { + header->state_transitions[i] = counters[i].transitions; + header->state_durations[i] = counters[i].dur.count(); + } + header->initial_sync_us = initialSync; + + // Pack warning flags + if (ops.isAmendmentBlocked()) + header->warning_flags |= WARNING_AMENDMENT_BLOCKED; + if (ops.isUNLBlocked()) + header->warning_flags |= WARNING_UNL_BLOCKED; + if (ops.isAmendmentWarned()) + header->warning_flags |= WARNING_AMENDMENT_WARNED; + + // Pack consensus info + auto& mConsensus = ops.getConsensus(); + header->proposer_count = mConsensus.prevProposers(); + header->converge_time_ms = mConsensus.prevRoundTime().count(); + + // Pack fetch pack size if present + auto& ledgerMaster = ops.getLedgerMaster(); + auto const lastClosed = ledgerMaster.getClosedLedger(); + auto const validated = ledgerMaster.getValidatedLedger(); + + if (lastClosed && validated) + { + auto consensus = + ledgerMaster.getLedgerByHash(lastClosed->info().hash); + if (!consensus) + consensus = app_.getInboundLedgers().acquire( + lastClosed->info().hash, + 0, + InboundLedger::Reason::CONSENSUS); + + if (consensus && + (!ledgerMaster.canBeCurrent(consensus) || + !ledgerMaster.isCompatible( + *consensus, + app_.journal("DatagramMonitor").debug(), + "Not switching"))) + { + header->warning_flags |= WARNING_NOT_SYNCED; + } + } + else + { + // If we don't have both lastClosed and validated ledgers, we're + // definitely not synced + header->warning_flags |= WARNING_NOT_SYNCED; + } + + auto const fp = ledgerMaster.getFetchPackCacheSize(); + if (fp != 0) + header->fetch_pack_size = fp; + + // Pack load factor info if not reporting + if (!app_.config().reporting()) + { + auto const escalationMetrics = + app_.getTxQ().getMetrics(*app_.openLedger().current()); + auto const loadFactorServer = app_.getFeeTrack().getLoadFactor(); + auto const loadBaseServer = app_.getFeeTrack().getLoadBase(); + auto const loadFactorFeeEscalation = + mulDiv( + escalationMetrics.openLedgerFeeLevel, + loadBaseServer, + escalationMetrics.referenceFeeLevel) + .second; + + header->load_factor = std::max( + safe_cast(loadFactorServer), + loadFactorFeeEscalation); + header->load_base = loadBaseServer; + } + +#if defined(__linux__) + // Get system info using sysinfo + struct sysinfo si; + if (sysinfo(&si) == 0) + { + header->system_memory_total = si.totalram * si.mem_unit; + header->system_memory_free = si.freeram * si.mem_unit; + header->system_memory_used = + header->system_memory_total - header->system_memory_free; + header->load_avg_1min = si.loads[0] / (float)(1 << SI_LOAD_SHIFT); + header->load_avg_5min = si.loads[1] / (float)(1 << SI_LOAD_SHIFT); + header->load_avg_15min = si.loads[2] / (float)(1 << SI_LOAD_SHIFT); + } +#elif defined(__APPLE__) + // Get total physical memory + int64_t physical_memory; + size_t length = sizeof(physical_memory); + if (sysctlbyname("hw.memsize", &physical_memory, &length, NULL, 0) == 0) + { + header->system_memory_total = physical_memory; + } + + // Get free and used memory + vm_statistics_data_t vm_stats; + mach_msg_type_number_t count = HOST_VM_INFO_COUNT; + if (host_statistics( + mach_host_self(), + HOST_VM_INFO, + (host_info_t)&vm_stats, + &count) == KERN_SUCCESS) + { + uint64_t page_size; + length = sizeof(page_size); + sysctlbyname("hw.pagesize", &page_size, &length, NULL, 0); + + header->system_memory_free = + (uint64_t)vm_stats.free_count * page_size; + header->system_memory_used = + header->system_memory_total - header->system_memory_free; + } + + // Get load averages + double loadavg[3]; + if (getloadavg(loadavg, 3) == 3) + { + header->load_avg_1min = loadavg[0]; + header->load_avg_5min = loadavg[1]; + header->load_avg_15min = loadavg[2]; + } +#endif + + // Get process memory usage + struct rusage usage; + getrusage(RUSAGE_SELF, &usage); + header->process_memory_pages = usage.ru_maxrss; + + // Get disk usage +#if defined(__linux__) + struct statvfs fs; + if (statvfs("/", &fs) == 0) + { + header->system_disk_total = fs.f_blocks * fs.f_frsize; + header->system_disk_free = fs.f_bfree * fs.f_frsize; + header->system_disk_used = + header->system_disk_total - header->system_disk_free; + } +#elif defined(__APPLE__) + struct statfs fs; + if (statfs("/", &fs) == 0) + { + header->system_disk_total = fs.f_blocks * fs.f_bsize; + header->system_disk_free = fs.f_bfree * fs.f_bsize; + header->system_disk_used = + header->system_disk_total - header->system_disk_free; + } +#endif + + // Get CPU core count + header->cpu_cores = getPhysicalCPUCount(); + + // Get rate statistics + auto rates = metrics_tracker_.getRates(currentMetrics); + header->rates.network_in = rates.network_in; + header->rates.network_out = rates.network_out; + header->rates.disk_read = rates.disk_read; + header->rates.disk_write = rates.disk_write; + + // Pack ledger info and ranges + auto lpClosed = ledgerMaster.getValidatedLedger(); + if (!lpClosed && !app_.config().reporting()) + lpClosed = ledgerMaster.getClosedLedger(); + + if (lpClosed) + { + header->ledger_seq = lpClosed->info().seq; + auto const& hash = lpClosed->info().hash; + std::memcpy(header->ledger_hash, hash.data(), 32); + header->reserve_base = lpClosed->fees().accountReserve(0).drops(); + header->reserve_inc = lpClosed->fees().increment.drops(); + } + + // Pack node public key + auto const& nodeKey = app_.nodeIdentity().first; + std::memcpy(header->node_public_key, nodeKey.data(), 33); + + // Pack version string + memset(&header->version_string, 0, 32); + memcpy( + &header->version_string, + BuildInfo::getVersionString().c_str(), + BuildInfo::getVersionString().size() > 32 + ? 32 + : BuildInfo::getVersionString().size()); + + header->dbg_counters = dbg_counters; + + // Set the complete ledger count + header->ledger_range_count = validRangeCount; + + // Append only non-zero ranges after the header + auto* rangeData = reinterpret_cast( + buffer.data() + sizeof(ServerInfoHeader)); + size_t i = 0; + for (auto const& interval : rangeSet) + { + // Only pack non-zero ranges + if (interval.lower() != 0 || interval.upper() != 0) + { + rangeData[i].start = interval.lower(); + rangeData[i].end = interval.upper(); + ++i; + } + } + + uint8_t* end_of_ranges = reinterpret_cast(buffer.data()) + + sizeof(ServerInfoHeader) + (validRangeCount * sizeof(LgrRange)); + + memset(end_of_ranges, 0, 64 * obj_count_map.size()); + + uint8_t* ptr = end_of_ranges; + for (auto& [name, val] : obj_count_map) + { + size_t to_write = name.size() > 56 ? 56 : name.size(); + memcpy(ptr, name.c_str(), to_write); + ptr += 56; + *reinterpret_cast(ptr) = val; + ptr += 8; + } + + return buffer; + } + void + monitorThread() + { + auto endpoint = parseEndpoint(app_.config().DATAGRAM_MONITOR); + int sock = createSocket(endpoint); + + while (running_) + { + try + { + auto info = generateServerInfo(); + sendPacket(sock, endpoint, info); + std::this_thread::sleep_for(std::chrono::seconds(1)); + } + catch (const std::exception& e) + { + // Log error but continue monitoring + JLOG(app_.journal("DatagramMonitor").error()) + << "Server info monitor error: " << e.what(); + } + } + + close(sock); + } + +public: + DatagramMonitor(Application& app) : app_(app) + { + } + + void + start() + { + if (!running_.exchange(true)) + { + monitor_thread_ = + std::thread(&DatagramMonitor::monitorThread, this); + } + } + + void + stop() + { + if (running_.exchange(false)) + { + if (monitor_thread_.joinable()) + monitor_thread_.join(); + } + } + + ~DatagramMonitor() + { + stop(); + } +}; +} // namespace ripple +#endif diff --git a/src/ripple/app/misc/NetworkOPs.cpp b/src/ripple/app/misc/NetworkOPs.cpp index 637f7725ef..df1b1ba08d 100644 --- a/src/ripple/app/misc/NetworkOPs.cpp +++ b/src/ripple/app/misc/NetworkOPs.cpp @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -70,7 +71,6 @@ #include #include #include -#include #include #include #include @@ -117,81 +117,6 @@ class NetworkOPsImp final : public NetworkOPs running, }; - static std::array const states_; - - /** - * State accounting records two attributes for each possible server state: - * 1) Amount of time spent in each state (in microseconds). This value is - * updated upon each state transition. - * 2) Number of transitions to each state. - * - * This data can be polled through server_info and represented by - * monitoring systems similarly to how bandwidth, CPU, and other - * counter-based metrics are managed. - * - * State accounting is more accurate than periodic sampling of server - * state. With periodic sampling, it is very likely that state transitions - * are missed, and accuracy of time spent in each state is very rough. - */ - class StateAccounting - { - struct Counters - { - explicit Counters() = default; - - std::uint64_t transitions = 0; - std::chrono::microseconds dur = std::chrono::microseconds(0); - }; - - OperatingMode mode_ = OperatingMode::DISCONNECTED; - std::array counters_; - mutable std::mutex mutex_; - std::chrono::steady_clock::time_point start_ = - std::chrono::steady_clock::now(); - std::chrono::steady_clock::time_point const processStart_ = start_; - std::uint64_t initialSyncUs_{0}; - static std::array const states_; - - public: - explicit StateAccounting() - { - counters_[static_cast(OperatingMode::DISCONNECTED)] - .transitions = 1; - } - - /** - * Record state transition. Update duration spent in previous - * state. - * - * @param om New state. - */ - void - mode(OperatingMode om); - - /** - * Output state counters in JSON format. - * - * @obj Json object to which to add state accounting data. - */ - void - json(Json::Value& obj) const; - - struct CounterData - { - decltype(counters_) counters; - decltype(mode_) mode; - decltype(start_) start; - decltype(initialSyncUs_) initialSyncUs; - }; - - CounterData - getCounterData() const - { - std::lock_guard lock(mutex_); - return {counters_, mode_, start_, initialSyncUs_}; - } - }; - //! Server fees published on `server` subscription struct ServerFeeSummary { @@ -273,6 +198,9 @@ class NetworkOPsImp final : public NetworkOPs std::string strOperatingMode(bool const admin = false) const override; + StateAccounting::CounterData + getStateAccountingData(); + // // Transaction operations. // @@ -777,11 +705,17 @@ class NetworkOPsImp final : public NetworkOPs DispatchState mDispatchState = DispatchState::none; std::vector mTransactions; - StateAccounting accounting_{}; + StateAccounting accounting_; std::set pendingValidations_; std::mutex validationsMutex_; + RCLConsensus& + getConsensus(); + + LedgerMaster& + getLedgerMaster(); + private: struct Stats { @@ -844,19 +778,6 @@ class NetworkOPsImp final : public NetworkOPs //------------------------------------------------------------------------------ -static std::array const stateNames{ - {"disconnected", "connected", "syncing", "tracking", "full"}}; - -std::array const NetworkOPsImp::states_ = stateNames; - -std::array const - NetworkOPsImp::StateAccounting::states_ = { - {Json::StaticString(stateNames[0]), - Json::StaticString(stateNames[1]), - Json::StaticString(stateNames[2]), - Json::StaticString(stateNames[3]), - Json::StaticString(stateNames[4])}}; - static auto const genesisAccountId = calcAccountID( generateKeyPair(KeyType::secp256k1, generateSeed("masterpassphrase")) .first); @@ -1131,7 +1052,7 @@ NetworkOPsImp::strOperatingMode(OperatingMode const mode, bool const admin) } } - return states_[static_cast(mode)]; + return {StateAccounting::states_[static_cast(mode)].c_str()}; } void @@ -2397,6 +2318,19 @@ NetworkOPsImp::getConsensusInfo() return mConsensus.getJson(true); } +// RHTODO: not threadsafe? +RCLConsensus& +NetworkOPsImp::getConsensus() +{ + return mConsensus; +} + +LedgerMaster& +NetworkOPsImp::getLedgerMaster() +{ + return m_ledgerMaster; +} + Json::Value NetworkOPsImp::getServerInfo(bool human, bool admin, bool counters) { @@ -4194,6 +4128,12 @@ NetworkOPsImp::stateAccounting(Json::Value& obj) accounting_.json(obj); } +StateAccounting::CounterData +NetworkOPsImp::getStateAccountingData() +{ + return accounting_.getCounterData(); +} + // <-- bool: true=erased, false=was not there bool NetworkOPsImp::unsubValidations(std::uint64_t uSeq) @@ -4664,50 +4604,6 @@ NetworkOPsImp::collect_metrics() counters[static_cast(OperatingMode::FULL)].transitions); } -void -NetworkOPsImp::StateAccounting::mode(OperatingMode om) -{ - auto now = std::chrono::steady_clock::now(); - - std::lock_guard lock(mutex_); - ++counters_[static_cast(om)].transitions; - if (om == OperatingMode::FULL && - counters_[static_cast(om)].transitions == 1) - { - initialSyncUs_ = std::chrono::duration_cast( - now - processStart_) - .count(); - } - counters_[static_cast(mode_)].dur += - std::chrono::duration_cast(now - start_); - - mode_ = om; - start_ = now; -} - -void -NetworkOPsImp::StateAccounting::json(Json::Value& obj) const -{ - auto [counters, mode, start, initialSync] = getCounterData(); - auto const current = std::chrono::duration_cast( - std::chrono::steady_clock::now() - start); - counters[static_cast(mode)].dur += current; - - obj[jss::state_accounting] = Json::objectValue; - for (std::size_t i = static_cast(OperatingMode::DISCONNECTED); - i <= static_cast(OperatingMode::FULL); - ++i) - { - obj[jss::state_accounting][states_[i]] = Json::objectValue; - auto& state = obj[jss::state_accounting][states_[i]]; - state[jss::transitions] = std::to_string(counters[i].transitions); - state[jss::duration_us] = std::to_string(counters[i].dur.count()); - } - obj[jss::server_state_duration_us] = std::to_string(current.count()); - if (initialSync) - obj[jss::initial_sync_duration_us] = std::to_string(initialSync); -} - //------------------------------------------------------------------------------ std::unique_ptr diff --git a/src/ripple/app/misc/NetworkOPs.h b/src/ripple/app/misc/NetworkOPs.h index d53127ed3b..350542404c 100644 --- a/src/ripple/app/misc/NetworkOPs.h +++ b/src/ripple/app/misc/NetworkOPs.h @@ -20,8 +20,10 @@ #ifndef RIPPLE_APP_MISC_NETWORKOPS_H_INCLUDED #define RIPPLE_APP_MISC_NETWORKOPS_H_INCLUDED +#include #include #include +#include #include #include #include @@ -42,35 +44,6 @@ class LedgerMaster; class Transaction; class ValidatorKeys; -// This is the primary interface into the "client" portion of the program. -// Code that wants to do normal operations on the network such as -// creating and monitoring accounts, creating transactions, and so on -// should use this interface. The RPC code will primarily be a light wrapper -// over this code. -// -// Eventually, it will check the node's operating mode (synched, unsynched, -// etectera) and defer to the correct means of processing. The current -// code assumes this node is synched (and will continue to do so until -// there's a functional network. -// - -/** Specifies the mode under which the server believes it's operating. - - This has implications about how the server processes transactions and - how it responds to requests (e.g. account balance request). - - @note Other code relies on the numerical values of these constants; do - not change them without verifying each use and ensuring that it is - not a breaking change. -*/ -enum class OperatingMode { - DISCONNECTED = 0, //!< not ready to process requests - CONNECTED = 1, //!< convinced we are talking to the network - SYNCING = 2, //!< fallen slightly behind - TRACKING = 3, //!< convinced we agree with the network - FULL = 4 //!< we have the ledger and can even validate -}; - /** Provides server functionality for clients. Clients include backend applications, local commands, and connected @@ -221,6 +194,13 @@ class NetworkOPs : public InfoSub::Source virtual Json::Value getConsensusInfo() = 0; + + virtual RCLConsensus& + getConsensus() = 0; + + virtual LedgerMaster& + getLedgerMaster() = 0; + virtual Json::Value getServerInfo(bool human, bool admin, bool counters) = 0; virtual void @@ -228,6 +208,9 @@ class NetworkOPs : public InfoSub::Source virtual Json::Value getLedgerFetchInfo() = 0; + virtual StateAccounting::CounterData + getStateAccountingData() = 0; + /** Accepts the current transaction tree, return the new ledger's sequence This API is only used via RPC with the server in STANDALONE mode and diff --git a/src/ripple/app/misc/StateAccounting.cpp b/src/ripple/app/misc/StateAccounting.cpp new file mode 100644 index 0000000000..ade601a806 --- /dev/null +++ b/src/ripple/app/misc/StateAccounting.cpp @@ -0,0 +1,49 @@ +#include + +namespace ripple { + +void +StateAccounting::mode(OperatingMode om) +{ + std::lock_guard lock(mutex_); + auto now = std::chrono::steady_clock::now(); + + ++counters_[static_cast(om)].transitions; + if (om == OperatingMode::FULL && + counters_[static_cast(om)].transitions == 1) + { + initialSyncUs_ = std::chrono::duration_cast( + now - processStart_) + .count(); + } + counters_[static_cast(mode_)].dur += + std::chrono::duration_cast(now - start_); + + mode_ = om; + start_ = now; +} + +void +StateAccounting::json(Json::Value& obj) +{ + auto [counters, mode, start, initialSync] = getCounterData(); + auto const current = std::chrono::duration_cast( + std::chrono::steady_clock::now() - start); + counters[static_cast(mode)].dur += current; + + obj[jss::state_accounting] = Json::objectValue; + for (std::size_t i = static_cast(OperatingMode::DISCONNECTED); + i <= static_cast(OperatingMode::FULL); + ++i) + { + obj[jss::state_accounting][states_[i]] = Json::objectValue; + auto& state = obj[jss::state_accounting][states_[i]]; + state[jss::transitions] = std::to_string(counters[i].transitions); + state[jss::duration_us] = std::to_string(counters[i].dur.count()); + } + obj[jss::server_state_duration_us] = std::to_string(current.count()); + if (initialSync) + obj[jss::initial_sync_duration_us] = std::to_string(initialSync); +} + +} // namespace ripple diff --git a/src/ripple/app/misc/StateAccounting.h b/src/ripple/app/misc/StateAccounting.h new file mode 100644 index 0000000000..3cbc5cc733 --- /dev/null +++ b/src/ripple/app/misc/StateAccounting.h @@ -0,0 +1,99 @@ +#ifndef RIPPLE_APP_MAIN_STATEACCOUNTING_H_INCLUDED +#define RIPPLE_APP_MAIN_STATEACCOUNTING_H_INCLUDED + +#include +#include +#include +#include +#include +#include + +namespace ripple { + +// This is the primary interface into the "client" portion of the program. +// Code that wants to do normal operations on the network such as +// creating and monitoring accounts, creating transactions, and so on +// should use this interface. The RPC code will primarily be a light wrapper +// over this code. +// +// Eventually, it will check the node's operating mode (synched, unsynched, +// etectera) and defer to the correct means of processing. The current +// code assumes this node is synched (and will continue to do so until +// there's a functional network. +// + +/** Specifies the mode under which the server believes it's operating. + + This has implications about how the server processes transactions and + how it responds to requests (e.g. account balance request). + + @note Other code relies on the numerical values of these constants; do + not change them without verifying each use and ensuring that it is + not a breaking change. +*/ +enum class OperatingMode { + DISCONNECTED = 0, //!< not ready to process requests + CONNECTED = 1, //!< convinced we are talking to the network + SYNCING = 2, //!< fallen slightly behind + TRACKING = 3, //!< convinced we agree with the network + FULL = 4 //!< we have the ledger and can even validate +}; + +class StateAccounting +{ +public: + constexpr static std::array const states_ = { + {Json::StaticString("disconnected"), + Json::StaticString("connected"), + Json::StaticString("syncing"), + Json::StaticString("tracking"), + Json::StaticString("full")}}; + + struct Counters + { + explicit Counters() = default; + + std::uint64_t transitions = 0; + std::chrono::microseconds dur = std::chrono::microseconds(0); + }; + +private: + OperatingMode mode_ = OperatingMode::DISCONNECTED; + std::array counters_; + mutable std::mutex mutex_; + std::chrono::steady_clock::time_point start_ = + std::chrono::steady_clock::now(); + std::chrono::steady_clock::time_point const processStart_ = start_; + std::uint64_t initialSyncUs_{0}; + +public: + explicit StateAccounting() + { + counters_[static_cast(OperatingMode::DISCONNECTED)] + .transitions = 1; + } + + //! Record state transition. Update duration spent in previous state. + void + mode(OperatingMode om); + + //! Output state counters in JSON format. + void + json(Json::Value& obj); + + using CounterData = std::tuple< + decltype(counters_), + decltype(mode_), + decltype(start_), + decltype(initialSyncUs_)>; + + CounterData + getCounterData() + { + return {counters_, mode_, start_, initialSyncUs_}; + } +}; + +} // namespace ripple + +#endif diff --git a/src/ripple/core/Config.h b/src/ripple/core/Config.h index 2779547e29..5d9977770c 100644 --- a/src/ripple/core/Config.h +++ b/src/ripple/core/Config.h @@ -155,6 +155,8 @@ class Config : public BasicConfig std::map IMPORT_VL_KEYS; // hex string -> class PublicKey (for caching purposes) + std::string DATAGRAM_MONITOR; + enum StartUpType { FRESH, NORMAL, diff --git a/src/ripple/core/ConfigSections.h b/src/ripple/core/ConfigSections.h index 27f38bc6e4..def5b3c82e 100644 --- a/src/ripple/core/ConfigSections.h +++ b/src/ripple/core/ConfigSections.h @@ -101,6 +101,7 @@ struct ConfigSection #define SECTION_SWEEP_INTERVAL "sweep_interval" #define SECTION_NETWORK_ID "network_id" #define SECTION_IMPORT_VL_KEYS "import_vl_keys" +#define SECTION_DATAGRAM_MONITOR "datagram_monitor" } // namespace ripple diff --git a/src/ripple/core/impl/Config.cpp b/src/ripple/core/impl/Config.cpp index 37315d8f22..6569937529 100644 --- a/src/ripple/core/impl/Config.cpp +++ b/src/ripple/core/impl/Config.cpp @@ -281,6 +281,9 @@ Config::setupControl(bool bQuiet, bool bSilent, bool bStandalone) // RAM and CPU resources. We default to "tiny" for standalone mode. if (!bStandalone) { + NODE_SIZE = 4; + return; + // First, check against 'minimum' RAM requirements per node size: auto const& threshold = sizedItems[std::underlying_type_t(SizedItem::ramSizeGB)]; @@ -465,26 +468,24 @@ Config::loadFromString(std::string const& fileContents) SNTP_SERVERS = *s; // if the user has specified ip:port then replace : with a space. - { - auto replaceColons = [](std::vector& strVec) { - const static std::regex e(":([0-9]+)$"); - for (auto& line : strVec) - { - // skip anything that might be an ipv6 address - if (std::count(line.begin(), line.end(), ':') != 1) - continue; - - std::string result = std::regex_replace(line, e, " $1"); - // sanity check the result of the replace, should be same length - // as input - if (result.size() == line.size()) - line = result; - } - }; + auto replaceColons = [](std::vector& strVec) { + const static std::regex e(":([0-9]+)$"); + for (auto& line : strVec) + { + // skip anything that might be an ipv6 address + if (std::count(line.begin(), line.end(), ':') != 1) + continue; + + std::string result = std::regex_replace(line, e, " $1"); + // sanity check the result of the replace, should be same length + // as input + if (result.size() == line.size()) + line = result; + } + }; - replaceColons(IPS_FIXED); - replaceColons(IPS); - } + replaceColons(IPS_FIXED); + replaceColons(IPS); { std::string dbPath; @@ -509,6 +510,13 @@ Config::loadFromString(std::string const& fileContents) NETWORK_ID = beast::lexicalCastThrow(strTemp); } + if (getSingleSection(secConfig, SECTION_DATAGRAM_MONITOR, strTemp, j_)) + { + std::vector vecTemp{strTemp}; + replaceColons(vecTemp); + DATAGRAM_MONITOR = vecTemp[0]; + } + if (getSingleSection(secConfig, SECTION_PEER_PRIVATE, strTemp, j_)) PEER_PRIVATE = beast::lexicalCastThrow(strTemp); From 532a471a356a0ea329adca39ec9f30fe5f740233 Mon Sep 17 00:00:00 2001 From: Richard Holland Date: Wed, 27 Nov 2024 10:16:22 +1100 Subject: [PATCH 07/17] fixReduceImport (#398) Co-authored-by: Denis Angell --- src/ripple/app/tx/impl/Import.cpp | 39 +++++ src/ripple/protocol/Feature.h | 3 +- src/ripple/protocol/TER.h | 1 + src/ripple/protocol/impl/Feature.cpp | 1 + src/ripple/protocol/impl/TER.cpp | 1 + src/test/app/Import_test.cpp | 216 ++++++++++++++++++++++++++- 6 files changed, 252 insertions(+), 9 deletions(-) diff --git a/src/ripple/app/tx/impl/Import.cpp b/src/ripple/app/tx/impl/Import.cpp index d40a80eba6..335cbe5817 100644 --- a/src/ripple/app/tx/impl/Import.cpp +++ b/src/ripple/app/tx/impl/Import.cpp @@ -889,6 +889,45 @@ Import::preclaim(PreclaimContext const& ctx) } auto const& sle = ctx.view.read(keylet::account(ctx.tx[sfAccount])); + + auto const tt = stpTrans->getTxnType(); + if ((tt == ttSIGNER_LIST_SET || tt == ttREGULAR_KEY_SET) && + ctx.view.rules().enabled(fixReduceImport) && sle) + { + // blackhole check + do + { + // if master key is not set then it is not blackholed + if (!(sle->getFlags() & lsfDisableMaster)) + break; + + // if a regular key is set then it must be acc 0, 1, or 2 otherwise + // not blackholed + if (sle->isFieldPresent(sfRegularKey)) + { + AccountID rk = sle->getAccountID(sfRegularKey); + static const AccountID ACCOUNT_ZERO(0); + static const AccountID ACCOUNT_ONE(1); + static const AccountID ACCOUNT_TWO(2); + + if (rk != ACCOUNT_ZERO && rk != ACCOUNT_ONE && + rk != ACCOUNT_TWO) + break; + } + + // if a signer list is set then it's not blackholed + auto const signerListKeylet = keylet::signers(ctx.tx[sfAccount]); + if (ctx.view.exists(signerListKeylet)) + break; + + // execution to here means it's blackholed + JLOG(ctx.j.warn()) + << "Import: during preclaim target account is blackholed " + << ctx.tx[sfAccount] << ", bailing."; + return tefIMPORT_BLACKHOLED; + } while (0); + } + if (sle && sle->isFieldPresent(sfImportSequence)) { uint32_t sleImportSequence = sle->getFieldU32(sfImportSequence); diff --git a/src/ripple/protocol/Feature.h b/src/ripple/protocol/Feature.h index 43d510c636..715f5dac6e 100644 --- a/src/ripple/protocol/Feature.h +++ b/src/ripple/protocol/Feature.h @@ -74,7 +74,7 @@ namespace detail { // Feature.cpp. Because it's only used to reserve storage, and determine how // large to make the FeatureBitset, it MAY be larger. It MUST NOT be less than // the actual number of amendments. A LogicError on startup will verify this. -static constexpr std::size_t numFeatures = 74; +static constexpr std::size_t numFeatures = 75; /** Amendments that this server supports and the default voting behavior. Whether they are enabled depends on the Rules defined in the validated @@ -362,6 +362,7 @@ extern uint256 const fix240819; extern uint256 const fixPageCap; extern uint256 const fix240911; extern uint256 const fixFloatDivide; +extern uint256 const fixReduceImport; } // namespace ripple diff --git a/src/ripple/protocol/TER.h b/src/ripple/protocol/TER.h index 42d3cabd35..7cd3cae422 100644 --- a/src/ripple/protocol/TER.h +++ b/src/ripple/protocol/TER.h @@ -184,6 +184,7 @@ enum TEFcodes : TERUnderlyingType { tefPAST_IMPORT_SEQ, tefPAST_IMPORT_VL_SEQ, tefNONDIR_EMIT, + tefIMPORT_BLACKHOLED, }; //------------------------------------------------------------------------------ diff --git a/src/ripple/protocol/impl/Feature.cpp b/src/ripple/protocol/impl/Feature.cpp index 23cbe236df..1c7fc931b1 100644 --- a/src/ripple/protocol/impl/Feature.cpp +++ b/src/ripple/protocol/impl/Feature.cpp @@ -468,6 +468,7 @@ REGISTER_FIX (fix240819, Supported::yes, VoteBehavior::De REGISTER_FIX (fixPageCap, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fix240911, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fixFloatDivide, Supported::yes, VoteBehavior::DefaultYes); +REGISTER_FIX (fixReduceImport, Supported::yes, VoteBehavior::DefaultYes); // The following amendments are obsolete, but must remain supported // because they could potentially get enabled. diff --git a/src/ripple/protocol/impl/TER.cpp b/src/ripple/protocol/impl/TER.cpp index be8be1dd42..e41134a0c4 100644 --- a/src/ripple/protocol/impl/TER.cpp +++ b/src/ripple/protocol/impl/TER.cpp @@ -116,6 +116,7 @@ transResults() MAKE_ERROR(tefNO_TICKET, "Ticket is not in ledger."), MAKE_ERROR(tefNFTOKEN_IS_NOT_TRANSFERABLE, "The specified NFToken is not transferable."), MAKE_ERROR(tefNONDIR_EMIT, "An emitted txn was injected into the ledger without a corresponding directory entry."), + MAKE_ERROR(tefIMPORT_BLACKHOLED, "Cannot import keying because target account is blackholed."), MAKE_ERROR(telLOCAL_ERROR, "Local failure."), MAKE_ERROR(telBAD_DOMAIN, "Domain too long."), diff --git a/src/test/app/Import_test.cpp b/src/test/app/Import_test.cpp index c115bcc4d4..09ffd4a023 100644 --- a/src/test/app/Import_test.cpp +++ b/src/test/app/Import_test.cpp @@ -79,7 +79,7 @@ class Import_test : public beast::unit_test::suite importVLSequence(jtx::Env const& env, PublicKey const& pk) { auto const sle = env.le(keylet::import_vlseq(pk)); - if (sle->isFieldPresent(sfImportSequence)) + if (sle && sle->isFieldPresent(sfImportSequence)) return (*sle)[sfImportSequence]; return 0; } @@ -2672,6 +2672,134 @@ class Import_test : public beast::unit_test::suite env(import::import(alice, tmpXpop), ter(temMALFORMED)); } + // tefIMPORT_BLACKHOLED - SetRegularKey (w/seed) AccountZero + { + test::jtx::Env env{ + *this, network::makeNetworkVLConfig(21337, keys)}; + auto const feeDrops = env.current()->fees().base; + + auto const alice = Account("alice"); + env.fund(XRP(1000), alice); + env.close(); + + // Set Regular Key + Json::Value jv; + jv[jss::Account] = alice.human(); + const AccountID ACCOUNT_ZERO(0); + jv["RegularKey"] = to_string(ACCOUNT_ZERO); + jv[jss::TransactionType] = jss::SetRegularKey; + env(jv, alice); + + // Disable Master Key + env(fset(alice, asfDisableMaster), sig(alice)); + env.close(); + + // Import with Master Key + Json::Value tmpXpop = + import::loadXpop(ImportTCSetRegularKey::w_seed); + env(import::import(alice, tmpXpop), + ter(tefIMPORT_BLACKHOLED), + fee(feeDrops * 10), + sig(alice)); + env.close(); + } + + // tefIMPORT_BLACKHOLED - SetRegularKey (w/seed) AccountOne + { + test::jtx::Env env{ + *this, network::makeNetworkVLConfig(21337, keys)}; + auto const feeDrops = env.current()->fees().base; + + auto const alice = Account("alice"); + env.fund(XRP(1000), alice); + env.close(); + + // Set Regular Key + Json::Value jv; + jv[jss::Account] = alice.human(); + const AccountID ACCOUNT_ONE(1); + jv["RegularKey"] = to_string(ACCOUNT_ONE); + jv[jss::TransactionType] = jss::SetRegularKey; + env(jv, alice); + + // Disable Master Key + env(fset(alice, asfDisableMaster), sig(alice)); + env.close(); + + // Import with Master Key + Json::Value tmpXpop = + import::loadXpop(ImportTCSetRegularKey::w_seed); + env(import::import(alice, tmpXpop), + ter(tefIMPORT_BLACKHOLED), + fee(feeDrops * 10), + sig(alice)); + env.close(); + } + + // tefIMPORT_BLACKHOLED - SetRegularKey (w/seed) AccountTwo + { + test::jtx::Env env{ + *this, network::makeNetworkVLConfig(21337, keys)}; + auto const feeDrops = env.current()->fees().base; + + auto const alice = Account("alice"); + env.fund(XRP(1000), alice); + env.close(); + + // Set Regular Key + Json::Value jv; + jv[jss::Account] = alice.human(); + const AccountID ACCOUNT_TWO(2); + jv["RegularKey"] = to_string(ACCOUNT_TWO); + jv[jss::TransactionType] = jss::SetRegularKey; + env(jv, alice); + + // Disable Master Key + env(fset(alice, asfDisableMaster), sig(alice)); + env.close(); + + // Import with Master Key + Json::Value tmpXpop = + import::loadXpop(ImportTCSetRegularKey::w_seed); + env(import::import(alice, tmpXpop), + ter(tefIMPORT_BLACKHOLED), + fee(feeDrops * 10), + sig(alice)); + env.close(); + } + + // tefIMPORT_BLACKHOLED - SignersListSet (w/seed) + { + test::jtx::Env env{ + *this, network::makeNetworkVLConfig(21337, keys)}; + auto const feeDrops = env.current()->fees().base; + + auto const alice = Account("alice"); + env.fund(XRP(1000), alice); + env.close(); + + // Set Regular Key + Json::Value jv; + jv[jss::Account] = alice.human(); + const AccountID ACCOUNT_ZERO(0); + jv["RegularKey"] = to_string(ACCOUNT_ZERO); + jv[jss::TransactionType] = jss::SetRegularKey; + env(jv, alice); + + // Disable Master Key + env(fset(alice, asfDisableMaster), sig(alice)); + env.close(); + + // Import with Master Key + Json::Value tmpXpop = + import::loadXpop(ImportTCSignersListSet::w_seed); + env(import::import(alice, tmpXpop), + ter(tefIMPORT_BLACKHOLED), + fee(feeDrops * 10), + sig(alice)); + env.close(); + } + // tefPAST_IMPORT_SEQ { test::jtx::Env env{ @@ -4580,14 +4708,22 @@ class Import_test : public beast::unit_test::suite // confirm signers set auto const [signers, signersSle] = signersKeyAndSle(*env.current(), alice); - auto const signerEntries = - signersSle->getFieldArray(sfSignerEntries); - BEAST_EXPECT(signerEntries.size() == 2); - BEAST_EXPECT(signerEntries[0u].getFieldU16(sfSignerWeight) == 1); BEAST_EXPECT( - signerEntries[0u].getAccountID(sfAccount) == carol.id()); - BEAST_EXPECT(signerEntries[1u].getFieldU16(sfSignerWeight) == 1); - BEAST_EXPECT(signerEntries[1u].getAccountID(sfAccount) == bob.id()); + signersSle && signersSle->isFieldPresent(sfSignerEntries)); + if (signersSle && signersSle->isFieldPresent(sfSignerEntries)) + { + auto const signerEntries = + signersSle->getFieldArray(sfSignerEntries); + BEAST_EXPECT(signerEntries.size() == 2); + BEAST_EXPECT( + signerEntries[0u].getFieldU16(sfSignerWeight) == 1); + BEAST_EXPECT( + signerEntries[0u].getAccountID(sfAccount) == carol.id()); + BEAST_EXPECT( + signerEntries[1u].getFieldU16(sfSignerWeight) == 1); + BEAST_EXPECT( + signerEntries[1u].getAccountID(sfAccount) == bob.id()); + } // confirm multisign tx env.close(); @@ -5986,6 +6122,69 @@ class Import_test : public beast::unit_test::suite } } + void + testBlackhole(FeatureBitset features) + { + testcase("blackhole"); + + using namespace test::jtx; + using namespace std::literals; + + auto blackholeAccount = [&](Env& env, Account const& acct) { + // Set Regular Key + Json::Value jv; + jv[jss::Account] = acct.human(); + const AccountID ACCOUNT_ZERO(0); + jv["RegularKey"] = to_string(ACCOUNT_ZERO); + jv[jss::TransactionType] = jss::SetRegularKey; + env(jv, acct); + + // Disable Master Key + env(fset(acct, asfDisableMaster), sig(acct)); + env.close(); + }; + + auto burnHeader = [&](Env& env) { + // confirm total coins header + auto const initCoins = env.current()->info().drops; + BEAST_EXPECT(initCoins == 100'000'000'000'000'000); + + // burn 10'000 xrp + auto const master = Account("masterpassphrase"); + env(noop(master), fee(100'000'000'000'000), ter(tesSUCCESS)); + env.close(); + + // confirm total coins header + auto const burnCoins = env.current()->info().drops; + BEAST_EXPECT(burnCoins == initCoins - 100'000'000'000'000); + }; + + // AccountSet (w/seed) + { + test::jtx::Env env{ + *this, network::makeNetworkVLConfig(21337, keys)}; + auto const feeDrops = env.current()->fees().base; + + // Burn Header + burnHeader(env); + + auto const alice = Account("alice"); + env.fund(XRP(1000), alice); + env.close(); + + // Blackhole Account + blackholeAccount(env, alice); + + // Import with Master Key + Json::Value tmpXpop = import::loadXpop(ImportTCAccountSet::w_seed); + env(import::import(alice, tmpXpop), + ter(tesSUCCESS), + fee(feeDrops * 10), + sig(alice)); + env.close(); + } + } + public: void run() override @@ -6026,6 +6225,7 @@ class Import_test : public beast::unit_test::suite testMaxSupply(features); testMinMax(features); testHalving(features - featureOwnerPaysFee); + testBlackhole(features); } }; From d878fd4a6e2db2f6e183b417da825428a5d719c6 Mon Sep 17 00:00:00 2001 From: RichardAH Date: Sat, 14 Dec 2024 08:44:40 +1000 Subject: [PATCH 08/17] allow multiple datagram monitor endpoints (#408) --- src/ripple/app/main/Application.cpp | 2 +- src/ripple/app/misc/DatagramMonitor.h | 20 ++++++++++++++++---- src/ripple/core/Config.h | 2 +- src/ripple/core/impl/Config.cpp | 7 +++---- 4 files changed, 21 insertions(+), 10 deletions(-) diff --git a/src/ripple/app/main/Application.cpp b/src/ripple/app/main/Application.cpp index be0a7b46a6..20427107b1 100644 --- a/src/ripple/app/main/Application.cpp +++ b/src/ripple/app/main/Application.cpp @@ -1527,7 +1527,7 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline) reportingETL_->start(); // Datagram monitor if applicable - if (!config_->standalone() && config_->DATAGRAM_MONITOR != "") + if (!config_->standalone() && !config_->DATAGRAM_MONITOR.empty()) { datagram_monitor_ = std::make_unique(*this); if (datagram_monitor_) diff --git a/src/ripple/app/misc/DatagramMonitor.h b/src/ripple/app/misc/DatagramMonitor.h index ba6ce02133..0330905813 100644 --- a/src/ripple/app/misc/DatagramMonitor.h +++ b/src/ripple/app/misc/DatagramMonitor.h @@ -996,15 +996,24 @@ class DatagramMonitor void monitorThread() { - auto endpoint = parseEndpoint(app_.config().DATAGRAM_MONITOR); - int sock = createSocket(endpoint); + std::vector> endpoints; + + for (auto const& epStr : app_.config().DATAGRAM_MONITOR) + { + auto endpoint = parseEndpoint(epStr); + endpoints.push_back( + std::make_pair(endpoint, createSocket(endpoint))); + } while (running_) { try { auto info = generateServerInfo(); - sendPacket(sock, endpoint, info); + for (auto const& ep : endpoints) + { + sendPacket(ep.second, ep.first, info); + } std::this_thread::sleep_for(std::chrono::seconds(1)); } catch (const std::exception& e) @@ -1015,7 +1024,10 @@ class DatagramMonitor } } - close(sock); + for (auto const& ep : endpoints) + { + close(ep.second); + } } public: diff --git a/src/ripple/core/Config.h b/src/ripple/core/Config.h index 5d9977770c..3e2c3c81a8 100644 --- a/src/ripple/core/Config.h +++ b/src/ripple/core/Config.h @@ -155,7 +155,7 @@ class Config : public BasicConfig std::map IMPORT_VL_KEYS; // hex string -> class PublicKey (for caching purposes) - std::string DATAGRAM_MONITOR; + std::vector DATAGRAM_MONITOR; enum StartUpType { FRESH, diff --git a/src/ripple/core/impl/Config.cpp b/src/ripple/core/impl/Config.cpp index 6569937529..7673d16ecd 100644 --- a/src/ripple/core/impl/Config.cpp +++ b/src/ripple/core/impl/Config.cpp @@ -510,11 +510,10 @@ Config::loadFromString(std::string const& fileContents) NETWORK_ID = beast::lexicalCastThrow(strTemp); } - if (getSingleSection(secConfig, SECTION_DATAGRAM_MONITOR, strTemp, j_)) + if (auto s = getIniFileSection(secConfig, SECTION_DATAGRAM_MONITOR)) { - std::vector vecTemp{strTemp}; - replaceColons(vecTemp); - DATAGRAM_MONITOR = vecTemp[0]; + DATAGRAM_MONITOR = *s; + replaceColons(DATAGRAM_MONITOR); } if (getSingleSection(secConfig, SECTION_PEER_PRIVATE, strTemp, j_)) From 85a752235a2e4870fd51cd8dafdefaf9cda78e05 Mon Sep 17 00:00:00 2001 From: tequ Date: Mon, 16 Dec 2024 15:10:01 +0900 Subject: [PATCH 09/17] add URITokenIssuer to account_flags for account_info (#404) --- src/ripple/rpc/handlers/AccountInfo.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/ripple/rpc/handlers/AccountInfo.cpp b/src/ripple/rpc/handlers/AccountInfo.cpp index b71fd48654..e811baf582 100644 --- a/src/ripple/rpc/handlers/AccountInfo.cpp +++ b/src/ripple/rpc/handlers/AccountInfo.cpp @@ -75,7 +75,7 @@ doAccountInfo(RPC::JsonContext& context) auto const accountID{std::move(id.value())}; static constexpr std:: - array, 10> + array, 11> lsFlags{ {{"defaultRipple", lsfDefaultRipple}, {"depositAuth", lsfDepositAuth}, @@ -86,7 +86,8 @@ doAccountInfo(RPC::JsonContext& context) {"passwordSpent", lsfPasswordSpent}, {"requireAuthorization", lsfRequireAuth}, {"tshCollect", lsfTshCollect}, - {"requireDestinationTag", lsfRequireDestTag}}}; + {"requireDestinationTag", lsfRequireDestTag}, + {"uriTokenIssuer", lsfURITokenIssuer}}}; static constexpr std:: array, 5> From 621ca9c86546df7267b38dc5dff0603e02090b5a Mon Sep 17 00:00:00 2001 From: tequ Date: Wed, 22 Jan 2025 07:34:33 +0900 Subject: [PATCH 10/17] Add space to `trace_float` log (#424) --- src/ripple/app/hook/impl/applyHook.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/ripple/app/hook/impl/applyHook.cpp b/src/ripple/app/hook/impl/applyHook.cpp index abd7ef1361..0507aca97f 100644 --- a/src/ripple/app/hook/impl/applyHook.cpp +++ b/src/ripple/app/hook/impl/applyHook.cpp @@ -4790,7 +4790,7 @@ DEFINE_HOOK_FUNCTION( if (float1 == 0) { - j.trace() << "HookTrace[" << HC_ACC() << "]:" + j.trace() << "HookTrace[" << HC_ACC() << "]: " << (read_len == 0 ? "" : std::string_view( From 446617523112cd773b01b20209289a8008108b99 Mon Sep 17 00:00:00 2001 From: tequ Date: Wed, 22 Jan 2025 07:38:12 +0900 Subject: [PATCH 11/17] Update boost link for build-full.sh (#421) --- build-full.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/build-full.sh b/build-full.sh index 4ea18fea09..3ae0251d75 100755 --- a/build-full.sh +++ b/build-full.sh @@ -92,7 +92,7 @@ pwd && tar -xzf cmake-3.23.1-linux-x86_64.tar.gz -C /hbb/ && echo "-- Install Boost 1.86.0 --" && pwd && -( wget -nc -q https://boostorg.jfrog.io/artifactory/main/release/1.86.0/source/boost_1_86_0.tar.gz; echo "" ) && +( wget -nc -q https://archives.boost.io/release/1.86.0/source/boost_1_86_0.tar.gz; echo "" ) && tar -xzf boost_1_86_0.tar.gz && cd boost_1_86_0 && ./bootstrap.sh && ./b2 link=static -j$3 && ./b2 install && cd ../ && From d17f7151ab35e4c7b7fbb18371a427f62b416d74 Mon Sep 17 00:00:00 2001 From: tequ Date: Wed, 22 Jan 2025 12:33:59 +0900 Subject: [PATCH 12/17] Fix HookResult(ExitType) when accept() is not called (#415) --- src/ripple/app/hook/impl/applyHook.cpp | 7 ++++--- src/ripple/app/tx/impl/Transactor.cpp | 24 ++++++++++++++++-------- src/ripple/protocol/Feature.h | 3 ++- src/ripple/protocol/impl/Feature.cpp | 1 + 4 files changed, 23 insertions(+), 12 deletions(-) diff --git a/src/ripple/app/hook/impl/applyHook.cpp b/src/ripple/app/hook/impl/applyHook.cpp index 0507aca97f..aeb9f3f503 100644 --- a/src/ripple/app/hook/impl/applyHook.cpp +++ b/src/ripple/app/hook/impl/applyHook.cpp @@ -1217,9 +1217,10 @@ hook::apply( .hookParamOverrides = hookParamOverrides, .hookParams = hookParams, .hookSkips = {}, - .exitType = - hook_api::ExitType::ROLLBACK, // default is to rollback unless - // hook calls accept() + .exitType = applyCtx.view().rules().enabled(fixXahauV3) + ? hook_api::ExitType::UNSET + : hook_api::ExitType::ROLLBACK, // default is to rollback + // unless hook calls accept() .exitReason = std::string(""), .exitCode = -1, .hasCallback = hasCallback, diff --git a/src/ripple/app/tx/impl/Transactor.cpp b/src/ripple/app/tx/impl/Transactor.cpp index 7c2734f208..180bf64a84 100644 --- a/src/ripple/app/tx/impl/Transactor.cpp +++ b/src/ripple/app/tx/impl/Transactor.cpp @@ -1270,10 +1270,18 @@ Transactor::executeHookChain( if (results.back().exitType == hook_api::ExitType::WASM_ERROR) { JLOG(j_.warn()) << "HookError[" << account << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " + << ctx_.tx.getAccountID(sfAccount) << "]: Execution failure (graceful) " << "HookHash: " << hookHash; } + if (results.back().exitType == hook_api::ExitType::UNSET) + { + JLOG(j_.warn()) + << "HookError[" << account << "-" + << ctx_.tx.getAccountID(sfAccount) + << "]: Execution failure (no exit type specified) " + << "HookHash: " << hookHash; + } return tecHOOK_REJECTED; } @@ -1298,7 +1306,7 @@ Transactor::executeHookChain( { JLOG(j_.warn()) << "HookError[" << account << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " + << ctx_.tx.getAccountID(sfAccount) << "]: Execution failure (exceptional) " << "Exception: " << e.what() << " HookHash: " << hookHash; @@ -1426,13 +1434,13 @@ Transactor::doHookCallback( finalizeHookResult(callbackResult, ctx_, success); JLOG(j_.trace()) << "HookInfo[" << callbackAccountID << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " - << "Callback finalizeHookResult = " << result; + << ctx_.tx.getAccountID(sfAccount) + << "]: Callback finalizeHookResult = " << result; } catch (std::exception& e) { JLOG(j_.fatal()) << "HookError[" << callbackAccountID << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " + << ctx_.tx.getAccountID(sfAccount) << "]: Callback failure " << e.what(); } } @@ -1678,13 +1686,13 @@ Transactor::doAgainAsWeak( results.push_back(aawResult); JLOG(j_.trace()) << "HookInfo[" << hookAccountID << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " - << " aaw Hook ExitCode = " << aawResult.exitCode; + << ctx_.tx.getAccountID(sfAccount) + << "]: aaw Hook ExitCode = " << aawResult.exitCode; } catch (std::exception& e) { JLOG(j_.fatal()) << "HookError[" << hookAccountID << "-" - << ctx_.tx.getAccountID(sfAccount) << "]: " + << ctx_.tx.getAccountID(sfAccount) << "]: aaw failure " << e.what(); } } diff --git a/src/ripple/protocol/Feature.h b/src/ripple/protocol/Feature.h index 715f5dac6e..f479ecba78 100644 --- a/src/ripple/protocol/Feature.h +++ b/src/ripple/protocol/Feature.h @@ -74,7 +74,7 @@ namespace detail { // Feature.cpp. Because it's only used to reserve storage, and determine how // large to make the FeatureBitset, it MAY be larger. It MUST NOT be less than // the actual number of amendments. A LogicError on startup will verify this. -static constexpr std::size_t numFeatures = 75; +static constexpr std::size_t numFeatures = 76; /** Amendments that this server supports and the default voting behavior. Whether they are enabled depends on the Rules defined in the validated @@ -363,6 +363,7 @@ extern uint256 const fixPageCap; extern uint256 const fix240911; extern uint256 const fixFloatDivide; extern uint256 const fixReduceImport; +extern uint256 const fixXahauV3; } // namespace ripple diff --git a/src/ripple/protocol/impl/Feature.cpp b/src/ripple/protocol/impl/Feature.cpp index 1c7fc931b1..12c7b66c84 100644 --- a/src/ripple/protocol/impl/Feature.cpp +++ b/src/ripple/protocol/impl/Feature.cpp @@ -469,6 +469,7 @@ REGISTER_FIX (fixPageCap, Supported::yes, VoteBehavior::De REGISTER_FIX (fix240911, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fixFloatDivide, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fixReduceImport, Supported::yes, VoteBehavior::DefaultYes); +REGISTER_FIX (fixXahauV3, Supported::yes, VoteBehavior::DefaultNo); // The following amendments are obsolete, but must remain supported // because they could potentially get enabled. From 12d8342c34d9500ef60cb56aee5873a0a4401178 Mon Sep 17 00:00:00 2001 From: Wietse Wind Date: Sat, 1 Feb 2025 08:57:25 +0100 Subject: [PATCH 13/17] Update artifact --- .github/workflows/levelization.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/workflows/levelization.yml b/.github/workflows/levelization.yml index c8284c5fb1..1295bd3936 100644 --- a/.github/workflows/levelization.yml +++ b/.github/workflows/levelization.yml @@ -18,8 +18,7 @@ jobs: git diff --exit-code | tee "levelization.patch" - name: Upload patch if: failure() && steps.assert.outcome == 'failure' - uses: actions/upload-artifact@v3 - continue-on-error: true + uses: actions/upload-artifact@v4 with: name: levelization.patch if-no-files-found: ignore From 412593d7bc178957875b9e71435b584bbadb5f36 Mon Sep 17 00:00:00 2001 From: Wietse Wind Date: Sat, 1 Feb 2025 08:57:48 +0100 Subject: [PATCH 14/17] Update artifact --- .github/workflows/clang-format.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/clang-format.yml b/.github/workflows/clang-format.yml index 0cc8dbe59a..52432addab 100644 --- a/.github/workflows/clang-format.yml +++ b/.github/workflows/clang-format.yml @@ -30,7 +30,7 @@ jobs: git diff --exit-code | tee "clang-format.patch" - name: Upload patch if: failure() && steps.assert.outcome == 'failure' - uses: actions/upload-artifact@v3 + uses: actions/upload-artifact@v4 continue-on-error: true with: name: clang-format.patch From fa71bda29c0ab1f5d8a0b76734ce886e325445a6 Mon Sep 17 00:00:00 2001 From: Wietse Wind Date: Sat, 1 Feb 2025 08:58:13 +0100 Subject: [PATCH 15/17] Artifact v4 continue on error --- .github/workflows/levelization.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/levelization.yml b/.github/workflows/levelization.yml index 1295bd3936..f99c1ca568 100644 --- a/.github/workflows/levelization.yml +++ b/.github/workflows/levelization.yml @@ -19,6 +19,7 @@ jobs: - name: Upload patch if: failure() && steps.assert.outcome == 'failure' uses: actions/upload-artifact@v4 + continue-on-error: true with: name: levelization.patch if-no-files-found: ignore From 2fd465bb3fbfaa821cd80d9f1bf4c4bd9f45ac6f Mon Sep 17 00:00:00 2001 From: RichardAH Date: Mon, 3 Feb 2025 10:33:19 +1000 Subject: [PATCH 16/17] fix20250131 (#428) Co-authored-by: Denis Angell --- src/ripple/app/hook/Guard.h | 33 ++++++++++++++++-- src/ripple/app/hook/guard_checker.cpp | 2 +- src/ripple/app/tx/impl/Change.cpp | 3 +- src/ripple/app/tx/impl/Remit.cpp | 10 ++++++ src/ripple/app/tx/impl/SetHook.cpp | 3 +- src/ripple/protocol/Feature.h | 3 +- src/ripple/protocol/impl/Feature.cpp | 3 +- src/ripple/protocol/impl/TxMeta.cpp | 4 ++- src/test/app/SetHookTSH_test.cpp | 1 - src/test/app/SetHook_test.cpp | 50 +++++++++++++++++++++++++++ 10 files changed, 103 insertions(+), 9 deletions(-) diff --git a/src/ripple/app/hook/Guard.h b/src/ripple/app/hook/Guard.h index 893fe9282d..f395af4481 100644 --- a/src/ripple/app/hook/Guard.h +++ b/src/ripple/app/hook/Guard.h @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -271,7 +272,19 @@ check_guard( int guard_func_idx, int last_import_idx, GuardLog guardLog, - std::string guardLogAccStr) + std::string guardLogAccStr, + /* RH NOTE: + * rules version is a bit field, so rule update 1 is 0x01, update 2 is 0x02 + * and update 3 is 0x04 ideally at rule version 3 all bits so far are set + * (0b111) so the ruleVersion = 7, however if a specific rule update must be + * rolled back due to unforeseen behaviour then this may no longer be the + * case. using a bit field here leaves us flexible to rollback changes that + * might have unforeseen consequences, without also rolling back further + * changes that are fine. + */ + uint64_t rulesVersion = 0 + +) { #define MAX_GUARD_CALLS 1024 uint32_t guard_count = 0; @@ -621,11 +634,17 @@ check_guard( } else if (fc_type == 10) // memory.copy { + if (rulesVersion & 0x02U) + GUARD_ERROR("Memory.copy instruction is not allowed."); + REQUIRE(2); ADVANCE(2); } else if (fc_type == 11) // memory.fill { + if (rulesVersion & 0x02U) + GUARD_ERROR("Memory.fill instruction is not allowed."); + ADVANCE(1); } else if (fc_type <= 7) // numeric instructions @@ -807,6 +826,15 @@ validateGuards( std::vector const& wasm, GuardLog guardLog, std::string guardLogAccStr, + /* RH NOTE: + * rules version is a bit field, so rule update 1 is 0x01, update 2 is 0x02 + * and update 3 is 0x04 ideally at rule version 3 all bits so far are set + * (0b111) so the ruleVersion = 7, however if a specific rule update must be + * rolled back due to unforeseen behaviour then this may no longer be the + * case. using a bit field here leaves us flexible to rollback changes that + * might have unforeseen consequences, without also rolling back further + * changes that are fine. + */ uint64_t rulesVersion = 0) { uint64_t byteCount = wasm.size(); @@ -1477,7 +1505,8 @@ validateGuards( guard_import_number, last_import_number, guardLog, - guardLogAccStr); + guardLogAccStr, + rulesVersion); if (!valid) return {}; diff --git a/src/ripple/app/hook/guard_checker.cpp b/src/ripple/app/hook/guard_checker.cpp index 634dd8a93f..f20d24617b 100644 --- a/src/ripple/app/hook/guard_checker.cpp +++ b/src/ripple/app/hook/guard_checker.cpp @@ -79,7 +79,7 @@ main(int argc, char** argv) close(fd); - auto result = validateGuards(hook, std::cout, "", 1); + auto result = validateGuards(hook, std::cout, "", 3); if (!result) { diff --git a/src/ripple/app/tx/impl/Change.cpp b/src/ripple/app/tx/impl/Change.cpp index c91b794039..61134ca251 100644 --- a/src/ripple/app/tx/impl/Change.cpp +++ b/src/ripple/app/tx/impl/Change.cpp @@ -587,7 +587,8 @@ Change::activateXahauGenesis() wasmBytes, // wasm to verify loggerStream, "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", - ctx_.view().rules().enabled(featureHooksUpdate1) ? 1 : 0); + (ctx_.view().rules().enabled(featureHooksUpdate1) ? 1 : 0) + + (ctx_.view().rules().enabled(fix20250131) ? 2 : 0)); if (!result) { diff --git a/src/ripple/app/tx/impl/Remit.cpp b/src/ripple/app/tx/impl/Remit.cpp index 084513e025..33d8a4c6f7 100644 --- a/src/ripple/app/tx/impl/Remit.cpp +++ b/src/ripple/app/tx/impl/Remit.cpp @@ -72,6 +72,16 @@ Remit::preflight(PreflightContext const& ctx) return temREDUNDANT; } + if (ctx.rules.enabled(fix20250131)) + { + if (!dstID || dstID == noAccount()) + { + JLOG(ctx.j.warn()) + << "Malformed transaction: Remit to invalid account."; + return temMALFORMED; + } + } + if (ctx.tx.isFieldPresent(sfInform)) { AccountID const infID = ctx.tx.getAccountID(sfInform); diff --git a/src/ripple/app/tx/impl/SetHook.cpp b/src/ripple/app/tx/impl/SetHook.cpp index aac02753c3..67e89d9931 100644 --- a/src/ripple/app/tx/impl/SetHook.cpp +++ b/src/ripple/app/tx/impl/SetHook.cpp @@ -479,7 +479,8 @@ SetHook::validateHookSetEntry(SetHookCtx& ctx, STObject const& hookSetObj) hook, // wasm to verify logger, hsacc, - ctx.rules.enabled(featureHooksUpdate1) ? 1 : 0); + (ctx.rules.enabled(featureHooksUpdate1) ? 1 : 0) + + (ctx.rules.enabled(fix20250131) ? 2 : 0)); if (ctx.j.trace()) { diff --git a/src/ripple/protocol/Feature.h b/src/ripple/protocol/Feature.h index f479ecba78..b242b2f7f3 100644 --- a/src/ripple/protocol/Feature.h +++ b/src/ripple/protocol/Feature.h @@ -74,7 +74,7 @@ namespace detail { // Feature.cpp. Because it's only used to reserve storage, and determine how // large to make the FeatureBitset, it MAY be larger. It MUST NOT be less than // the actual number of amendments. A LogicError on startup will verify this. -static constexpr std::size_t numFeatures = 76; +static constexpr std::size_t numFeatures = 77; /** Amendments that this server supports and the default voting behavior. Whether they are enabled depends on the Rules defined in the validated @@ -364,6 +364,7 @@ extern uint256 const fix240911; extern uint256 const fixFloatDivide; extern uint256 const fixReduceImport; extern uint256 const fixXahauV3; +extern uint256 const fix20250131; } // namespace ripple diff --git a/src/ripple/protocol/impl/Feature.cpp b/src/ripple/protocol/impl/Feature.cpp index 12c7b66c84..73db671edb 100644 --- a/src/ripple/protocol/impl/Feature.cpp +++ b/src/ripple/protocol/impl/Feature.cpp @@ -469,7 +469,8 @@ REGISTER_FIX (fixPageCap, Supported::yes, VoteBehavior::De REGISTER_FIX (fix240911, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fixFloatDivide, Supported::yes, VoteBehavior::DefaultYes); REGISTER_FIX (fixReduceImport, Supported::yes, VoteBehavior::DefaultYes); -REGISTER_FIX (fixXahauV3, Supported::yes, VoteBehavior::DefaultNo); +REGISTER_FIX (fixXahauV3, Supported::yes, VoteBehavior::DefaultYes); +REGISTER_FIX (fix20250131, Supported::yes, VoteBehavior::DefaultYes); // The following amendments are obsolete, but must remain supported // because they could potentially get enabled. diff --git a/src/ripple/protocol/impl/TxMeta.cpp b/src/ripple/protocol/impl/TxMeta.cpp index 4b48f5eb4a..506c7f2a73 100644 --- a/src/ripple/protocol/impl/TxMeta.cpp +++ b/src/ripple/protocol/impl/TxMeta.cpp @@ -240,7 +240,9 @@ TxMeta::addRaw(Serializer& s, TER result, std::uint32_t index) { mResult = TERtoInt(result); mIndex = index; - assert((mResult == 0) || ((mResult > 100) && (mResult <= 255))); + assert( + (mResult == 0 || mResult == 1) || + ((mResult > 100) && (mResult <= 255))); mNodes.sort([](STObject const& o1, STObject const& o2) { return o1.getFieldH256(sfLedgerIndex) < o2.getFieldH256(sfLedgerIndex); diff --git a/src/test/app/SetHookTSH_test.cpp b/src/test/app/SetHookTSH_test.cpp index 30a561113b..9b19c39369 100644 --- a/src/test/app/SetHookTSH_test.cpp +++ b/src/test/app/SetHookTSH_test.cpp @@ -5544,7 +5544,6 @@ struct SetHookTSH_test : public beast::unit_test::suite testTSH(sa - fixXahauV1 - fixXahauV2); testTSH(sa - fixXahauV2); testTSH(sa); - testEmittedTxn(sa - fixXahauV2); testEmittedTxn(sa); } }; diff --git a/src/test/app/SetHook_test.cpp b/src/test/app/SetHook_test.cpp index f0e97a8594..7f42e97005 100644 --- a/src/test/app/SetHook_test.cpp +++ b/src/test/app/SetHook_test.cpp @@ -1138,6 +1138,54 @@ class SetHook_test : public beast::unit_test::suite : preHookCount + 66); } + void + testFillCopy(FeatureBitset features) + { + testcase("Test fill/copy"); + + // a hook containing memory.fill instruction + std::string hookFill = + "0061736d0100000001130360027f7f017f60037f7f7e017e60017f017e02" + "170203656e76025f67000003656e76066163636570740001030201020503" + "0100020621057f01418088040b7f004180080b7f004180080b7f00418088" + "040b7f004180080b07080104686f6f6b00020aa4800001a0800001017e23" + "01412a41e400fc0b004101410110001a41004100420010011a20010b"; + + // a hook containing memory.copy instruction + std::string hookCopy = + "0061736d0100000001130360027f7f017f60037f7f7e017e60017f017e02" + "170203656e76025f67000003656e76066163636570740001030201020503" + "0100020621057f01418088040b7f004180080b7f004180080b7f00418088" + "040b7f004180080b07080104686f6f6b00020aa5800001a1800001017e23" + "00230141e400fc0a00004101410110001a41004100420010011a20010b"; + + using namespace jtx; + + for (int withFix = 0; withFix < 2; ++withFix) + { + auto f = withFix ? features : features - fix20250131; + Env env{*this, f}; + + auto const alice = Account{"alice"}; + env.fund(XRP(10000), alice); + + auto const bob = Account{"bob"}; + env.fund(XRP(10000), bob); + + env(ripple::test::jtx::hook(alice, {{hso(hookFill)}}, 0), + M(withFix ? "hookFill - with fix" : "hookFill - zonder fix"), + HSFEE, + withFix ? ter(temMALFORMED) : ter(tesSUCCESS)); + + env(ripple::test::jtx::hook(bob, {{hso(hookCopy)}}, 0), + M(withFix ? "hookCopy - with fix" : "hookCopy - zonder fix"), + HSFEE, + withFix ? ter(temMALFORMED) : ter(tesSUCCESS)); + + env.close(); + } + } + void testCreate(FeatureBitset features) { @@ -11973,6 +12021,8 @@ class SetHook_test : public beast::unit_test::suite testNSDeletePartial(features); testPageCap(features); + testFillCopy(features); + testWasm(features); test_accept(features); test_rollback(features); From 317bd4bc6e6c216c040cd407c620913322538d65 Mon Sep 17 00:00:00 2001 From: RichardAH Date: Mon, 3 Feb 2025 17:56:08 +1000 Subject: [PATCH 17/17] add strict filtering to account_tx api (#429) --- src/ripple/app/rdb/RelationalDatabase.h | 3 + src/ripple/app/rdb/backend/RWDBDatabase.h | 130 ++++++++++++++++-- .../app/rdb/backend/detail/impl/Node.cpp | 65 +++++++-- src/ripple/basics/strHex.h | 11 ++ src/ripple/rpc/handlers/AccountTx.cpp | 6 +- src/test/rpc/AccountTx_test.cpp | 62 +++++++++ 6 files changed, 247 insertions(+), 30 deletions(-) diff --git a/src/ripple/app/rdb/RelationalDatabase.h b/src/ripple/app/rdb/RelationalDatabase.h index a269bf256c..11c5924851 100644 --- a/src/ripple/app/rdb/RelationalDatabase.h +++ b/src/ripple/app/rdb/RelationalDatabase.h @@ -69,6 +69,7 @@ class RelationalDatabase std::uint32_t offset; std::uint32_t limit; bool bUnlimited; + bool strict; }; struct AccountTxPageOptions @@ -79,6 +80,7 @@ class RelationalDatabase std::optional marker; std::uint32_t limit; bool bAdmin; + bool strict; }; using AccountTx = @@ -101,6 +103,7 @@ class RelationalDatabase bool forward = false; uint32_t limit = 0; std::optional marker; + bool strict; }; struct AccountTxResult diff --git a/src/ripple/app/rdb/backend/RWDBDatabase.h b/src/ripple/app/rdb/backend/RWDBDatabase.h index 9c6d70e7e7..2858497a84 100644 --- a/src/ripple/app/rdb/backend/RWDBDatabase.h +++ b/src/ripple/app/rdb/backend/RWDBDatabase.h @@ -43,6 +43,62 @@ class RWDBDatabase : public SQLiteDatabase std::map transactionMap_; std::map accountTxMap_; + // helper function to scan for an account ID inside the tx and meta blobs + // used for strict filtering of account_tx + bool + isAccountInvolvedInTx(AccountID const& account, AccountTx const& accountTx) + { + auto const& txn = accountTx.first; + auto const& meta = accountTx.second; + + // Search metadata, excluding RegularKey false positives + Blob const metaBlob = meta->getAsObject().getSerializer().peekData(); + if (metaBlob.size() >= account.size()) + { + auto it = metaBlob.begin(); + while (true) + { + // Find next occurrence of account + it = std::search( + it, + metaBlob.end(), + account.data(), + account.data() + account.size()); + + if (it == metaBlob.end()) + break; + + // Check if this is a RegularKey field (0x8814 prefix) + if (it >= metaBlob.begin() + 2) + { + auto prefix = *(it - 2); + auto prefix2 = *(it - 1); + if (prefix != 0x88 || prefix2 != 0x14) + { + // Found account not preceded by RegularKey prefix + return true; + } + } + else + { + // Too close to start to be RegularKey + return true; + } + + ++it; // Move past this occurrence + } + } + + // Search transaction blob + Blob const txnBlob = txn->getSTransaction()->getSerializer().peekData(); + return txnBlob.size() >= account.size() && + std::search( + txnBlob.begin(), + txnBlob.end(), + account.data(), + account.data() + account.size()) != txnBlob.end(); + } + public: RWDBDatabase(Application& app, Config const& config, JobQueue& jobQueue) : app_(app), useTxTables_(config.useTxTables()) @@ -193,7 +249,17 @@ class RWDBDatabase : public SQLiteDatabase std::size_t count = 0; for (const auto& [_, accountData] : accountTxMap_) { - count += accountData.transactions.size(); + for (const auto& tx : accountData.transactions) + { + // RH NOTE: options isn't provided to this function + // but this function is probably only used internally + // so make it reflect the true number (unfiltered) + + // if (options.strict && + // !isAccountInvolvedInTx(options.account, tx)) + // continue; + count++; + } } return count; } @@ -607,12 +673,17 @@ class RWDBDatabase : public SQLiteDatabase { for (const auto& [txSeq, txIndex] : txIt->second) { + AccountTx const accountTx = accountData.transactions[txIndex]; + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + if (skipped < options.offset) { ++skipped; continue; } - AccountTx const accountTx = accountData.transactions[txIndex]; + std::uint32_t const inLedger = rangeCheckedCast( accountTx.second->getLgrSeq()); accountTx.first->setStatus(COMMITTED); @@ -652,13 +723,18 @@ class RWDBDatabase : public SQLiteDatabase innerRIt != rIt->second.rend(); ++innerRIt) { + AccountTx const accountTx = + accountData.transactions[innerRIt->second]; + + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + if (skipped < options.offset) { ++skipped; continue; } - AccountTx const accountTx = - accountData.transactions[innerRIt->second]; std::uint32_t const inLedger = rangeCheckedCast( accountTx.second->getLgrSeq()); accountTx.first->setLedger(inLedger); @@ -694,12 +770,19 @@ class RWDBDatabase : public SQLiteDatabase { for (const auto& [txSeq, txIndex] : txIt->second) { + AccountTx const accountTx = accountData.transactions[txIndex]; + + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + + const auto& [txn, txMeta] = accountTx; + if (skipped < options.offset) { ++skipped; continue; } - const auto& [txn, txMeta] = accountData.transactions[txIndex]; result.emplace_back( txn->getSTransaction()->getSerializer().peekData(), txMeta->getAsObject().getSerializer().peekData(), @@ -738,13 +821,20 @@ class RWDBDatabase : public SQLiteDatabase innerRIt != rIt->second.rend(); ++innerRIt) { + AccountTx const accountTx = + accountData.transactions[innerRIt->second]; + + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + + const auto& [txn, txMeta] = accountTx; + if (skipped < options.offset) { ++skipped; continue; } - const auto& [txn, txMeta] = - accountData.transactions[innerRIt->second]; result.emplace_back( txn->getSTransaction()->getSerializer().peekData(), txMeta->getAsObject().getSerializer().peekData(), @@ -838,18 +928,23 @@ class RWDBDatabase : public SQLiteDatabase return {newmarker, total}; } - Blob rawTxn = accountData.transactions[index] - .first->getSTransaction() + AccountTx const& accountTx = + accountData.transactions[index]; + + Blob rawTxn = accountTx.first->getSTransaction() ->getSerializer() .peekData(); - Blob rawMeta = accountData.transactions[index] - .second->getAsObject() + Blob rawMeta = accountTx.second->getAsObject() .getSerializer() .peekData(); if (rawMeta.size() == 0) onUnsavedLedger(ledgerSeq); + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + onTransaction( rangeCheckedCast(ledgerSeq), "COMMITTED", @@ -893,18 +988,23 @@ class RWDBDatabase : public SQLiteDatabase return {newmarker, total}; } - Blob rawTxn = accountData.transactions[index] - .first->getSTransaction() + AccountTx const& accountTx = + accountData.transactions[index]; + + Blob rawTxn = accountTx.first->getSTransaction() ->getSerializer() .peekData(); - Blob rawMeta = accountData.transactions[index] - .second->getAsObject() + Blob rawMeta = accountTx.second->getAsObject() .getSerializer() .peekData(); if (rawMeta.size() == 0) onUnsavedLedger(ledgerSeq); + if (options.strict && + !isAccountInvolvedInTx(options.account, accountTx)) + continue; + onTransaction( rangeCheckedCast(ledgerSeq), "COMMITTED", diff --git a/src/ripple/app/rdb/backend/detail/impl/Node.cpp b/src/ripple/app/rdb/backend/detail/impl/Node.cpp index c80038ef74..8838afff08 100644 --- a/src/ripple/app/rdb/backend/detail/impl/Node.cpp +++ b/src/ripple/app/rdb/backend/detail/impl/Node.cpp @@ -27,6 +27,7 @@ #include #include #include +#include #include #include #include @@ -758,14 +759,34 @@ transactionsSQL( options.minLedger); } + // Convert account ID to hex string for binary search + std::string accountHex = + strHex(options.account.data(), options.account.size()); + std::string sql; + // For metadata search: + // 1. Look for account ID not preceded by 8814 (RegularKey field) + // 2. OR look for account in raw transaction + std::string filterClause = options.strict ? "AND ((" + "hex(TxnMeta) LIKE '%" + + accountHex + + "%' AND " + "hex(TxnMeta) NOT LIKE '%8814" + + accountHex + + "%'" + ") OR hex(RawTxn) LIKE '%" + + accountHex + "%')" + : ""; + if (count) sql = boost::str( boost::format("SELECT %s FROM AccountTransactions " - "WHERE Account = '%s' %s %s LIMIT %u, %u;") % - selection % toBase58(options.account) % maxClause % minClause % - beast::lexicalCastThrow(options.offset) % + "INNER JOIN Transactions ON Transactions.TransID = " + "AccountTransactions.TransID " + "WHERE Account = '%s' %s %s %s LIMIT %u, %u;") % + selection % toBase58(options.account) % filterClause % maxClause % + minClause % beast::lexicalCastThrow(options.offset) % beast::lexicalCastThrow(numberOfResults)); else sql = boost::str( @@ -773,15 +794,16 @@ transactionsSQL( "SELECT %s FROM " "AccountTransactions INNER JOIN Transactions " "ON Transactions.TransID = AccountTransactions.TransID " - "WHERE Account = '%s' %s %s " + "WHERE Account = '%s' %s %s %s " "ORDER BY AccountTransactions.LedgerSeq %s, " "AccountTransactions.TxnSeq %s, AccountTransactions.TransID %s " "LIMIT %u, %u;") % - selection % toBase58(options.account) % maxClause % minClause % + selection % toBase58(options.account) % filterClause % maxClause % + minClause % (descending ? "DESC" : "ASC") % (descending ? "DESC" : "ASC") % (descending ? "DESC" : "ASC") % - (descending ? "DESC" : "ASC") % beast::lexicalCastThrow(options.offset) % beast::lexicalCastThrow(numberOfResults)); + JLOG(j.trace()) << "txSQL query: " << sql; return sql; } @@ -1114,6 +1136,21 @@ accountTxPage( if (limit_used > 0) newmarker = options.marker; + // Convert account ID to hex string for binary search + std::string accountHex = + strHex(options.account.data(), options.account.size()); + + // Add metadata search filter similar to transactionsSQL + std::string filterClause = options.strict + ? " AND ((hex(TxnMeta) LIKE '%" + accountHex + + "%' " + "AND hex(TxnMeta) NOT LIKE '%8814" + + accountHex + + "%') " + "OR hex(RawTxn) LIKE '%" + + accountHex + "%')" + : ""; + static std::string const prefix( R"(SELECT AccountTransactions.LedgerSeq,AccountTransactions.TxnSeq, Status,RawTxn,TxnMeta @@ -1132,12 +1169,12 @@ accountTxPage( { sql = boost::str( boost::format( - prefix + (R"(AccountTransactions.LedgerSeq BETWEEN %u AND %u + prefix + (R"(AccountTransactions.LedgerSeq BETWEEN %u AND %u %s ORDER BY AccountTransactions.LedgerSeq %s, AccountTransactions.TxnSeq %s LIMIT %u;)")) % toBase58(options.account) % options.minLedger % options.maxLedger % - order % order % queryLimit); + filterClause % order % order % queryLimit); } else { @@ -1150,25 +1187,25 @@ accountTxPage( auto b58acct = toBase58(options.account); sql = boost::str( boost::format(( - R"(SELECT AccountTransactions.LedgerSeq,AccountTransactions.TxnSeq, - Status,RawTxn,TxnMeta + R"(SELECT AccountTransactions.LedgerSeq,AccountTransactions.TxnSeq,Status,RawTxn,TxnMeta FROM AccountTransactions, Transactions WHERE (AccountTransactions.TransID = Transactions.TransID AND AccountTransactions.Account = '%s' AND - AccountTransactions.LedgerSeq BETWEEN %u AND %u) + AccountTransactions.LedgerSeq BETWEEN %u AND %u) %s UNION SELECT AccountTransactions.LedgerSeq,AccountTransactions.TxnSeq,Status,RawTxn,TxnMeta FROM AccountTransactions, Transactions WHERE (AccountTransactions.TransID = Transactions.TransID AND AccountTransactions.Account = '%s' AND AccountTransactions.LedgerSeq = %u AND - AccountTransactions.TxnSeq %s %u) + AccountTransactions.TxnSeq %s %u) %s ORDER BY AccountTransactions.LedgerSeq %s, AccountTransactions.TxnSeq %s LIMIT %u; )")) % - b58acct % minLedger % maxLedger % b58acct % findLedger % compare % - findSeq % order % order % queryLimit); + b58acct % minLedger % maxLedger % filterClause % b58acct % + findLedger % compare % findSeq % filterClause % order % order % + queryLimit); } { diff --git a/src/ripple/basics/strHex.h b/src/ripple/basics/strHex.h index 257fb540b3..b55ee9e874 100644 --- a/src/ripple/basics/strHex.h +++ b/src/ripple/basics/strHex.h @@ -40,6 +40,17 @@ strHex(FwdIt begin, FwdIt end) return result; } +template +std::string +strHex(FwdIt begin, std::size_t length) +{ + std::string result; + result.reserve(2 * length); + boost::algorithm::hex( + begin, std::next(begin, length), std::back_inserter(result)); + return result; +} + template ().begin())> std::string strHex(T const& from) diff --git a/src/ripple/rpc/handlers/AccountTx.cpp b/src/ripple/rpc/handlers/AccountTx.cpp index f65657d92e..52389f4e5c 100644 --- a/src/ripple/rpc/handlers/AccountTx.cpp +++ b/src/ripple/rpc/handlers/AccountTx.cpp @@ -223,7 +223,8 @@ doAccountTxHelp(RPC::Context& context, AccountTxArgs const& args) result.ledgerRange.max, result.marker, args.limit, - isUnlimited(context.role)}; + isUnlimited(context.role), + args.strict}; auto const db = dynamic_cast(&context.app.getRelationalDatabase()); @@ -369,6 +370,9 @@ doAccountTxJson(RPC::JsonContext& context) args.forward = params.isMember(jss::forward) && params[jss::forward].asBool(); + args.strict = + params.isMember(jss::strict) ? params[jss::strict].asBool() : true; + if (!params.isMember(jss::account)) return rpcError(rpcINVALID_PARAMS); diff --git a/src/test/rpc/AccountTx_test.cpp b/src/test/rpc/AccountTx_test.cpp index 85dd2978d9..6a658e5b89 100644 --- a/src/test/rpc/AccountTx_test.cpp +++ b/src/test/rpc/AccountTx_test.cpp @@ -245,6 +245,68 @@ class AccountTx_test : public beast::unit_test::suite p[jss::ledger_hash] = to_string(env.closed()->info().parentHash); BEAST_EXPECT(noTxs(env.rpc("json", "account_tx", to_string(p)))); } + + // Strict + { + Account S1{"S1"}; + Account S2{"S2"}; + Account S3{"S3"}; + env.fund(XRP(10000), S1); + env.fund(XRP(10000), S2); + env.fund(XRP(10000), S3); + env.close(); + + // Regular key set + env(regkey(S1, S2)); + env.close(); + + // we'll make a payment between S1 and S3 + env(pay(S1, S3, XRP(100))); + env.close(); + + auto hasTxs = [](Json::Value const& j, bool strict) { + if (!j.isMember(jss::result) || + j[jss::result][jss::status] != "success") + return false; + + if (strict) + { + return (j[jss::result][jss::transactions].size() == 3) && + (j[jss::result][jss::transactions][0u][jss::tx] + [jss::TransactionType] == jss::SetRegularKey) && + (j[jss::result][jss::transactions][1u][jss::tx] + [jss::TransactionType] == jss::AccountSet) && + (j[jss::result][jss::transactions][2u][jss::tx] + [jss::TransactionType] == jss::Payment); + } + + return (j[jss::result][jss::transactions].size() == 4) && + (j[jss::result][jss::transactions][0u][jss::tx] + [jss::TransactionType] == jss::Payment) && + (j[jss::result][jss::transactions][1u][jss::tx] + [jss::TransactionType] == jss::SetRegularKey) && + (j[jss::result][jss::transactions][2u][jss::tx] + [jss::TransactionType] == jss::AccountSet) && + (j[jss::result][jss::transactions][3u][jss::tx] + [jss::TransactionType] == jss::Payment); + }; + + Json::Value p{jParms}; + p[jss::account] = S2.human(); + + BEAST_EXPECT( + hasTxs(env.rpc("json", "account_tx", to_string(p)), true)); + + p[jss::strict] = true; + + BEAST_EXPECT( + hasTxs(env.rpc("json", "account_tx", to_string(p)), true)); + + p[jss::strict] = false; + + BEAST_EXPECT( + hasTxs(env.rpc("json", "account_tx", to_string(p)), false)); + } } void