Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(replicator): replicator Handles message, deletedMessages in event out of order. #1631

Closed
TecSong opened this issue Jan 24, 2024 · 1 comment
Labels
s-noop Cannot be worked on

Comments

@TecSong
Copy link
Contributor

TecSong commented Jan 24, 2024

What is the bug?
A concise, high level description of the bug and how it affects you
After modifying the profile, the data in the replicator pg table record is not new, but old.
image
image

Others also have the same problem, please see: #1603

How can it be reproduced? (optional)
Background: I run a hub and a replicator, and the replicator syncs data from the hub to pg.

When I change the display name in warpcast, the hub will generate an event, the details are as follows:
{ "type": "HUB_EVENT_TYPE_MERGE_MESSAGE", "id": 395450384842752, "mergeMessageBody": { "message": { "data": { "type": "MESSAGE_TYPE_USER_DATA_ADD", "fid": 19062, "timestamp": 96545504, "network": "FARCASTER_NETWORK_MAINNET", "userDataBody": { "type": "USER_DATA_TYPE_DISPLAY", "value": "yikosong" } }, "hash": "0x96240ec645f6de86fd77ce363b4ecb3bb73f7246", "hashScheme": "HASH_SCHEME_BLAKE3", "signature": "4wcqCGUGgD2tHfG9iQsrUqwZJN/86PkXEPOWWFehq7DK9nn6njIHjik+91lKR1+EyaJmfUDFD2DOEiP4vYnuAw==", "signatureScheme": "SIGNATURE_SCHEME_ED25519", "signer": "0x11637e5ee0cc28ec00d58a8bdade374e22a796a73334a69b1f42dc37b3b9b518" }, "deletedMessages": [ { "data": { "type": "MESSAGE_TYPE_USER_DATA_ADD", "fid": 19062, "timestamp": 96540291, "network": "FARCASTER_NETWORK_MAINNET", "userDataBody": { "type": "USER_DATA_TYPE_DISPLAY", "value": "yikosong.eth" } }, "hash": "0x30ba0804cb09a353a83b6cc4d23bcfa7400e709e", "hashScheme": "HASH_SCHEME_BLAKE3", "signature": "+6Ccg+vh6QbgCWQMIxQLEfGRQLPAfwigp7h3K2A5qfD+9uwi8WmYTjxT/vHeJ8mri/SwWG3ZYjEU/iKWqx++Bw==", "signatureScheme": "SIGNATURE_SCHEME_ED25519", "signer": "0x11637e5ee0cc28ec00d58a8bdade374e22a796a73334a69b1f42dc37b3b9b518" } ] } }

  • It contains message and deletedMessages, and the replicator handles the event, but in the log, it processes the message first, and then the deletedMessages, and the update operation of the user_data table causes the data that should be deleted to overwrite the latest data. I found the location in the replicator repo where the resolution handles the event, as follows:
  const { message, deletedMessages } = hubEvent.mergeMessageBody;
  await executeTx(db, async (trx) => {
    await mergeMessage(message, trx, log, redis);

    for (const deletedMessage of deletedMessages) {
      await deleteMessage(deletedMessage, trx, log, redis);
    }
  });
}```
file path: hub-monorepo/apps/replicator/src/processors/index.ts

**Additional context (optional)**
So I'd like to ask why (message->deletedMessages) is handled in this order, whether it's a bug, and whether it can be fixed in a timely manner, this problem has existed for a long time
@github-actions github-actions bot added the s-triage Needs to be reviewed, designed and prioritized label Jan 24, 2024
@sds sds added s-noop Cannot be worked on and removed s-triage Needs to be reviewed, designed and prioritized labels Jun 3, 2024
@sds
Copy link
Member

sds commented Jun 3, 2024

Thanks for the report, sorry for the delay in response.

The old replicator at this point has a number of issues. We recommend using the Shuttle project instead.

@sds sds closed this as completed Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
s-noop Cannot be worked on
Projects
None yet
Development

No branches or pull requests

2 participants