[DO-NOT-MERGE] .ef optimization experiments #12907
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wmitsuda
added
the
do-not-merge
|
note: code quality is "hacky", pls do not consider for review yet, only for evaluation of the overall solution + validating the strategy |
|
now that the prototype 1 is out, let me rebase it to main as it is 20 days outdated... |
wmitsuda
force-pushed
the
wmitsuda/ef-optimization-experiments
branch
5 times, most recently
from
3ea3702 to
9e5031a
Compare
wmitsuda
force-pushed
the
wmitsuda/ef-optimization-experiments
branch
3 times, most recently
from
9e7c1b2 to
ae6c0bf
Compare
AskAlexSharov
pushed a commit
that referenced
this pull request
Dec 10, 2024
As part of #12907 I'll have other iterator-like sequence implementations. It makes sense to generalize the ErrEliasFanoIterExhausted and move it to a common package and reuse it, rather than making a bunch of IteratorExhausted-like for each implementation.
wmitsuda
force-pushed
the
wmitsuda/ef-optimization-experiments
branch
from
ae6c0bf to
6d7f604
Compare
|
Current status:
Next steps:
|
|
@wmitsuda FYI: maybe next commands may help to test files |
wmitsuda
force-pushed
the
wmitsuda/ef-optimization-experiments
branch
2 times, most recently
from
928dc9d to
b0c3db9
Compare
|
Current status:
so, we can conclude data is correct + size reduction is the expected one so far. |
|
from now on, I think we are ready to advance one more step towards making it stable:
|
wmitsuda
force-pushed
the
wmitsuda/ef-optimization-experiments
branch
from
f5dfa8c to
b6db9b9
Compare
awskii
reviewed
Dec 19, 2024
| } | ||
|
|
||
| // simple encoding | ||
| if data[0]&0b11110000 == 0b10000000 { |
There was a problem hiding this comment.
forgot to use the existing SimpleEncoding const... will also add one for the comparison mask
| // from raw data | ||
| func Count(baseNum uint64, data []byte) uint64 { | ||
| // plain elias fano (legacy) | ||
| if data[0]&0b10000000 == 0 { |
There was a problem hiding this comment.
thats Simple Encoding constant value, probably zero bit could be just compared to 0b0 const
There was a problem hiding this comment.
no, because if the 8th bit is 0, meaning it is legacy elias fano encoding type, I'm using that entire byte as part of the following elias fano struct, so in theory the remaining 7 bits of data[0] will be used to decode elias fano.
anyway, will add more consts to clarify that comparison.
| } | ||
|
|
||
| // simple encoding | ||
| if raw[0]&0b11110000 == 0b10000000 { |
There was a problem hiding this comment.
whats that mask 0xf0? using const is better, also seems can use just 2 bits instead of 8, have something to encode up to 2^6? :)
There was a problem hiding this comment.
because I'm reserving the range [0b10000000, 0b100001111] to SimpleEncoding. I'm using the it to encode at the same time the type (SimpleEncoding) + the number of elements (up to 16) in the same byte and save space.
So, the masking + comparison is saying: ignore the less significant 4 bits, if it is 0b1000????, then it is SimpleEncoding.
yeah, will add consts, forgot it :)
I didn't understand the part about 2^6?
| it.Seek(uint64(v)) | ||
| } | ||
| return it | ||
| } else if s.currentEnc == PlainEliasFano || s.currentEnc == RebasedEliasFano { |
There was a problem hiding this comment.
btw since constants are growing could just do s.currentEnc < threshold. So maybe reorder them as PlainEF, RebasedEF, and then SimpleEnc?
There was a problem hiding this comment.
hm, problem here is that the consts are at the same time the type itself and the bit fragments that are used when encoding them, i.e., they are not 1, 2 and 3, but 0, 128 and 144.
I think I could do it, but I think the code now is clearer to read and understand?
Also, if we are going to merge it now, maybe when can even remove the plain elias-fano choice since it would not be used anymore in this part of code and have one less comparison.
…everything must be written optimized from now on
…tal flag being skipped
…of step being verified
|
Current status:
|
|
regarding syncing from scratch:
In order to solve those last points, I'm syncing holesky again from scratch and will do another comparison with holesky converted vs. holesky from scratch from this PR. If both snapshots are identical, then the code is correct. |
|
updated the PR to consolidate the rollout tools:
Convert an existing node:
The |
|
Status: finished another round of tests on holesky, now comparing a fresh synced node vs. this branch PR from scratch. Methodology:
Results:Expected:
NOT expected:
|
|
Next steps: another round of sync from scratch, now I'll try to force use the same salt as the snapshots served by Erigon in order to reproduce the same indexes. @AskAlexSharov @awskii I could use some advice here if there is something obvious I'm not seeing:
Is there any other moving part other than the salt that does not make the generation deterministic? |
|
Commitment is not enough determenistic. - it’s known issue. |
FYI: according to Giulio on Discord, it is probably due to purification process |
|
Status, another round on holesky completed, comparing holesky beta 1 vs. holesky from scratch + PR Regarding .ef contents, everything matches, except |
|
Regarding comparing converted holesky vs. generated holesky (only snapshots dir), it seems now everything matches, except:
|
|
Given the overall results on holesky now seems satisfactory, now triggering a regeneration run on mainnet, should be done in < 1 week |
|
Status: mainnet sync from scratch still running after 1 week, at block 17.9M. note: mainnet sync performance noticeably drops after block 16M, going from +700Mgas/s -> 400-500Mgas/s. |
|
Status: mainnet sync from scratch done after 11 days, now proceeding to check data integrity. FYI: a few hours after reaching the tip I got: #13676 , but I think it has nothing to do with this change, so ignoring in this context. |
|
data seems good, diffs are similar to what I found on holesky. I have another node that is not converted, but new data is being generated in the new format, and new .ef files are equivalent to plain E3 ones. |
|
given our previous conversation about leaving this one post-E3 final, I'll keep merging main and running multiple rounds of sync from scratch on mainnet from now on in order to catch any regressions. |
Open
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.