Single source revcheck code/data #174

alfsb · 2024-10-31T19:09:06Z

This PR is a work in progress to concentrate revcheck code and generation of revcheck data into a single place.

In doc-base (this PR):

Change revcheck lib code to generate similar data of actual revcheck scripts;
Change doc-base/scripts/revcheck.php to use revcheck lib;
Create doc-base/scripts/translations/genrevdb.php. (proposed)

In web-doc (other PRs)

New script or code to call genrevdb.php; (systens 54)
Rewrite HTML generating pages to use new db (web-doc 58);
Remove most/all code of web-doc/scripts/rev.php. (ditto)

This PR does not try to fix the issues reported on #133 or #132, for now. The priority is removing code duplication, and after that to possibly change revcheck semantics of [skip-revcheck] (and other possible issues)

Reviews are welcome. Cosmetic changes delayed in favor of resolving issues found.

Girgias

@jimwins as you have been working on doc-web, mind keeping an eye on this?

scripts/translation/lib/all.php

scripts/translation/lib/RevcheckRun.php

jimwins · 2024-11-04T18:51:00Z

@jimwins as you have been working on doc-web, mind keeping an eye on this?

Yes, I'm ready to take a look when it is past the draft stage, and move over the systems side of doc.php.net to stop running web-doc/scripts/rev.php and switch it over to doc-base/scripts/revcheck.php (or configure.php) with the necessary configuration to generate the SQLite database or whatever that web-docs/www/rev.php will read.

alfsb · 2024-11-06T18:00:31Z

Yes, I'm ready to take a look when it is past the draft stage, and move over the systems side of doc.php.net to stop running web-doc/scripts/rev.php and switch it over to doc-base/scripts/revcheck.php

If switching over to doc-base/scripts/revcheck.php is possible, it could avoid the last step, of serializing data in a secure format, and directly use more code from this PR, that I rewrote using the popos instead arrays (to facilitate debug).

alfsb · 2024-11-06T18:14:41Z

I finished the rewriting of doc-base/scripts/revcheck.php, with back ported counting and listing behaviors to match actual code, and so facilitate migration. After all revcheck code (or data) becoming single sourced, I will then try open separated GH issues for some cases I found while rewriting, including PR 132.

There are two cosmetic modifications that i would like to make, namely:

Placing changed lines in line in old/wip list, to better use the horizontal space
Add a copy button before the actual Change column, that copies to clipboard the full path/name of file.

This won't change any data or structure, but will make the generated HTML differ, and so I may wait for web-doc code sync or the merging of this PR, before making these changes.

jimwins · 2024-11-06T20:49:00Z

Seems fine to me. What I'd like to see is a way to pass a list of languages to check to doc-base/scripts/revcheck.php, not just one, and an argument to output an SQLite database with the information as currently done by web-doc/scripts/rev.php (and not output any HTML, of course).

alfsb · 2024-11-07T00:43:43Z

That would be a partial rewrite of web-doc/scripts/rev.php, to both eliminate the code duplication and to generate the same database output as currently implemented. The base data generating code can be as minimal as:

require_once "path/to/doc-base/scripts/translation/lib/all.php";
foreach( $langs as $lang ) {
    $revcheck = new RevcheckRun( $lang );
    generateSql( $revcheck->revData );
}

By the way, the database output appears to me to be overly complicated, using 8 tables (and some indexes) where is only really necessary 3 tables: languages, translations and files. I propose to write and submit a PR for an alternative web-doc/scripts/revdb.php that does the rewrite and simplification.

After this alternative is installed, running and generating a simplified db, it would only be necessary to modify the SELECTs on HTML generating pages to use the new db/tables, with no joining necessary. For this last part I ask someone else to volunteer, before starting this alternative revdb.php.

jimwins · 2024-11-07T00:55:19Z

The script for generating the database should live in the doc-base repo because we will not do cross-repo code dependencies. I have no problem with simplifying the schema, and will handle updating the display code in web-doc.

alfsb · 2024-11-07T01:31:32Z

So, instead of cross-repo code including, I can write an doc-base/scripts/translations/genrevdb.php to generate an Sqlite file from a language list. That would be acceptable?

The proposed syntax would be: genrevdb.php [relative/path/file.db] [langdir1,...,langdirN]

Invoked from a directory above $LANG checkouts, that is afaik the same convention of doc-base/configure.php.

jimwins · 2024-11-07T02:02:18Z

That sounds good. But the path to the SQLite database may be an absolute path, too.

alfsb · 2024-11-07T14:56:02Z

Pushed with a new script named doc-base/scripts/translation/genrevdb.php. Tested in one repository, and in the following days I will test the new revcheck.php and genrevdb.php in all translations listed CI for automatic testing.

Reviews and tests are welcome. Runs in about 28s on 14y old CPU and SSD for pt_BR repo.

jimwins · 2024-11-07T19:08:50Z

Haven't played around with the resulting database and might not have time to do today, but it looks like it works for me, at least.

It would be preferable to specify languages as individual arguments instead of one argument that you explode on ,.

One thing I did notice is that the intros in the languages table are wrapped in the <intro> tag.

$ php doc-base/scripts/translation/genrevdb.php out.sqlite de,es,fr,it,ja,pl,pt_br,ro,ru,tr,uk,zh
[2024-11-07 18:54] Creating revdata database out.sqlite for languages de,es,fr,it,ja,pl,pt_br,ro,ru,tr,uk,zh.
[2024-11-07 18:54] Language de run
[2024-11-07 18:55] Language de done (elapsed 19.00s)
[2024-11-07 18:55] Language es run
[2024-11-07 18:56] Language es done (elapsed 68.00s)
[2024-11-07 18:56] Language fr run
[2024-11-07 18:57] Language fr done (elapsed 58.00s)
[2024-11-07 18:57] Language it run
[2024-11-07 18:57] Language it done (elapsed 5.00s)
[2024-11-07 18:57] Language ja run
[2024-11-07 18:58] Language ja done (elapsed 45.00s)
[2024-11-07 18:58] Language pl run
[2024-11-07 18:58] Language pl done (elapsed 4.00s)
[2024-11-07 18:58] Language pt_br run
[2024-11-07 18:58] Language pt_br done (elapsed 30.00s)
[2024-11-07 18:58] Language ro run
[2024-11-07 18:58] Language ro done (elapsed 4.00s)
[2024-11-07 18:58] Language ru run
[2024-11-07 19:00] Language ru done (elapsed 80.00s)
[2024-11-07 19:00] Language tr run
[2024-11-07 19:00] Language tr done (elapsed 17.00s)
[2024-11-07 19:00] Language uk run
[2024-11-07 19:00] Language uk done (elapsed 3.00s)
[2024-11-07 19:00] Language zh run
[2024-11-07 19:00] Language zh done (elapsed 24.00s)
[2024-11-07 19:00] Revdata database out.sqlite complete.

alfsb · 2024-11-07T20:41:50Z

It would be preferable to specify languages as individual arguments instead of one argument that you explode on ,.

Changed.

One thing I did notice is that the intros in the languages table are wrapped in the <intro> tag.

It does not affect rendering, but is certainly invalid HTML. Working with DOMDocumentFragment or detaching root-less XML is always awkward. Piecewise XML node saving is no prettier, but generates way less code. Fixed.

scripts/translation/genrevdb.php

alfsb · 2024-11-08T15:06:15Z

I am now confident this PR produces identical results in most cases, and near identical in a few cases, comparing counts and listings of old and lib versions of revcheck.

For example, ro revcheck counts 20 notinen files, but lists 21. The code that extracts hashes, determines files final status and count cases are spread in so many places that it is impossible to make further progress. But as doc-base and web-doc revchecks are run with different times (and checkout states), some differences are expected.

And with other PRs in place (listed above), I think it's time to push for this. I plan to:

Merge this PR in the following week, if there are no objections;
In the meantime, to open various separated issues some minor, some major, that changes revcheck results;
The minor issues to be pushed separately;
Two major issues, dealing with [skip-revcheck], to be pushed around early December.

This new "lib revcheck" code is somewhat future proofed, in the sense that the modifications above only generate exchanged data, and no further changes in database or HTML generating would be necessary.

jimwins

LGTM.

alfsb · 2024-11-11T12:05:27Z

Merged. Further issues/PRs will consider this as baseline. Is necessary to comment/notify the PRs of system and web-doc, above?

jimwins · 2024-11-11T21:03:33Z

All handled now, new data is live on https://doc.php.net/.

alfsb · 2024-11-11T21:32:07Z

Yay! Let's celebrate. No more duplicated revcheck code in various repositories. I will treat myself with a good wine tonight.

alfsb · 2024-11-12T11:03:35Z

I spoke to soon. The pt_BR translation is not shown, other are ok. See: https://doc.php.net/revcheck.php?lang=pt_br

André L F S Bacci added 6 commits October 31, 2024 15:43

DTO for revcheck data.

f9921a1

Preparation for revcheck data exporting.

d72bb52

Backport loose and fixed [skip-revcheck] modes

72453ce

Additional revcheck data.

600714c

Revcheck deduplication: clean up, header.

221e0dc

Revcheck deduplication: translators, file summary.

83fb651

Girgias reviewed Nov 4, 2024

View reviewed changes

scripts/translation/lib/all.php Outdated Show resolved Hide resolved

scripts/translation/lib/RevcheckRun.php Outdated Show resolved Hide resolved

André L F S Bacci added 7 commits November 5, 2024 12:31

Revcheck deduplication: old/wip files.

3b62b38

Move backport code and docs to a separate file.

156b0b5

Revcheck deduplication: backport couting behaviour.

4405435

Revcheck deduplication: more backport couting woes.

31cf0db

Revcheck deduplication: notinen, revtag, unstranslated.

324f48c

Revcheck deduplication: ignore, regex, xml.

60b5f1e

Revcheck deduplication: adds/dels, clean up.

a0ddca6

alfsb marked this pull request as ready for review November 6, 2024 18:00

Revcheck deduplicatoin: genrevdb.php.

55bae33

André L F S Bacci added 2 commits November 7, 2024 12:13

Revcheck deduplicatoin: genrevdb.php.

1ab908c

More backport and small fixes.

a08561f

André L F S Bacci added 2 commits November 7, 2024 17:27

Review changes/fixes.

d254ced

Better log messages.

117a61d

jimwins reviewed Nov 8, 2024

View reviewed changes

scripts/translation/genrevdb.php Outdated Show resolved Hide resolved

This was referenced Nov 8, 2024

Use new generation script that lives in doc-base php/systems#64

Merged

Use revcheck information generated by doc-base/scripts/translation/genrevdb.php php/web-doc#58

Merged

Remove unrelated filter.

3eb4ef5

jimwins approved these changes Nov 8, 2024

View reviewed changes

alfsb mentioned this pull request Nov 10, 2024

Revcheck changes #175

Closed

alfsb merged commit bee795d into php:master Nov 11, 2024
9 of 12 checks passed

Girgias mentioned this pull request Dec 5, 2024

Manual infra-estructure #199

Open

32 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Single source revcheck code/data #174

Single source revcheck code/data #174

alfsb commented Oct 31, 2024 •

edited

Loading

Girgias left a comment

jimwins commented Nov 4, 2024

alfsb commented Nov 6, 2024

alfsb commented Nov 6, 2024

jimwins commented Nov 6, 2024

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024 •

edited

Loading

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024

alfsb commented Nov 7, 2024

alfsb commented Nov 8, 2024

jimwins left a comment

alfsb commented Nov 11, 2024

jimwins commented Nov 11, 2024

alfsb commented Nov 11, 2024 via email

alfsb commented Nov 12, 2024

Single source revcheck code/data #174

Single source revcheck code/data #174

Conversation

alfsb commented Oct 31, 2024 • edited Loading

Girgias left a comment

Choose a reason for hiding this comment

jimwins commented Nov 4, 2024

alfsb commented Nov 6, 2024

alfsb commented Nov 6, 2024

jimwins commented Nov 6, 2024

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024 • edited Loading

alfsb commented Nov 7, 2024

jimwins commented Nov 7, 2024

alfsb commented Nov 7, 2024

alfsb commented Nov 8, 2024

jimwins left a comment

Choose a reason for hiding this comment

alfsb commented Nov 11, 2024

jimwins commented Nov 11, 2024

alfsb commented Nov 11, 2024 via email

alfsb commented Nov 12, 2024

alfsb commented Oct 31, 2024 •

edited

Loading

jimwins commented Nov 7, 2024 •

edited

Loading