ThreadPool as a default way to make jobs separated into small units

[IvanChepurnyi]

I am working right now on a library that is going to facilitate high performant interprocess communication via socket pair and in the future with shared memory. Idea is to make use main process to find files to process and use a thread pool to parse each PHP file and return data back to the main process to build a graph of class dependencies.

Any thoughts are welcome

[kandy]

Is it similar to what ZMQ provides? See https://zeromq.org/socket-api/ and http://api.zeromq.org/master:zmq-inproc
Also, swoole extension has a similar ability https://www.php.net/manual/en/swoole-process.statqueue.php

[IvanChepurnyi]

@kandy we don't want to require extension to be able to use it. Also there is no need for message queue as we just need mpsc style channel for thread syncing

[ilnytskyi]

@IvanChepurnyi
I tried to profile setup:di:compile and I see a few challenges.

1. XML merging. Most of time is spent on

\Magento\Framework\Config\Dom::_getMatchedNode
that caused by slow \DOMXPath::query

But in terms of general picture, xml merging is performed fast. Maybe there is a room to improve it by 5-10 seconds.
2. Actual operations that run one by one in a single thread

\Magento\Setup\Module\Di\App\Task\Manager::process

I tried make them async with pcntl_fork like this

            $pid = pcntl_fork();
            if ($pid == -1) {
                exit("Error forking...\n");
            }
            else if ($pid == 0) {
                echo $operation->getName() . ' Started' . PHP_EOL;
                $operation->doOperation();
                echo $operation->getName() . ' Completed successfully' . PHP_EOL;
                exit();
            }

That seemed to work with no problems
3. Then I tried to compare generated results form generated folder and number of files was the same, but with different file size

The only difference was related to generated/metadata folder.
Some class configs were missing or different arguments passed (e.g. Some\Class instead Some\Class\Interceptor)

Some thoughts:

1. Maybe we can run only part of operations independently and then orchestrate everything somehow.
2. Maybe it's better to keep operations in a single thread but make smaller tasks async (like: actual file generation)
   e.g. 2.1 Interceptors config read can be independent per area.
   
   and 2.2 Code generation as well
   
3. Maybe do both, but not to overload CPU with too many processes :)

Maybe with async approach we can reduce setup:di:compile time to it's longest operation that depending on CPU might only take 20-30 seconds in total.

[IvanChepurnyi]

Hey Stanislav, I plan to make it async on the lower level for each file. ThreadPool is going to fork a process before doing processing and then the forked processes to parse files and send result over IPC back