Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Weighed random lb #11

Merged
merged 460 commits into from
Apr 17, 2024
Merged

Conversation

Jo-stfc
Copy link

@Jo-stfc Jo-stfc commented Apr 17, 2024

No description provided.

amadio and others added 30 commits July 11, 2023 08:53
When setting up mapping rules, if the user feature was utilized, a
logic error would cause the rule to be ignored.

Fixes xrootd#2056
This commit reintroduces a header parser for Transfer-Encoding: chunked to maintain compatibility with the EGI Nagios probe. This issue was first mentioned in xrootd#2058.

Fixes: d96b2b7, xrootd#2009
         This patch provides backward compatiblity for old auth files.
CTest parses the build output and will return a non-zero value
if the configure/build phase of isa-l appears in the output, which
fails CI builds that run via CTest with the test.cmake script.
When dlclose(libHandle) is called, some test threads may still be
using the library being unloaded, so sometimes it causes the tests
to crash.
We need to use target_include_directories() instead of set_property() on
the uuid::uuid target, since setting the property directly does not make
those directories used during compilation. This is needed for CMS-SW, as it
uses a specific version of libuuid installed in a non-standard location.
However, on macOS, using target_include_directories() causes kernel header
paths to be added in the compilation ahead of the C++ compiler's own paths,
which causes compilation failures with clang from Xcode. This is why we
were using set_property() up to now. For target_include_directories() to
work in both cases (macOS and Linux with non-standard install path), we
need to set CMAKE_FIND_FRAMEWORKS to LAST before looking for the uuid.h
header. This way the kernel framework header paths are skipped, and uuid.h
is found in one of the regular macOS SDK header paths instead.

We now also perform a search for the uuid library using find_library, to
avoid unnecessary calls to pkg-config. We keep it only as a last resort to
find the library, as in some cases pkg-config works where we'd have to set
CMAKE_PREFIX_PATH for find_library to work. Nevertheless, when we do need
to fallback to pkg-config, the include directory set by the uuid.pc file is
"wrong", i.e. it reports <prefix>/include/uuid, not just <prefix>/include
as we expect, so we need to fix it by stripping the last component of the
include directory in that case.

Finally, in order to let pkg-config work, we were clearing CMake cache
variables in the beginning of the Findlibuuid module, which had the
unfortunate side effect of preventing users from setting UUID_INCLUDE_DIR
and UUID_LIBRARY in the CMake command line to pick up a specific version of
libuuid. We moved this section within the block where pkg-config is used to
correct this problem.
This lets users export the CMAKE_ARGS environment variable to pass extra
options to CMake when configuring to allow users to customize the build.
If possible, arguments are split with shlex so that quotes are preserved
for options that may contain space characters. If shlex is not available,
then arguments are split on blank space. For example, to set the CXXFLAGS
to "-Wall -g", use:

$ export CMAKE_ARGS="-DCMAKE_CXX_FLAGS='-Wall -g'"
$ python3 -m pip wheel --verbose .

Alternatively, one may set options in a CMake cache file:

$ echo 'set(CMAKE_CXX_FLAGS "-Wall -g" CACHE STRING "" FORCE)' > cfg.cmake
$ env CMAKE_ARGS='-C cfg.cmake' python3 -m pip wheel --verbose .

Of course, some variables like CXXFLAGS are used automatically by CMake,
so in this case you can use the variable directly rather than use the
CMAKE_ARGS environment variable. For example, to compile the Python
bindings with Clang, one can use directly the CC and CXX variables:

env CC=clang CXX=clang++ python3 -m pip wheel --verbose .

Closes: xrootd#2062
In order to build binary wheels with manylinux images, we need to
not depend on Development.Embed, since on those images, libpython
is not available and CMake will not consider Python found without
this change. XRootD only needs the Python.h header to build the
Python bindings. Issue: xrootd#1833.

See also: pypa/manylinux#484
Co-Authored-by: Andreas Joachim Peters <andreas.joachim.peters@cern.ch>

Fixes: xrootd#2080
amadio and others added 29 commits February 22, 2024 13:16
In some cases, like when a server is configured for third-party
transfers, the server calls the client directly. Therefore, it
is a good idea to install both server and client when running
dnf install xrootd.
- Update DEB/RPM install instructions
- Mention that XRootD is also available in conda-forge
- Replace relative with full links, so that links work in PyPI

Fixes: xrootd#2203
If a token has a broad scope and all the listed restricted paths
are more specific, then try to generate ACLs that include the
restricted paths.

Example: suppose we have a token with scope `storage.read:/` but
restricted paths `/foo` and `/bar`.  Currently, we will reject the
scope and give the token no permissions.  With this commit, we will
generate acls of `read:/foo` and `read:/bar`.  So, as long as the
access being attempted is within the restricted path, we can permit
it.
If there's an ACL for `/foo`, do not permit `/foobar` but do permit
`/foo/bar`.
This handles the case where the token path ends in `/`; because the
scitokens-cpp library normalizes the paths to not end with the '/'
character, the only case to handle is the root directory.
stat permissions are implicitly required for storage systems for create, modify  and delete operations. Currently tokens with purely create and modify permissions would fail with permission denied on stat on attempts to perform those operations
…sts more than 100 streams for the HTTP-TPC PULL transfer
As required by WLCG token spec, allow the creation of superfolders if
a filepath is requested with a create or modify scope.

https://github.com/WLCG-AuthZ-WG/common-jwt-profile/blob/master/profile.md

Fixes: xrootd#2185
In 37b7513, stopping the poller was
(possibly inadvertently) moved to happen after stopping the task
manager. However, if the task manager is stopped first, an event may
become ready for read/write after the task manager has already been
destroyed, which may cause a socket callback to try to reinitialize
the poller by calling Poller->Init() while Poller->Stop() is running
on another thread, which may cause a crash.
…ng gcovr

Important to ensure that, say, a build with gcc-14/g++-14 uses the
matching gcov-14 for collecting coverage information, otherwise it
fails.
On my machine, before this commit:
Total Test time (real) = 299.07 sec

and after this commit:
Total Test time (real) = 144.01 sec
…lbyLoad based selection.

This can be enabled by adding the configuration option:
cms.sched randlb 1.
@Jo-stfc Jo-stfc merged commit cf4b34f into stfc:weighted_random_lb Apr 17, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.