GH-41816: [C++] Add Minimal Meson Build of libarrow #45441

WillAyd · 2025-02-06T00:45:37Z

Rationale for this change

The Meson build system may be more user friendly to some developers, and may make it easier to perform tasks like valgrind, coverage, or ASAN/UBSAN coverage. There is also a prior art for using meson in the nanoarrow and arrow-adbc projects.

What changes are included in this PR?

This PR implements a Meson configuration that can build a minimal libarrow.

Are these changes tested?

A nightly CI job is also added to detect regressions.

Are there any user-facing changes?

No

GitHub Issue: [C++] Meson Support for Arrow #41816

WillAyd · 2025-02-06T00:51:32Z

@kou this is a simplified attempt at adding Meson to address your comment #41816 (comment)

This is currently slower than the CMake configuration by a good deal (pending investigation) and the configuration file is not 100% complete, but this should give us an idea of what a Meson configuration may look like.

To use, from the cpp directory developers can:

meson setup builddir -Dtests=true -Dcompute=true
meson compile -C builddir
meson test -C builddir

For ASAN/UBSAN, users could simply:

meson setup builddir -Dtests=true -Dcompute=true -Db_sanitize=address,undefined

Or if the project is already setup run:

meson configure -C builddir -Db_sanitize=address,undefined

Coverage can be enabled with:

meson configure -C builddir -Db_coverage=true

and tests can be run under valgrind with:

meson test -C builddir --wrap='valgrind --track-origins=yes --leak-check=full' --print-errorlog

kou · 2025-02-06T02:15:28Z

Thanks.

Can we start from more simplified version? For example, we don't need libarrow_testing, tests and so on at the first step.

We want to add a nightly CI for this to detect regression.

We want to update version information automatically in release process. For example:

arrow/dev/release/utils-prepare.sh

Lines 39 to 44 in 0556905

    
           pushd "${ARROW_DIR}/c_glib" 
        
           sed -i.bak -E -e \ 
        
             "s/^version = '.+'/version = '${version}'/" \ 
        
             meson.build 
        
           rm -f meson.build.bak 
        
           git add meson.build

(I can do it in this branch later.)

Anyway, I didn't know that meson format exists! I want to use it in c_glib/ too.

WillAyd · 2025-02-06T16:33:54Z

Sure I can strip down further. So do think just something that builds libarrow is the right starting point?

kou · 2025-02-07T01:39:16Z

Yes. Only minimal libarrow (e.g. no compute, no filesystem and so on) is the right starting point.

WillAyd

Thanks @kou . Hope this is more in line with your expectation.

As far as nightlies go, can you point me to an existing nightly CI setup in the repo? I was able to grep for some R nightlies, but not sure if there is existing infrastructure for C++ nightly jobs where this would be better placed

WillAyd · 2025-02-11T21:08:13Z

cpp/src/arrow/meson.build

+    objects: objlibs,
+    include_directories: [include_dir],
+    install: true,
+    # compute/expression.cc may have undefined IPC symbols in non-IPC builds


Wasn't sure if this was intentional or not in the existing code base

Ah, we should disable features that depend on IPC in cpp/src/arrow/compute/expression.cc like GH-45171.

Could you open an issue for this?

No problem - see #45512

WillAyd · 2025-02-11T21:09:22Z

cpp/src/arrow/meson.build

+    override_options: {'b_lundef': 'false'},
+)
+
+# Meson does not allow you to glob for headers to install. See also


Meson is pretty strict about wildcards for the reasons outlined in their FAQ. Idiomatically, Meson would want you to put the files you want to install in a separate directory and call install_subdir, but that would go beyond the scope of this initial PR I think

WillAyd · 2025-02-11T21:09:48Z

cpp/src/arrow/meson.build

+arrow_so_version = (ver_major.to_int() * 100 + ver_minor.to_int()).to_string()
+arrow_full_so_version = '@0@.@1@.@2@'.format(arrow_so_version, ver_patch, 0)
+
+# TODO: The Meson generated .pc file does not include the Apache license


I did not research this in too much detail yet; figured I'd check if it was a big deal before investing time into a resolution

We don't need it.
arrow.pc.in has the license header because it's in this repository. Files that don't exist in this repository don't need it.

cpp/src/arrow/util/meson.build

kou

Can I push some commits to this branch for nightly CI and auto version update?

kou · 2025-02-12T01:16:35Z

cpp/meson.build

+    'cpp',
+    'c',
+    version: '19.0.0-SNAPSHOT',
+    license: 'Apache 2.0',


Suggested change

license: 'Apache 2.0',

license: 'Apache-2.0',

kou · 2025-02-12T01:26:44Z

cpp/meson.build

+
+git_id = get_option('git_id')
+if git_id == ''
+    git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()


Can we ignore git log error? This will fail when we use source archive.

Suggested change

git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()

git_id = run_command('git', 'log', '-n1', '--format=%H').stdout().strip()

kou · 2025-02-12T01:27:03Z

cpp/meson.build

+
+git_description = get_option('git_description')
+if git_description == ''
+    git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()


Suggested change

git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()

git_description = run_command('git', 'describe', '--tags').stdout().strip()

kou · 2025-02-12T01:29:14Z

cpp/src/arrow/meson.build

+
+# Meson does not natively support object libraries
+# https://github.com/mesonbuild/meson/issues/13843
+objlib_sources = {


We don't need to emulate objlib in Meson. It's just for faster build.
We can just use library().

I have a pretty poor understanding of how object libraries actually work, but I don't think we can just convert these to using the library call unless we want the linker to allow undefined symbols in these libraries as well (?)

Trying to do what I think you suggest generates a huge amount of errors like:

[7/96] Linking target src/arrow/libarrow_io.so FAILED: src/arrow/libarrow_io.so c++ -o src/arrow/libarrow_io.so src/arrow/libarrow_io.so.p/io_buffered.cc.o src/arrow/libarrow_io.so.p/io_caching.cc.o src/arrow/libarrow_io.so.p/io_compressed.cc.o src/arrow/libarrow_io.so.p/io_file.cc.o src/arrow/libarrow_io.so.p/io_hdfs.cc.o src/arrow/libarrow_io.so.p/io_hdfs_internal.cc.o src/arrow/libarrow_io.so.p/io_interfaces.cc.o src/arrow/libarrow_io.so.p/io_memory.cc.o src/arrow/libarrow_io.so.p/io_slow.cc.o src/arrow/libarrow_io.so.p/io_stdio.cc.o src/arrow/libarrow_io.so.p/io_transform.cc.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,libarrow_io.so -Wl,--end-group /usr/bin/ld: src/arrow/libarrow_io.so.p/io_buffered.cc.o: in function `arrow::io::BufferedInputStream::SetBufferSize(long)': buffered.cc:(.text+0x13a9): undefined reference to `arrow::Buffer::CheckCPU() const' /usr/bin/ld: buffered.cc:(.text+0x13b1): undefined reference to `arrow::Buffer::CheckMutable() const' /usr/bin/ld: buffered.cc:(.text+0x1407): undefined reference to `arrow::util::detail::StringStreamWrapper::StringStreamWrapper()' /usr/bin/ld: buffered.cc:(.text+0x149e): undefined reference to `arrow::util::detail::StringStreamWrapper::str[abi:cxx11]()' /usr/bin/ld: buffered.cc:(.text+0x14a6): undefined reference to `arrow::util::detail::StringStreamWrapper::~StringStreamWrapper()' /usr/bin/ld: buffered.cc:(.text+0x14b6): undefined reference to `arrow::Status::Status(arrow::StatusCode, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'

Ah, we can remove objlib_sources and static_library() for them entirely.
We can just add sources in objlib_sources to arrow_srcs and use it in one library() (that already exists in this PR).

(We don't need objlib feature entirely with Meson.)

Makes sense. I've still kept the top level dict for parity with CMake, and since some of the sources vary depending on compilation options (like arrow compute). Let me know if this is more in line with what you are thinking

kou · 2025-02-12T01:32:57Z

cpp/src/arrow/meson.build

+    objects: objlibs,
+    include_directories: [include_dir],
+    install: true,
+    # compute/expression.cc may have undefined IPC symbols in non-IPC builds


Ah, we should disable features that depend on IPC in cpp/src/arrow/compute/expression.cc like GH-45171.

Could you open an issue for this?

kou · 2025-02-12T02:21:26Z

cpp/src/arrow/meson.build

+arrow_so_version = (ver_major.to_int() * 100 + ver_minor.to_int()).to_string()
+arrow_full_so_version = '@0@.@1@.@2@'.format(arrow_so_version, ver_patch, 0)
+
+# TODO: The Meson generated .pc file does not include the Apache license


We don't need it.
arrow.pc.in has the license header because it's in this repository. Files that don't exist in this repository don't need it.

cpp/src/arrow/util/meson.build

kou · 2025-02-12T02:26:29Z

cpp/src/arrow/util/meson.build

+    configuration: conf_data,
+    format: 'cmake@',
+    install: true,
+    install_dir: 'arrow',


Suggested change

install_dir: 'arrow',

install_dir: 'arrow/util',

kou · 2025-02-12T02:27:11Z

cpp/src/arrow/util/meson.build

+foreach cmakedefine : [
+    'ARROW_COMPUTE',
+    'ARROW_CSV',
+    'ARROW_CUDA',
+    'ARROW_DATASET',
+    'ARROW_FILESYSTEM',
+    'ARROW_FLIGHT',
+    'ARROW_FLIGHT_SQL',
+    'ARROW_IPC',
+    'ARROW_JEMALLOC',
+    'ARROW_JEMALLOC_VENDORED',
+    'ARROW_JSON',
+    'ARROW_MIMALLOC',
+    'ARROW_ORC',
+    'ARROW_PARQUET',
+    'ARROW_SUBSTRAIT',
+    'ARROW_AZURE',
+    'ARROW_ENABLE_THREADING',
+    'ARROW_GCS',
+    'ARROW_HDFS',
+    'ARROW_S3',
+    'ARROW_USE_GLOG',
+    'ARROW_USE_NATIVE_INT128',
+    'ARROW_WITH_BROTLI',
+    'ARROW_WITH_BZ2',
+    'ARROW_WITH_LZ4',
+    'ARROW_WITH_MUSL',
+    'ARROW_WITH_OPENTELEMETRY',
+    'ARROW_WITH_RE2',
+    'ARROW_WITH_SNAPPY',
+    'ARROW_WITH_UCX',
+    'ARROW_WITH_UTF8PROC',
+    'ARROW_WITH_ZLIB',
+    'ARROW_WITH_ZSTD',
+    'PARQUET_REQUIRE_ENCRYPTION',
+]
+    conf_data.set(cmakedefine, false)
+endforeach


It seems that we don't need to use foreach here:

conf_data.set('ARROW_COMPUTE', false) conf_data.set('ARROW_CSV', false) ...

kou · 2025-02-12T02:27:47Z

cpp/src/arrow/util/meson.build

+    install: true,
+    install_dir: 'arrow',


We don't need to install _internal.h:

Suggested change

install: true,

install_dir: 'arrow',

WillAyd · 2025-02-12T02:47:53Z

Sure no problem

Remove extraneous files

kou · 2025-02-17T01:41:24Z

@github-actions crossbow submit test-conda-cpp-meson

github-actions · 2025-02-17T01:43:34Z

Revision: 564df5f

Submitted crossbow builds: ursacomputing/crossbow @ actions-527e5649ae

Task	Status
test-conda-cpp-meson

kou

I've rebased on main, added support for updating version in release process and added nightly CI job for Meson.

Could you update the PR description?

kou · 2025-02-17T01:43:05Z

cpp/src/arrow/meson.build

+# Meson does not natively support object libraries
+# https://github.com/mesonbuild/meson/issues/13843


Can we remove this comment?
We don't need this comment now, right?

kou · 2025-02-17T01:44:35Z

cpp/src/arrow/meson.build

+        'device.h',
+        'extension_type.h',
+        'memory_pool.h',
+        'memory_pool_test.h',


Suggested change

'memory_pool_test.h',

kou · 2025-02-17T01:45:39Z

cpp/src/arrow/meson.build

+        'extension_type.h',
+        'memory_pool.h',
+        'memory_pool_test.h',
+        'pch.h',


Suggested change

'pch.h',

kou · 2025-02-17T01:46:00Z

cpp/src/arrow/meson.build

+        'stl_allocator.h',
+        'stl.h',
+        'stl_iterator.h',


Could you sort them?

raulcd · 2025-02-17T09:13:45Z

ci/scripts/cpp_test.sh

+  meson test \
+    --print-errorlogs \
+    "$@"


We aren't building any tests now, are we? On the current CI:

+ meson test --print-errorlogs No tests defined.

For now we aren't actually building any tests - just creating an MRE for Meson with libarrow

Right. We'll add tests later.

WillAyd · 2025-02-18T01:42:03Z

I've rebased on main, added support for updating version in release process and added nightly CI job for Meson.

Could you update the PR description?

I've addressed your feedback. Not sure I am doing what you are asking with respect to the PR description but I changed it to be more exact. Let me know if there is something else should be doing.

kou · 2025-02-18T02:01:48Z

Not sure I am doing what you are asking with respect to the PR description but I changed it to be more exact.

We use the PR description for a commit message when we merge this. So we want to update the PR description to reflect the latest PR changes.

BTW, it seems that you didn't change the PR description yet. (You might forget to save your PR description change.)

kou · 2025-02-18T02:01:58Z

@github-actions crossbow submit test-conda-cpp-meson

github-actions · 2025-02-18T02:04:09Z

Revision: 428c2f3

Submitted crossbow builds: ursacomputing/crossbow @ actions-3c4e1b441c

Task	Status
test-conda-cpp-meson

WillAyd · 2025-02-18T02:26:27Z

Do you mean the commit message? I can squash up and change that if so. On Github I've already change the PR title

kou · 2025-02-18T05:51:12Z

I mean the merge commit's commit message not commit messages of commits in this PRs.

You don't need to squash commits in this PR. Because we use "Squash and merge" feature provided by GitHub. We use the following commit message:

${PR_TITLE}

${PR_DESCRIPTION}

For example, ec3d283 's commit message uses #44375 's title and description. It doesn't use commit messages of commits in #44375 .

On Github I've already change the PR title

Could you also update the PR description?

WillAyd · 2025-02-18T15:01:20Z

Ah got it - I've updated the OP

kou

+1

conbench-apache-arrow · 2025-02-19T08:57:04Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit c7a9100.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 6 possible false positives for unstable benchmarks that are known to sometimes produce them.

WillAyd marked this pull request as draft February 6, 2025 00:45

github-actions bot added Component: C++ awaiting review Awaiting review labels Feb 6, 2025

WillAyd force-pushed the meson-builds branch from 54ece5a to 2f7048f Compare February 6, 2025 00:53

WillAyd force-pushed the meson-builds branch 2 times, most recently from c492403 to 61f4bbb Compare February 11, 2025 21:03

WillAyd commented Feb 11, 2025

View reviewed changes

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Feb 11, 2025

kou reviewed Feb 12, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Feb 12, 2025

WillAyd mentioned this pull request Feb 12, 2025

[C++] Minimal Arrow Library Contains Undefined Symbols from IPC sources #45512

Closed

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Feb 12, 2025

WillAyd force-pushed the meson-builds branch 2 times, most recently from b7a25a4 to 78d5fdb Compare February 13, 2025 15:40

WillAyd marked this pull request as ready for review February 13, 2025 20:26

WillAyd added 3 commits February 17, 2025 10:08

apacheGH-41816: [C++] Meson Build System Support

b03b6aa

Build minimal library without options

f18d495

Remove extraneous files

Complete configuration file

400155f

kou requested review from assignUser, jonkeane and raulcd as code owners February 17, 2025 01:40

kou reviewed Feb 17, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 17, 2025

raulcd reviewed Feb 17, 2025

View reviewed changes

Remove comment and sort header installs

428c2f3

github-actions bot removed the awaiting changes Awaiting changes label Feb 18, 2025

WillAyd changed the title ~~GH-41816: [C++] Meson Build System Support~~ GH-41816: [C++] Add Minimal Meson Build of libarrow to CI Feb 18, 2025

github-actions bot added the awaiting change review Awaiting change review label Feb 18, 2025

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 18, 2025

kou approved these changes Feb 19, 2025

View reviewed changes

kou changed the title ~~GH-41816: [C++] Add Minimal Meson Build of libarrow to CI~~ GH-41816: [C++] Add Minimal Meson Build of libarrow Feb 19, 2025

kou merged commit c7a9100 into apache:main Feb 19, 2025
44 of 45 checks passed

kou removed the awaiting changes Awaiting changes label Feb 19, 2025

github-actions bot added the awaiting merge Awaiting merge label Feb 19, 2025

kou mentioned this pull request Feb 19, 2025

[C++] Meson Support for Arrow #41816

Closed

WillAyd deleted the meson-builds branch February 19, 2025 05:06

	git_id = run_command('git', 'log', '-n1', '--format=%H', check: true).stdout().strip()
	git_id = run_command('git', 'log', '-n1', '--format=%H').stdout().strip()

	git_description = run_command('git', 'describe', '--tags', check: true).stdout().strip()
	git_description = run_command('git', 'describe', '--tags').stdout().strip()

		# Meson does not natively support object libraries
		# https://github.com/mesonbuild/meson/issues/13843

GH-41816: [C++] Add Minimal Meson Build of libarrow #45441

GH-41816: [C++] Add Minimal Meson Build of libarrow #45441

Conversation

WillAyd commented Feb 6, 2025 • edited Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

WillAyd commented Feb 6, 2025

kou commented Feb 6, 2025

WillAyd commented Feb 6, 2025

kou commented Feb 7, 2025

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Feb 12, 2025

kou commented Feb 17, 2025

github-actions bot commented Feb 17, 2025

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Feb 18, 2025

kou commented Feb 18, 2025

kou commented Feb 18, 2025

github-actions bot commented Feb 18, 2025

WillAyd commented Feb 18, 2025

kou commented Feb 18, 2025

WillAyd commented Feb 18, 2025

kou left a comment

Choose a reason for hiding this comment

conbench-apache-arrow bot commented Feb 19, 2025

WillAyd commented Feb 6, 2025 •

edited

Loading