Missing dependency between C++ modules and include files

[redstar]

I recently converted a project to use C++ modules and xmake at the same time. I am using LLVM-style .def "database" files. It seems that the dependency between the C++ module and the .def file is not known/ignored when building.

E.g. without modules:

.
├── src
│   ├── Token.cpp
│   └── Token.def
└── xmake.lua

// Token.def
#ifndef TOKEN
#error "Missing TOKEN"
#endif

TOKEN(dot, ".")
TOKEN(arrow, "->")

#undef TOKEN

// Token.cpp
enum class Token {
#define TOKEN(id, name) id,
#include "Token.def"
};

const char *TokenName[] = {
#define TOKEN(id, name) name,
#include "Token.def"
};

-- xmake.lua
add_rules("mode.release")

set_languages("c++20")

target("libtest")
    set_kind("static")
    add_files("src/*.cpp")

This works as expected:

$ xmake config --toolchain=clang
checking for platform ... linux
checking for architecture ... x86_64
$ xmake build
[ 50%]: cache compiling.release src/Token.cpp
[ 75%]: archiving.release liblibtest.a
[100%]: build ok, spent 1.734s
$ touch src/Token.def
$ xmake build
[ 50%]: cache compiling.release src/Token.cpp
[ 75%]: archiving.release liblibtest.a
[100%]: build ok, spent 0.135s

With the conversion to C++ modules I now have the following:

.
├── src
│   ├── Token.cppm
│   └── Token.def
└── xmake.lua

// Token.cppm
export module token;

export {
enum class Token {
#define TOKEN(id, name) id,
#include "Token.def"
};

const char *TokenName[] = {
#define TOKEN(id, name) name,
#include "Token.def"
};
}

-- xmake.lua
add_rules("mode.release")

set_languages("c++20")

target("libtest")
    set_kind("static")
    add_files("src/*.cppm", {public = true})

Now the object file of the module is not rebuild when the .def file changes:

$ xmake config --toolchain=clang
checking for platform ... linux
checking for architecture ... x86_64
$ xmake build
[  0%]: <libtest> generating.module.deps src/Token.cppm
[ 60%]: <libtest> compiling.module.release token
[ 80%]: archiving.release liblibtest.a
[100%]: build ok, spent 1.763s
warning: std and std.compat modules not found! maybe try to add --sdk=<PATH/TO/LLVM> or install libc++
$ touch src/Token.def
$ xmake build
[100%]: build ok, spent 0.023s
warning: std and std.compat modules not found! maybe try to add --sdk=<PATH/TO/LLVM> or install libc++

What do I need to change that the missing dependency is added?

I am using xmake v2.9.7+dev.b029e65ed, clang 19.1.1, Ubuntu 24.10.

Thanks!

[waruqi]

no any cppm file are compiled. you should use set_kind("moduleonly") instead of static kind if no any cpp files.

[redstar]

I am sorry but I do not fully understand. The variable TokenName is defined in the module, therefore imho it must be compiled into an object file to avoid multiple copies. (However, your hint applies to another problem I still have not tackled, so thank you!) Maybe my example is to small. Same happens when I add a function to the module, which certainly must be compiled. There is no change in behaviour when Token.def is changed.

export module token;

export {
enum class Token {
#define TOKEN(id, name) id,
#include "Token.def"
};

const char *getTokenName(Token);
} // of export

const char *TokenName[] = {
#define TOKEN(id, name) name,
#include "Token.def"
};

const char *getTokenName(Token T) {
  return TokenName[static_cast<unsigned>(T)];
}

Original files:
Module (plus other files in the same directory)
def file
xmake.lue

[waruqi]

clang-scan-deps does not provide any dependency information about Token.def.

So xmake can't do anything. It's clang-scan-deps issue, not xmake.

[  0%]: <libtest> generating.module.deps src/test.cppm
/usr/local/Cellar/llvm/19.1.2/bin/clang-scan-deps --format=p1689 -- /usr/local/Cellar/llvm/19.1.2/bin/clang -x c++ -c src/test.c
ppm -o build/.objs/libtest/macosx/x86_64/release/src/test.cppm.o -Qunused-arguments -m64 -isysroot /Applications/Xcode.app/Conte
nts/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX15.1.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c
++20 -DNDEBUG

{
  "revision": 0,
  "rules": [
    {
      "primary-output": "build/.objs/libtest/macosx/x86_64/release/src/test.cppm.o",
      "provides": [
        {
          "is-interface": true,
          "logical-name": "token",
          "source-path": "src/test.cppm"
        }
      ]
    }
  ],
  "version": 1
}

[redstar]

Thank you. I did some more research on this topic. Here is what I found so far:

1. There is no way to express the dependency information for header files in the latest P1689 file. In r4, the fields needed for that were removed. See https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2022/p1689r5.html, 2.2:

Removed inputs, outputs, depends from rules. These were intended to replace depfiles (i.e., -M flags in gcc), but for module dependencies, implementation experience showed that parsing this information was extraneous work for tools that just needed the module dependency information.

clang seems to implements this. As result, the dependency information for headers is missing.

2. The current clang documentation https://clang.llvm.org/docs/StandardCPlusPlusModules.html#discovering-dependencies (at the end of the section) states:

Users may want the scanner to get the transitional dependency information for headers. Otherwise, the project has to be scanned twice, once for headers and once for modules. To address this, clang-scan-deps will recognize the specified preprocessor options in the given command line and generate the corresponding dependency information.

...

When clang-scan-deps detects the -MF option, it will try to write the dependency information for headers to the file specified by -MF.

It seems to me the intention is that the build tool should get the dependencies between modules from the P1689 file, and the dependencies for preprocessor-includes should come from the traditional Makefile dependency files (via options -MD, -MT, -MF). The latter part is missing as seen in the log in your comment. Is there an easy way to combine both approaches?

Btw, I also tried the gcc toolchain (gcc 14.2). Here, the -MD, -MT, -MF options are used on the command line in addition to the module options, however, the results seems not to be used. The gcc toolchain has exactly the same problem - no rebuild when I touch Token.def