How to test kiwix-lib/kiwix-android integration ?

[Aditya-Sood]

I'm working on fixing kiwix/kiwix-android/#765 for which I had to compile kiwix-lib into the app, which I was told required using kiwix-build

During setup, upon running kiwix-build kiwix-lib -v, I obtained the following output:

.
.
.
[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[==========] Running 1 test from 1 test case.
[==========] Running 7 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 8 tests from DirentTest
[----------] 3 tests from UuidTest
[----------] Global test environment set-up.
[----------] 5 tests from ClusterTest
[ RUN      ] DirentTest.set_get_data_dirent
[ RUN      ] UuidTest.construct
[----------] 1 test from HeaderTest
[ RUN      ] ClusterTest.create_cluster
[       OK ] DirentTest.set_get_data_dirent (0 ms)
[       OK ] UuidTest.construct (0 ms)
[ RUN      ] HeaderTest.read_write_header
[       OK ] ClusterTest.create_cluster (0 ms)
[ RUN      ] DirentTest.read_write_article_dirent
[ RUN      ] UuidTest.generate
[       OK ] HeaderTest.read_write_header (0 ms)
[ RUN      ] ClusterTest.read_write_cluster
[       OK ] DirentTest.read_write_article_dirent (0 ms)
[----------] 1 test from HeaderTest (1 ms total)

[       OK ] ClusterTest.read_write_cluster (0 ms)
[ RUN      ] DirentTest.read_write_article_dirent_unicode
[----------] Global test environment tear-down
[ RUN      ] ClusterTest.read_write_empty
[       OK ] DirentTest.read_write_article_dirent_unicode (0 ms)
[==========] 1 test from 1 test case ran. (2 ms total)
[  PASSED  ] 1 test.
[       OK ] ClusterTest.read_write_empty (0 ms)
[ RUN      ] DirentTest.read_write_redirect_dirent
[ RUN      ] ClusterTest.read_write_clusterZ
[       OK ] DirentTest.read_write_redirect_dirent (0 ms)
[ RUN      ] DirentTest.read_write_linktarget_dirent
[       OK ] ClusterTest.read_write_clusterZ (2 ms)
[       OK ] DirentTest.read_write_linktarget_dirent (0 ms)
[ RUN      ] ClusterTest.read_write_clusterLzma
[ RUN      ] DirentTest.read_write_deleted_dirent
[       OK ] DirentTest.read_write_deleted_dirent (0 ms)
[ RUN      ] DirentTest.dirent_size
[       OK ] DirentTest.dirent_size (0 ms)
[ RUN      ] DirentTest.redirect_dirent_size
[       OK ] ClusterTest.read_write_clusterLzma (37 ms)
[       OK ] DirentTest.redirect_dirent_size (0 ms)
[----------] 5 tests from ClusterTest (91 ms total)

[----------] 8 tests from DirentTest (91 ms total)

[----------] 2 tests from CluterTest
[----------] Global test environment tear-down
[ RUN      ] CluterTest.read_write_extended_cluster
[==========] 8 tests from 1 test case ran. (93 ms total)
[  PASSED  ] 8 tests.
[==========] Running 3 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 3 tests from TemplateTest
[ RUN      ] TemplateTest.ZeroTemplate
[       OK ] TemplateTest.ZeroTemplate (0 ms)
[ RUN      ] TemplateTest.Token
[       OK ] TemplateTest.Token (0 ms)
[ RUN      ] TemplateTest.Link
[       OK ] TemplateTest.Link (0 ms)
[----------] 3 tests from TemplateTest (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1 ms total)
[  PASSED  ] 3 tests.
[       OK ] UuidTest.generate (1000 ms)
[ RUN      ] UuidTest.output
[       OK ] UuidTest.output (0 ms)
[----------] 3 tests from UuidTest (1001 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 1 test case ran. (1002 ms total)
[  PASSED  ] 3 tests.
cluster time out (After 480 seconds)
1/5 cluster                                 TIMEOUT 488.94 s 
2/5 dirent                                  OK       0.74 s 
3/5 header                                  OK       0.72 s 
4/5 uuid                                    OK       1.65 s 
5/5 template                                OK       0.05 s 

Ok:                    4
Expected Fail:         0
Fail:                  1
Unexpected Pass:       0
Skipped:               0
Timeout:               1


Stopping build due to errors

The entire output of the command is here

There was a test libzim : run command 'meson test --verbose -t 8' line in the middle of that output, running which gave this error:

Only ninja backend is supported to rebuild tests before running them.
Meson test encountered an error:

Directory $'.../Kiwix/kiwix-build' does not seem to be a Meson build directory.

I'm guessing this is related to the timeout bit in #188

I am basically looking for a way to test the changes I have made to kiwix-lib (commit) & kiwix-android (commit)

Any help is much appreciated!

[Aditya-Sood]

@mgautierfr could you please have a look at this? I've been stuck on this for a week

[mgautierfr]

What is you system ?

---

It will not fix the issue, but if you what a library for android, you must use the following command to build kiwix-lib for android :
kiwix-build --target-platform android_<yourarch> kiwix-lib
Else, you will simply compile kiwix-lib for your system (and it is not android).
To build kiwix-android apk with you custom lib, you should see with @macgills. He recently (or will) change the kiwix-android build system and it (will) takes the kiwix-lib from the jfrog repository.

[macgills]

@Aditya-Sood this is already the case, you should not have to use kiwix-build.

  if (file("../kiwixlib/src/main").list().length == 1) {
    implementation 'org.kiwix.kiwixlib:kiwixlib:1.0.12'
  } else {
    implementation project(":kiwixlib")
    archs = file("../kiwixlib/src/main/jniLibs").list()
  }

really I should just delete kiwixlib folder (I'll make a ticket) from the android project but the important line is this
implementation 'org.kiwix.kiwixlib:kiwixlib:1.0.12' , from checking on bintray you can check the latest version is actually 5.1.0 but because of adhoc versioning I think these artefacts are largely similar?
If you need bleeding edge versions of kiwix-lib then hopefully Matthieu and I will resolve snapshot uploads soon but that will require your changes be merged to master and for a nightly build to happen.
I am not sure of the best strategy for in dev testing of library changes but snapshots could be the way?

[mgautierfr]

This is bit more than bleeding edge versions. This is custom version.
Here it seems that @Aditya-Sood want to change a thing in the kiwix-lib and try it in kiwix-android.
We want to do this without needing to upload a artefact on bintray or whatever. We must me able to do this locally without passing by travis/github/bintray...

[macgills]

I view this as much like a release of a new api. In previous roles an api update had to go through testing/review on another project and go through ci before viable to be pushed to staging (here snapshot) while my integration branch of the app was updated based on the given spec and tested with mocked responses and would await a staging release. It is not the quickest but is good process.

That was a much more senior team thought, I am not sure of Aditya's experience with kiwixlib and how complex the changes are. A suite of unit tests would really give confidence here.

For now Aditya can run the command you suggested kiwix-build --target-platform android_<yourarch> kiwix-lib and we can think on our preferred method of work going forward

[mgautierfr]

What you say is totally exact. An API update should got through a CI build/testing before being push as new release/snapshot.

But while I'm developing (fix bug or changing api), I want to be able to test my new lib version without passing through a CI. The command I give will only build kiwix-lib for the correct android arch. But (s)he will have to test kiwix-android with this library.
For now, kiwix-build (using kiwix-build --target-platform android --android-arch <yourarch> kiwix-android) handle this and create a apk with the kiwix-lib just compiled. But we are just about to remove this feature (#344, kiwix/libkiwix#231).

I don't know what the best way to do it with gradle but it seems necessary to be able to locally build an apk with a locally build library without having to push the lib somewhere on a server.

[macgills]

If we can produce kiwixlib with the .so files in the jniLib dir then we can copy and paste it into the android project and it will pick it up and use it instead of the remote dependency.

There should be no necessity for the android project to build kiwixlib itself

[mgautierfr]

Well, if you do this (really I should just delete kiwixlib folder (I'll make a ticket) from the android project ...) it will not be possible no ? Or I've missed something ?

[macgills]

Yeah, I have yet to make the ticket and now it seems pointless.

If we have no way of assuring ourselves of the functionality of kiwix-lib outside of integrating it with kiwix-android then there is no other way

[macgills]

Or your build task can ouput the .aar file and that is pretty easy to work with on the android side

[mgautierfr]

Yes work with the .aar is probably better. As far as it is possible to build the apk with a local aar.

[macgills]

Xpost:
Then I would propose a test project in kiwixlib if only to run automated tests or something similar. A full integration with kiwix-android should not be necessary to make sure additions to a separate project work.
If this gets integrated with ci then we can independently validate on each PR. I am assuming unit tests are all but impossible

[mgautierfr]

I've just recreate an issue for the original issue (very few things have been discuss about it here).
I was about to rename this issue to "How to test kiwix-lib/kiwix-android integration without publishing kiwix-lib to a server ?" but the last part is somehow a beginning of response to the question.
I've have personal pov on this (not publish on a server) but I'm open to a good argumentation from @macgills 😉

[macgills]

After more thought my proposal is now:
We create a simple Android app module in kiwix-lib-publisher that does only the integration with kiwixlib. I still feel that having to go all the way to kiwix-android to debug development is too much.
We use it for automated tests so the CI can verify kiwix-lib changes or build it locally.
What are the barriers (if there are any) to kiwix-lib assembling the .sos itself, copying them to lib-publisher along with the java files on a local build command?

[kelson42]

I have read the comments and discuss a bit with @mgautierfr and @mhutti1. They are a few things I want to avoid:

• Forcing a libkiwix release to then test the Kiwix-Android app
• Having to use Bintray to test cutting-edge/pre-release libkiwix code
• Having code in kiwix-build applying custom libkiwix in kiwix-android
• Not in favour either of having Android code/test app in libkiwix

I believe that assuming that we can release libkiwix without testing integration in kiwix-android is not the most optimal approach. So, I think I quite disagree with @macgills here, if I have understand him well.

Here is how I propose to solve that problem:

• libkiwix/kiwix-build is responsible and can already compile easily the .aar file
• We just need a small documentation/small script/gradle buildtype in kiwix-android repo to use this local file in place of the bintray one
• kiwix-build should not be involved in joining libkiwix and kiwix-android... but it has or had the code to force kiwix-android using a local libkiwix, so this piece of code might be reused!?

I'm also quite in favour of publishing, through libkiwix CI, all new git master HEAD of libkiwix to a bintray dev version. That way anybody could use a cutting edge version of the libkiwix without having to compile the libkiwix himself. This would be complementary of what I have proposed earlier. This is the purpose of #345 and I also support it, but not as main solution to solve this ticket.

[kelson42]

After discussing with @mhutti1, it looks like we can share the location of the .aar file via a Gradle argument. This would easily allow to deal with custom .aar file via command line and Android Studio. We would just need to briefly update the README.md. So no dedicated script.

[mhutti1]

Is it possible to build an aar without an application project though?

[macgills]

Yes, an aar is built for a library module so that is no problem.

I believe that assuming that we can release libkiwix without testing integration in kiwix-android is not the most optimal approach.

I am of the opinion that each project should be capable of validation independent of another project. This would speed up development I believe.
After making an update to lib it should be possible to write a unit test(s) or run a small automation test to guarantee that your functionality change behaves appropriately. Using kiwix-android as a validator for lib does not seem to be the most productive to me.
Most other android libraries would have a "sample" app for demo purposes, I am unsure if they use it for testing and the like but it would seem like the quickest feedback loop to me.

I may be unaware of the types of errors that occur during integration but I think this solution should cover them

[kelson42]

@macgills Quite agree with your comment. I have created https://github.com/kiwix/kiwix-lib/issues/249 as an attempt to automated test the .aar properly on the top (or as part) of libkiwix automated testing solution. We should keep/implement this current ticket to have an easy solution for developers to deal with cutting edge versions of the libkiwix .aar. Hope that approach satisfies you.

[Aditya-Sood]

@mgautierfr

What is you system ?

I'm using ubuntu 18.04, and my phone is moto g5s+ (uses snapdragon 625, which is arm afaik)

It will not fix the issue, but if you what a library for android, you must use the following command to build kiwix-lib for android :
kiwix-build --target-platform android_ kiwix-lib
Else, you will simply compile kiwix-lib for your system (and it is not android).

But wouldn't this use the kiwix-lib in its SOURCE directory, instead of the kiwix-lib repo I'm working with?

[Aditya-Sood]

@macgills

@Aditya-Sood this is already the case, you should not have to use kiwix-build.

  if (file("../kiwixlib/src/main").list().length == 1) {
    implementation 'org.kiwix.kiwixlib:kiwixlib:1.0.12'
  } else {
    implementation project(":kiwixlib")
    archs = file("../kiwixlib/src/main/jniLibs").list()
  }

So essentially I build kiwix-lib for my platform and then replace the kiwixlib directory in kiwix-android?

[macgills]

@Aditya-Sood that is the current solution yes.
In the future we hope to have a more contained solution

[mhutti1]

@Aditya-Sood My steps:

Implement the gradle changes in: kiwix/kiwix-android#1348

Build kiwix lib using kiwix-build.
kiwix-build kiwix-lib-app --target-platform android
ln kiwix-build/BUILD_android/kiwix-lib-app/kiwixLibAndroid/build/outputs/aar kiwix-android/app/libs

Changes the files in SOURCE/kiwix-lib and recompile and re sync to get the new changes.

[Aditya-Sood]

Building kiwix-lib works for android_arm (failing for android because of:

chmod: cannot operate on dangling symlink 'ss_err.c'
Makefile:436: recipe for target 'ss_err.h' failed
make[1]: *** [ss_err.h] Error 1
make[1]: *** Waiting for unfinished jobs....

Also, for the link command:
ln kiwix-build/BUILD_android/kiwix-lib-app/kiwixLibAndroid/build/outputs/aar kiwix-android/app/libs
How will it be modified for BUILD_android_arm? Couldn't find a kiwixLibAndroid directory in it (has kiwix-lib though)

[mhutti1]

I am not sure about the crash. @mgautierfr might know more.
With regards to the ln command. The directory will exist once the build completes.

[Aditya-Sood]

@mhutti1 I think it's just kiwix-lib, so: kiwix-build kiwix-lib --target-platform android
This time it ran without error (got a [CLEAN]) at the end, but there is still no folder in BUILD_android other than INSTALL, LOGS, and TOOLCHAINS

[mhutti1]

It is not just kiwix-lib. It is definitely the command I told you. If you just build kiwix-lib you won't build any of the android wrapper.

[Aditya-Sood]

Running the command that you specified gives this error:

kiwix-build: error: argument TARGET: invalid choice: 'kiwix-lib-app' (choose from 'alldependencies',
'android-ndk', 'android-sdk', 'aria2', 'armhf', 'org.kde', 'gradle', 'gumbo', 'icu4c', '_ios_fat_lib',
'mustache', 'kiwix-android', 'kiwix-android-custom', 'kiwix-desktop', 'kiwix-lib', 'kiwix-tools', 'libcurl',
'libmagic', 'libmicrohttpd', 'libzim', 'lzma', 'qt', 'qtwebengine', 'pugixml', 'uuid', 'xapian-core', 'zim-tools',
'zimwriterfs', 'zlib')

Maybe it's one of the other choices?

[mhutti1]

Are you sure you have the latest version of kiwix-build. It 100% works for me. You must not have followed a step in the instructions. Have you done pip install . and it has worked and installed kiwix-build.

My targets are:
kiwix-build: error: argument TARGET: invalid choice: 'bob' (choose from 'alldependencies', 'android-ndk', 'android-sdk', 'aria2', 'armhf', 'org.kde', 'gradle', 'gumbo', 'icu4c', '_ios_fat_lib', 'mustache', 'kiwix-android', 'kiwix-android-custom', 'kiwix-desktop', 'kiwix-lib', 'kiwix-lib-app', 'kiwix-tools', 'libcurl', 'libmagic', 'libmicrohttpd', 'libzim', 'lzma', 'qt', 'qtwebengine', 'pugixml', 'uuid', 'xapian-core', 'zim-tools', 'zimwriterfs', 'zlib')

[Aditya-Sood]

So pip install . has been running for over 20 minutes now, I'm guessing it takes time. Will update post its completion

[mhutti1]

Make sure to use the --upgrade and --user flags as needed.

[Aditya-Sood]

I'll run it again

[Aditya-Sood]

@mhutti1 so I got the following error with pip:

Collecting meson==0.49.2 (from kiwix-build==0.1)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/42/a1/e5b17fe46cd6060edb49f776eecb31c9527a1c3742bbb60630ee439a6e22/meson-0.49.2.tar.gz (1.3MB)
    100% |████████████████████████████████| 1.3MB 440kB/s 
    Complete output from command python setup.py egg_info:
    Tried to install with an unsupported version of Python. Meson requires Python 3.5.0 or greater
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-YL9YLj/meson/

Resolved by using pip3 instead: pip3 install . --upgrade --user. Got a

Successfully built kiwix-build
Installing collected packages: meson, kiwix-build
Successfully installed kiwix-build-0.1 meson-0.49.2

Followed the other steps in the readme, still getting the same error (about kiwix-lib-app not being a valid choice)

[Aditya-Sood]

Is it possible that kiwix-build needs python3 and is using python (2) instead? I think my system uses 2 internally, while I downloaded 3 later

[mhutti1]

What does which kiwix-build say? Have you read the README? It says that you need python 3. What is the commit hash of your directory?

[Aditya-Sood]

Sorry for the trouble, I copied the lines for execution wrong (was using python instead of python3)

I'm now getting a: FileNotFoundError: [Errno 2] No such file or directory: '.../kiwix-build/SOURCE/kiwix-lib/android-kiwix-lib-publisher'
Didn't find it in any of the previous commits, so going to wipe the local repo and start over

[kelson42]

@mhutti1 @macgills Please write the documentation corresponding to kiwix/kiwix-android#1348 in Kiwix-Andoird CONTRIBUTING.md. No building "magic" without proper developer documentation. I want also to close that PR as this PR has solved it. If @Aditya-Sood can not compile the libkiwix with kiwix-build, I would like a proper dedicated ticket.

[Aditya-Sood]

@kelson42 I had most probably caused the problem myself by copying one of the commands incorrectly, and I didn't pay attention to the new issue @mgautierfr had created to help me out - I'll ask for help on the new issue in case I face problems. Thanks a lot

[mhutti1]

Documentation is kiwix/kiwix-android#1313

[kelson42]

@Aditya-Sood @macgills I think we should/close that ticket as the situation is now clear and solution is documented. Do you?

[Aditya-Sood]

@kelson42 @macgills just one query: when you add the local, updated kiwix-lib repo in the SOURCE directory of kiwix-build, it automatically checks out the master branch of this local copy of kiwix-lib repo during the building process (irrespective of what you had checked out before). Maybe this is the expected behaviour, but I had made my changes on a separate branch from master (as is the convention) and so repeatedly ended up with a kiwix-lib based on master branch (and not my checked-out branch).
The fix I used was to merge my changes directly into master before copying the kiwix-lib repo into SOURCE; and then undo the merge after copying.

So should I add a line somewhere saying that the kiwix-lib branch with the changes should be merged into master before copying it into SOURCE, so that kiwix-build uses the updated code for creating the kiwix-lib .aar?

[kelson42]

@Aditya-Sood The problem you report is not already reported in another ticket of this repo?

[kelson42]

@Aditya-Sood The scenario you commented about is a bug in kiwix-build, see for example #378. This should not be documented. This should be fixed.

[Aditya-Sood]

@kelson42 my problem was slightly different in the sense that I didn't want it to checkout the master branch (like it does right now, based on this comment), instead I wanted it to use the currently checked out non-master branch on which I had made the changes.

I got around this by temporarily merging my changes to the local master and then copying the repository (so that when the master was checked out during building of kiwix-lib, it already had the updated code).

So I was thinking it might be useful to mention somewhere in the readme that the changes to kiwix-lib need to be merged into master before copying the local kiwix-lib repository to the SOURCE directory of kiwix-build.

[kelson42]

@Aditya-Sood I have created a ticket #395

[Aditya-Sood]

Thanks! I'll give a gist of the problem there