wikis2pages

This is a static generator using Hugo to transform GitHub/GitLab's wiki pages (or any similar markdown) into HTML to create documentation websites.

Dependencies

You will need to install ce-dev (which relies on Docker and Docker Compose).

Each repository you want to build a docs website for will also need to have a .wiki2pages.yml file in the root, which contains the variables Ansible needs to generate Hugo's config file (which you can see at ce-dev/ansible/config.toml.j2). There is an example wiki here you can copy:

• https://github.com/codeenigma/wikis2pages-demo

Here is the annotated ce-provision YAML file for reference:

ce-provision-2.x: # path to project within the content and public directories
  src: https://github.com/codeenigma/ce-provision.git # repository containing the Markdown files to convert
  src_branch: 2.x # branch of the repo with the Markdown
  src_subdir: 'docs' # subdirectory of the repo where the Markdown is kept (if applicable)
  # This step requires a 'git push', we script the adding of the token for pushing in our CI (GitHub Actions in this case) directly
  dest: https://github.com/codeenigma/ce-provision-docs.git # destination repo for the generated HTML
  dest_branch: master # branch of the target repo to push the HTML to
  dest_subdir: 2.x # subdirectory of the target repo to save the generated HTML files into (if applicable)
  title: ce-provision # the main page title of the HTML pages
  type: doc # the site 'type' (currently only supports 'doc' or an empty string, 'doc' enables automatic sidebar building)
  base_url: https://codeenigma.github.io/ce-provision-docs/2.x # the eventual URL of the published HTML website, including the 'dest_subdir' if you set one

Once you have ce-dev installed and your repo has the above file, you're ready to get started.

Installing and running wiki2pages

Assuming you have installed ce-dev and you have created a suitable .wiki2pages.yml config file in your target repository pointing to a valid set of Markdown pages to convert to HTML, you can fetch wiki2pages itself. In a suitable location on your computer:

git clone git@github.com:codeenigma/wikis2pages.git
cd wikis2pages
/bin/sh init.sh --repo <my repo> --branch [my branch]

For example, to build the documentation for ce-provision at version 2.x you should use this command to initialise the documentation:

/bin/sh init.sh --repo git@github.com:codeenigma/ce-provision.git --branch 2.x

Behind the scenes this creates variables for your wiki in the ce-dev/ansible directory, starts up ce-dev and runs a ce-dev provision command, which executes the Ansible playbook at ce-dev/ansible/provision.yml in your ce-dev target container. You can now run the included set-up script, generated by the provision step, that configures and allows you to manage multiple wiki2pages projects at once. It is a simple command:

/bin/sh set-current.sh

This script will pull down or update the local copy of the documentation source markdown files, write the variables for the Ansible playbook at ce-dev/ansible/deploy.yml and run a ce-dev deploy command, which executes that playbook in the target container for ce-dev.

If you know the name of the project you want to set to current, you can pass it as a parameter and avoid the project list like this:

/bin/sh set-current.sh --project ce-provision-2.x

If you want to just run Hugo again you can run ce-dev deploy any time.

At this point Hugo's web server will have started on the target container and you should be able to browse your documentation website. See the easy browsing section below for the usage of ce-dev browse.

Where is my HTML?

There will be a public directory that gets generated in the repository root and this is the root of the Hugo web server. To understand where your code will be published you need to look at your .wiki2pages.yml file in the target repo. For example, the name of the project in the file for ce-deploy is ce-deploy-1.x so that will be the directory name in public, because this is the value that will get copied into Hugo's config.toml file at runtime.

However, this will not necesarily be the URL in Hugo to access the files. That will be set according to the base_url variable in the same .wiki2pages.yml file, which is https://codeenigma.github.io/ce-deploy-docs/1.x in the ce-deploy example. The templated hugo-daemon.sh script, which you will find at /opt/hugo-daemon.sh on your ce-dev container, is recreated every time you run set-current.sh or the ce-dev deploy command, as you can see here:

• https://github.com/codeenigma/wikis2pages/blob/master/ce-dev/ansible/deploy.yml

And we can see in the template for the Hugo daemon script that it uses the Jinja2 urlsplit() function to build the path Hugo will serve on, based on the contents of the base_url Ansible variable for the wiki, in this case ce-deploy-docs/1.x.

So to recap, the ce-deploy config file in the repo root, .wiki2pages.yml, looks like this:

ce-deploy-1.x:
  # other vars here...
  base_url: https://codeenigma.github.io/ce-deploy-docs/1.x

Which means the code will be built in the wiki2pages repo under the path public/ce-deploy-1.x/1.x and will be served by Hugo on the URL http://wikis2pages-hugo:4000/ce-deploy-docs/1.x/

Configuring ce-dev for easy browsing

It's possible to make it easier to browse your generated docs by adding URLs to your ce-dev configuration. By default this is configured for our ce-deploy and ce-provision projects, as we have not yet devised a means to automate it:

  urls:
    - http://wikis2pages-hugo:4000/ce-deploy-docs/1.x
    - http://wikis2pages-hugo:4000/ce-provision-docs/2.x

But you can edit the file at wiki2pages/ce-dev/ce-dev.compose.yml and change the list of URLs there. Once that list is correct, run the following to rebuild ce-dev:

ce-dev destroy
ce-dev init
/bin/sh set-current.sh
ce-dev browse

The last command there, ce-dev browse should open the URLs defined automatically in your default browser.

Adding another wiki

If you want to add a second wiki, just run the init.sh script again. For example, if I want to add a docs project for our ce-deploy product at version 1.x I can just execute this command in the wiki2pages repo root:

/bin/sh init.sh --repo git@github.com:codeenigma/ce-deploy.git --branch 1.x

Once I have run that command, if I run set-current.sh again, as above, I will have my ce-deploy docs in the list of options.

Building a site without ce-dev

Under certain circumstances, usually when using CI to automatically build docs, you may not want to use ce-dev, for example when using CI to build documentation. In this case the steps are different. You can look at the CI for ce-provision as an example:

• https://github.com/codeenigma/ce-provision/blob/2.x/.github/workflows/ce-provision-publish-docs.yml#L45

      - name: Initialise wiki2pages for ce-provision 2.x
        run: |
          /usr/bin/su - ce-dev -c "cd /home/ce-dev/build/wiki2pages && /bin/sh init.sh --repo https://github.com/codeenigma/ce-provision.git --branch 2.x --no-ce-dev"
          /usr/bin/su - ce-dev -c "cd /home/ce-dev/build/wiki2pages && /home/ce-dev/ansible/bin/ansible-playbook -e 'wiki2pages_build_path=/home/ce-dev/build/wiki2pages' -i /home/ce-dev/ansible/bin/hosts /home/ce-dev/build/wiki2pages/ce-dev/ansible/provision.yml"
          /usr/bin/su - ce-dev -c "cd /home/ce-dev/build/wiki2pages && /bin/sh set-current.sh --project ce-provision-2.x --no-ce-dev"
          /usr/bin/su - ce-dev -c "cd /home/ce-dev/build/wiki2pages && /home/ce-dev/ansible/bin/ansible-playbook -e 'wiki2pages_build_path=/home/ce-dev/build/wiki2pages launch_hugo_server=false' -i /home/ce-dev/ansible/bin/hosts /home/ce-dev/build/wiki2pages/ce-dev/ansible/deploy.yml"

      - name: Run Hugo
        run: |
          /usr/bin/su - ce-dev -c "cd /home/ce-dev/build/wiki2pages && hugo"
          ls -la /home/ce-dev/build/wiki2pages/public/ce-provision-2.x/

As you can see, we run init.sh normally at first, but with an extra --no-ce-dev flag to prevent it trying to start up ce-dev for us in an environment where it doesn't exist. We then run the equivalent of ce-dev provision manually with Ansible, before running the set-current.sh script which gets generated by the provision.yml playbook. Finally we run theb deploy.yml playbook manually with Ansible and with an important extra variable, launch_hugo_server=false. We do not start Hugo for browsing, because we only want to generate the HTML. This is done in the Run Hugo step where we simply change to the directory where Hugo's config.toml file was created and the other set-up was carried out and run the hugo command. The variables in config.toml should provide everything Hugo needs to build the site without any other input.

Known issues and limitations

GitLab wikis lets you reference pages in relative link by name instead of file path, and interpolates them at rendering time. E.g.

[We are Code Enigma](We are Code Enigma)

This is a valid link in GitLab's wiki syntax and will "magically" be transformed to /we-are-code-enigma in the href (and we-are-code-enigma.md in the background).

However, this is not possible with Hugo, and all links need to be standardised:

[We are Code Enigma](we-are-code-enigma)

This ensures they will work in both places.