feat: Script to create .po files for Transifex Translations (#2041)

* feat: Add a script for creating i18n .po files

i18n .po files are necessary to create Transifex Projects. This script
was originally written by @ghassanmas. This script first turns the docs
into .pot files and then via sphinx-intl turns them into .po files.

* feat: Add python script to edit .tx/config

Transifex only allows uniquely named files. Since we often use similarly
named files, such as `index.rst`, this would replace each file in
transifex with the similarly named one, which is not ideal. Instead, a
script changes the titles of the sections in the config file to make
them unique.

* feat: Add i18n support to Makefile

* feat: Update the requirements to support i18n

Python reqs such as sphinx-intl and transifex-client are necessary to
make the translation files and upload them to transifex.

* fix: Remove CI test

* fix: Add default Project Name, update exit message

* fix: Catch certain common errors

`set -eoux pipefail` was added to catch certain common errors. And the
todo item was removed.

* docs: Add docstring with explanation and example

* fix: Remove x option

The x option prints all lines run in script, making the output
ridiculously messy.

* feat: Sys argv replaced with Click arguments

* fix: Make get more pythonic with fallback

In addition, the script was reformatted as it all falls under the
function now.

* fix: Add maxsplit to limit # of splits

The maxsplit has been set to 1, meaning it splits into 2 parts, only
splitting on the first '.'.

* fix: Add delimiter between dir and file name

The delimiter used to split up the project path is also used to separate
the file name from the delimited project path.

Co-authored-by: Ghassan Maslamani <ghassan.maslamani@gmail.com>

.gitignore

@@ -35,3 +35,6 @@ autodeploy.properties
 
 ### Doc health data
 health.csv
+
+### Translations and Transifex
+.tx/

Makefile

@@ -29,3 +29,6 @@ clean: ## remove built html files
 
 test: ## build all the docs
 	./run_tests.sh
+
+i18n: ## Creates the .pot files and .tx/config file for uploading translation strings to Transifex
+	./i18n.sh

i18n.sh

@@ -0,0 +1,78 @@
+#!/bin/bash
+
+# This script purpose is to ease process needed to support i18n
+# for the doc. It does that by first generating pot from the
+# rst files of a  selected projects using sphinx to generate gettext
+# (which outputs the pot files) and then the next step
+# is to use sphinx-intl to write to the ./tx/config file, hence
+# this script cerates the ./tx/config file. Note: if the file,
+# already exists it will be overwritten!.
+# This script assumes sphinx-intl is already installed as well,
+# It can be installed by `pip install sphinx-intl`
+# For more information, about sphinx-intl please visit:
+# https://www.sphinx-doc.org/en/master/usage/advanced/intl.html
+
+
+set -eou pipefail
+
+
+IFS= read -r -p "Enter the project name as it appears in Transifex[open-edx-documentation-project]: " PROJECT_NAME
+PROJECT_NAME=${PROJECT_NAME:-open-edx-documentation-project}
+
+# Root directory
+BASE_DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
+# Config file
+TX_CONFIG_FILE=$BASE_DIR/.tx/config
+
+# Success  exit message, wrapped in green color.
+EXIT_MESSAGE="\033[0;32mYou should now be able to push the files using tx client \n
+Typically this done by 'tx push -s', please note: you should have \n
+already installed tx client and configured it such that there is \n
+a project called '$PROJECT_NAME' in the organization \n
+and that you already have authenticated with in '~/.transifexrc', \n
+for more  information, please visit: \n
+https://docs.transifex.com/client/client-configuration.\033[0m"
+
+# Preparing the config file:
+rm -rf .tx
+mkdir $BASE_DIR/.tx
+touch $TX_CONFIG_FILE
+echo "[main]" >> $TX_CONFIG_FILE
+echo "host = https://www.transifex.com" >> $TX_CONFIG_FILE
+
+
+# The set of books to run translations for.
+# Currently this is only a subset of the books in this repo because we are in the middle of
+# assesing which books will remain a part of openedx and which ones are actually specific
+# to edx.org
+projects=(
+    "en_us/open_edx_students"
+    "en_us/open_edx_course_authors"
+    "en_us/open_edx_release_notes"
+    )
+
+# Iterating over each project defined above, to generate the pot the files
+# and then mapping the files accordingly in the tx config file
+for project in "${projects[@]}"; do
+    cd $BASE_DIR/$project
+    echo "--> Start generating the pot files for ${project#'en_us/'}"
+
+    # Extract translatable messages into pot files
+    make gettext
+
+    # Adding the pot files to tx config file
+    echo "--> Start writing tx configuration for the ${project#'en_us/'}"
+
+    cd $BASE_DIR
+    sphinx-intl update-txconfig-resources \
+    --pot-dir $BASE_DIR/$project/build/locale/ \
+    --locale-dir $BASE_DIR/$project/locales/ \
+    --transifex-project-name "$PROJECT_NAME"
+
+    python tx_config_fix.py "$project"
+
+done
+
+echo -e ${EXIT_MESSAGE}
+
+

requirements/base.in

@@ -3,3 +3,5 @@
 
 edx-sphinx-theme                    # edX theme for the generated documentation
 sphinx                              # Documentation builder
+sphinx-intl                         # i18n tool to create .po files
+transifex-client                    # Transifex CLI to upload .po files

requirements/base.txt

@@ -2,22 +2,30 @@
 # This file is autogenerated by pip-compile with python 3.8
 # To update, run:
 #
-#    make upgrade
+#    pip-compile --output-file=requirements/base.txt requirements/base.in
 #
 alabaster==0.7.12
     # via sphinx
 babel==2.9.1
-    # via sphinx
+    # via
+    #   sphinx
+    #   sphinx-intl
 certifi==2021.10.8
     # via requests
 charset-normalizer==2.0.10
     # via requests
+click==8.0.4
+    # via sphinx-intl
 docutils==0.16
     # via
     #   -c requirements/common_constraints.txt
     #   sphinx
 edx-sphinx-theme==3.0.0
     # via -r requirements/base.in
+gitdb==4.0.9
+    # via gitpython
+gitpython==3.1.27
+    # via transifex-client
 idna==3.3
     # via requests
 imagesize==1.3.0
@@ -34,18 +42,29 @@ pygments==2.11.2
     # via sphinx
 pyparsing==3.0.7
     # via packaging
+python-slugify==4.0.1
+    # via transifex-client
 pytz==2021.3
     # via babel
 requests==2.27.1
-    # via sphinx
+    # via
+    #   sphinx
+    #   transifex-client
 six==1.16.0
-    # via edx-sphinx-theme
+    # via
+    #   edx-sphinx-theme
+    #   transifex-client
+smmap==5.0.0
+    # via gitdb
 snowballstemmer==2.2.0
     # via sphinx
 sphinx==4.4.0
     # via
     #   -r requirements/base.in
     #   edx-sphinx-theme
+    #   sphinx-intl
+sphinx-intl==2.0.1
+    # via -r requirements/base.in
 sphinxcontrib-applehelp==1.0.2
     # via sphinx
 sphinxcontrib-devhelp==1.0.2
@@ -58,7 +77,16 @@ sphinxcontrib-qthelp==1.0.3
     # via sphinx
 sphinxcontrib-serializinghtml==1.1.5
     # via sphinx
+text-unidecode==1.3
+    # via python-slugify
+transifex-client==0.14.4
+    # via -r requirements/base.in
 urllib3==1.26.8
-    # via requests
+    # via
+    #   requests
+    #   transifex-client
 zipp==3.7.0
     # via importlib-metadata
+
+# The following packages are considered to be unsafe in a requirements file:
+# setuptools

requirements/dev.txt

@@ -2,7 +2,7 @@
 # This file is autogenerated by pip-compile with python 3.8
 # To update, run:
 #
-#    make upgrade
+#    pip-compile --output-file=requirements/dev.txt requirements/dev.in
 #
 alabaster==0.7.12
     # via
@@ -12,6 +12,7 @@ babel==2.9.1
     # via
     #   -r requirements/base.txt
     #   sphinx
+    #   sphinx-intl
 certifi==2021.10.8
     # via
     #   -r requirements/base.txt
@@ -20,14 +21,25 @@ charset-normalizer==2.0.10
     # via
     #   -r requirements/base.txt
     #   requests
-click==8.0.3
-    # via pip-tools
+click==8.0.4
+    # via
+    #   -r requirements/base.txt
+    #   pip-tools
+    #   sphinx-intl
 docutils==0.16
     # via
     #   -r requirements/base.txt
     #   sphinx
 edx-sphinx-theme==3.0.0
     # via -r requirements/base.txt
+gitdb==4.0.9
+    # via
+    #   -r requirements/base.txt
+    #   gitpython
+gitpython==3.1.27
+    # via
+    #   -r requirements/base.txt
+    #   transifex-client
 idna==3.3
     # via
     #   -r requirements/base.txt
@@ -64,6 +76,10 @@ pyparsing==3.0.7
     # via
     #   -r requirements/base.txt
     #   packaging
+python-slugify==4.0.1
+    # via
+    #   -r requirements/base.txt
+    #   transifex-client
 pytz==2021.3
     # via
     #   -r requirements/base.txt
@@ -72,10 +88,16 @@ requests==2.27.1
     # via
     #   -r requirements/base.txt
     #   sphinx
+    #   transifex-client
 six==1.16.0
     # via
     #   -r requirements/base.txt
     #   edx-sphinx-theme
+    #   transifex-client
+smmap==5.0.0
+    # via
+    #   -r requirements/base.txt
+    #   gitdb
 snowballstemmer==2.2.0
     # via
     #   -r requirements/base.txt
@@ -84,6 +106,9 @@ sphinx==4.4.0
     # via
     #   -r requirements/base.txt
     #   edx-sphinx-theme
+    #   sphinx-intl
+sphinx-intl==2.0.1
+    # via -r requirements/base.txt
 sphinxcontrib-applehelp==1.0.2
     # via
     #   -r requirements/base.txt
@@ -108,12 +133,19 @@ sphinxcontrib-serializinghtml==1.1.5
     # via
     #   -r requirements/base.txt
     #   sphinx
+text-unidecode==1.3
+    # via
+    #   -r requirements/base.txt
+    #   python-slugify
 tomli==2.0.0
     # via pep517
+transifex-client==0.14.4
+    # via -r requirements/base.txt
 urllib3==1.26.8
     # via
     #   -r requirements/base.txt
     #   requests
+    #   transifex-client
 wheel==0.37.1
     # via pip-tools
 zipp==3.7.0

tx_config_fix.py

@@ -0,0 +1,53 @@
+from configparser import ConfigParser
+
+import click
+
+
+@click.command()
+@click.argument("project_path")
+def delimit_sections(project_path):
+    """
+    This script changes the section titles of the tx config file to be more unique based
+    on the directory of the file that is used to make the original section title.
+
+    The script loops through the sections of the tx config and changes the sections that
+    match the project_path argument variable. The script then adds a delimited version of
+    the directory to the section title making the title more unique. These section titles
+    become the file names on Transifex, and if they are not unique, they overwrite
+    eachother.
+
+    Example:
+    If we had a file "example/index.rst" that we wanted to translate, in tx config we
+    would have a section with the title [index] and the source_file: "example/index.rst".
+    But this filename is common in Sphinx Projects. This script would rename the section
+    title to be [example--title], keeping all other values under it untouched.
+    """
+    delimiter = "--"
+    project_path_friendly = project_path.replace("/", delimiter)
+
+    config_path = "./.tx/config"
+
+    config = ConfigParser()
+    config.read(config_path)
+
+    for section_title in config.sections():
+        source_file_path = config.get(section_title, "source_file", fallback=None)
+        if source_file_path and source_file_path.startswith(project_path):
+            split_section_title = section_title.split(".", maxsplit=1)
+            split_section_title[1] = (
+                project_path_friendly + delimiter + split_section_title[1]
+            )
+            new_section_title = ".".join(split_section_title)
+            config.add_section(new_section_title)
+            for option_name in config.options(section_title):
+                config.set(
+                    new_section_title, option_name, config[section_title][option_name]
+                )
+            config.remove_section(section_title)
+
+    with open(config_path, "w") as file:
+        config.write(file)
+
+
+if __name__ == "__main__":
+    delimit_sections()