Adding Red Queen CLI

README.md

@@ -1,63 +1,124 @@
-> "Well, in our country," said Alice, still panting a little, "you'd generally 
-> get to somewhere else—if you run very fast for a long time, as we've been 
+> "Well, in our country," said Alice, still panting a little, "you'd generally
+> get to somewhere else—if you run very fast for a long time, as we've been
 > doing."
 >
-> "A slow sort of country!" said the Queen. "Now, here, you see, it takes all 
-> the running you can do, to keep in the same place. If you want to get 
+> "A slow sort of country!" said the Queen. "Now, here, you see, it takes all
+> the running you can do, to keep in the same place. If you want to get
 > somewhere else, you must run at least twice as fast as that!"
 >
 > [Carroll, Lewis: Through the Looking-Glass, Chapter 2](
     https://www.gutenberg.org/files/12/12-h/12-h.htm)
 
-## About
+<br>
+
+<h1>About</h1>
+
 The Red Queen benchmark framework was created to facilitate the benchmarking
 algorithms used in quantum compilation.
 
-The framework is tightly integrated into `pytest`.  Therefore, to use it 
-effectively, you should know the basics of `pytest` first. Take a look at the 
+The framework is tightly integrated into `pytest`.  Therefore, to use it
+effectively, you should know the basics of `pytest` first. Take a look at the
 [introductory material](https://docs.pytest.org/en/latest/getting-started.html).
 
-## Usage
+
+<br>
+
+<h1>Usage</h1>
+
+
 Red Queen is a framework for benchmarking quantum compilation algorithms. Since
-it was not designed as a package, there is no notion of installation. Hence, you
-must clone this repository to use it:
+is is still in early development, you must clone this repository to use it:
+
+
 ```bash
 git clone git@github.com:Qiskit/red-queen.git
 ```
 
-To run benchmarks, you must first go to the `red-queen` directory and install 
+To run benchmarks, you must first go to the `red-queen` directory and install
 the required packages:
+
+
 ```bash
 cd red-queen
-pip install -r requirements.txt
+pip install red-queen
 ```
 
+
+Red Queen has a `CLI` (command line interface) that you can use to execute benchmarks. 
+
+<br>
+
+The general templete for the `CLI` is as follows:
+
+<br>
+
+
+```bash
+red-queen -c <complier> -t <benchmark_type> -b <benchmark_name>
+```
+
+
 Now, suppose you want to run the mapping benchmarks using only `tweedledum`.
-You must do it using `pytest`
+You can do this via the `CLI` or with `pytest`
+
+
+<br>
+
+[For MacOs and Linux]
+
+With `CLI`
+
+```bash
+red-queen -c tweedledum -t mapping -b map_queko.py
+```
+
+With `pytest`
+
+```bash
+python -m pytest games/mapping/map_queko.py -m tweedledum --store
+```
+
+<br>
+
+[For Windows]
+
+With `CLI`
+
+```bash
+red-queen -c tweedledum -t mapping -b map_queko.py
+```
+
+With `pytest`
+
 ```bash
-pytest red_queen/games/mapping/map_queko.py -m tweedledum --store
+python -m pytest -s games/mapping/map_queko.py -m tweedledum --store
 ```
 
-To run pytest on Windows, you will have to use `python -m` in order to run the 
-`pytest` command. You will also need to add `-s` to your pytest call to disable 
+To run pytest or any python script on Windows, you will have to use `python -m` in order to run the
+`pytest` command. You will also need to add `-s` to your pytest call to disable
 stdin handling.
+
 ```bash
-python -m pytest -s red_queen/games/mapping/map_queko.py -m tweedledum --store
+python -m pytest -s games/mapping/map_queko.py -m tweedledum --store
 ```
+<br>
 
-The benchmark suite will consider all functions named `bench_*` in 
-`red_queen/games/mapping/map_queko.py`. Because we set the `-m` option, only the the ones
-marked with `tweedledum` will be run. (We could easy do the same for `qiskit`).
-If you don't define a `-m` option, all `bench_*` functions will be run.
 
+
+The benchmark suite will consider all functions named `bench_*` in
+`games/mapping/map_queko.py`. Because
 The `--store` option tells the framework to store the results in json file in
 the `results` directory. To see the results as a table, you can use the you can
 use:
+
 ```bash
 python -m report.console_tables --storage results/0001_bench.json
 ```
 
+<br>
+
 ## Warning
+
 This code is still under development. There are many razer sharp edges.
 
 For information of how execution works and other details about the framwork
@@ -73,6 +134,7 @@ on the knowledge of the internals of the following established `pytest` plugins:
 
 ## License
 
+
 This software is licensed under the Apache 2.0 licence (see 
 [LICENSE](https://github.com/Qiskit/red-queen/blob/main/LICENSE))
 

red_queen/cli.py

@@ -0,0 +1,185 @@
+# ------------------------------------------------------------------------------
+# Part of Red Queen Project.  This file is distributed under the Apache 2.0.
+# See accompanying file /LICENSE for details.
+# ------------------------------------------------------------------------------
+"""Shebang is used to make this code an python executable"""
+#!/usr/bin/env python3
+import os
+import platform
+import subprocess
+import configparser
+import click
+
+
+def benchmark_retrieval():
+    benchmark_dict = {}
+    type_list = []
+    list_of_benchmarks = []
+    dir_path = "red_queen/games/"
+    for entry in os.scandir(dir_path):
+        if entry.is_dir():
+            benchmark_dict[entry.name] = []
+            sub_dict = {}
+            for sub in os.scandir(f"{dir_path}{entry.name}"):
+                if not sub.name.startswith("_") and sub.name.endswith(".py") and sub.is_file():
+                    sub_dict[sub.name] = sub.path
+            benchmark_dict[entry.name] = sub_dict
+
+    type_list = list(benchmark_dict.keys())
+    for benchmark_pairs in benchmark_dict.values():
+        for keys in benchmark_pairs.keys():
+            list_of_benchmarks.append(keys)
+
+    return benchmark_dict, type_list, list_of_benchmarks
+
+
+def complier_retrieval():
+    list_of_compliers = []
+    config = configparser.ConfigParser()
+    config.read("pytest.ini")
+    for complier in config["pytest"]["markers"].split("\n"):
+        if complier != "":
+            list_of_compliers.append(complier)
+    return list_of_compliers
+
+
+def result_retrieval():
+    results = {}
+    dir_path = "results"
+    for entry in os.scandir(dir_path):
+        if entry.is_file():
+            filename = entry.name
+            result_count = int(filename.split("_")[0])
+            results[result_count] = {entry.name: entry.path}
+    return results
+
+
+def run_benchmarks(pytest_paths: str, m_tag: str, compiler: str):
+    command_list = ["pytest"]
+    compiler_command = [m_tag, compiler, "--store"]
+    if platform.system() == "Windows":
+        command_list.insert(0, "-m")
+        command_list.insert(0, "python")
+        command_list.append("-s")
+        for _, string in enumerate(pytest_paths):
+            command_list.append(string)
+        for _, string in enumerate(compiler_command):
+            command_list.append(string)
+        subprocess.run(
+            command_list,
+            check=True,
+        )
+    else:
+        for _, string in enumerate(pytest_paths):
+            command_list.append(string)
+        for _, string in enumerate(compiler_command):
+            command_list.append(string)
+        subprocess.run(
+            command_list,
+            check=True,
+        )
+
+
+def show_result():
+    results_dict = result_retrieval()
+    result_num = max(results_dict.keys())
+    result_path = tuple(results_dict[result_num].values())[0]
+    command_list = ["python3", "-m", "report.console_tables", "--storage"]
+    command_list.append(str(result_path))
+    subprocess.run(
+        command_list,
+        check=True,
+    )
+    click.echo("To view the table again:")
+    click.echo(" ".join(command_list))
+
+
+benchmark_category, benchmark_types, benchmarks = benchmark_retrieval()
+complier_list = complier_retrieval()
+
+
+@click.command()
+# @click.option("--version", action="version", version="%(prog)s 0.0.1")
+@click.option(
+    "-c",
+    "--compiler",
+    "compiler",
+    multiple=True,
+    type=click.Choice(complier_list),
+    help="enter a compliler here",
+)
+@click.option(
+    "-t",
+    "--benchmarkType",
+    "benchmarkType",
+    multiple=True,
+    type=click.Choice(benchmark_types),
+    help="enter the type of benchmark(s) here",
+)
+@click.option(
+    "-b",
+    "--benchmark",
+    "benchmark",
+    multiple=True,
+    type=click.Choice(benchmarks),
+    help="enter the specfic benchmark(s) here",
+)
+def main(compiler=None, benchmarkType=None, benchmark=None):
+    benchmark_paths = []
+    pytest_paths = ""
+    mydict = {}
+    m_tag = ""
+    if len(compiler) > 0:
+        m_tag = "-m"
+        compiler = compiler[0]
+    else:
+        compiler = ""
+    # ### This for loop ensures that we are able to run various benchmark types
+    i = 0
+    j = 0
+    ### Has a benchmark type been specified?
+    if len(benchmarkType) > 0:
+        # click.echo("passed test 0")
+        while i < len(benchmarkType):
+            ### Is the inputted benchmark type valid?
+            if set(benchmarkType).issubset(benchmark_category):
+                # click.echo("passed test 1")
+                ### Has a benchmark been specified?
+                if len(benchmark) > 0:
+                    # click.echo("passed test 2")
+                    ### Are the inputted benchmark(s) valid?
+                    if set(benchmark).issubset(benchmarks):
+                        # click.echo("passed test 3")
+                        ### Is the inputted benchmark within the inputted benchmark type suite?
+                        if set(benchmark).issubset(set(benchmark_category[benchmarkType[i]])):
+                            # click.echo("passed test 4")
+                            for j, _ in enumerate(benchmark):
+                                benchmark_paths.append(
+                                    benchmark_category[benchmarkType[0]][benchmark[j]]
+                                )
+                            pytest_paths = benchmark_paths
+                            run_benchmarks(pytest_paths, m_tag, compiler)
+                            # show_result()
+                            i += 1
+                else:
+                    mydict = benchmark_category[benchmarkType[0]]
+                    for v in mydict.values():
+                        benchmark_paths.append(v)
+                    pytest_paths = " ".join(tuple(benchmark_paths))
+                    run_benchmarks(pytest_paths, m_tag, compiler)
+                    show_result()
+                    i += 1
+    else:
+        question = input(f"Would you like to run all {len(benchmarks)} available benchmarks (y/n) ")
+        if question.lower() == "y":
+            for benchmark_list in benchmark_category.values():
+                for paths in benchmark_list.values():
+                    benchmark_paths.append(paths)
+            pytest_paths = " ".join(tuple(benchmark_paths))
+            run_benchmarks(pytest_paths, m_tag, compiler)
+            show_result()
+
+
+if __name__ == "__main__":
+    # main()
+    show_result()

requirements.txt

@@ -6,4 +6,5 @@ qiskit-terra
 qiskit-aer
 pytket>1.0,<2.0
 setproctitle
-rich
+rich
+click

setup.py

@@ -59,5 +59,10 @@
         "Documentation": "https://qiskit.org/documentation/",
         "Source Code": "https://github.com/Qiskit/red-queen",
     },
+    entry_points={
+        'console_scripts': [
+            "red-queen = red_queen.cli:main",
+        ],
+    },
     zip_safe=False,
 )