diff --git a/.gitignore b/.gitignore index 1dc0672..4f4619f 100644 --- a/.gitignore +++ b/.gitignore @@ -112,6 +112,7 @@ celerybeat.pid # Environments .env .venv +.virtualenv env/ venv/ ENV/ diff --git a/README.md b/README.md index 4a6b8db..28ca719 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,143 @@ -> "Well, in our country," said Alice, still panting a little, "you'd generally -> get to somewhere else—if you run very fast for a long time, as we've been +> "Well, in our country," said Alice, still panting a little, "you'd generally +> get to somewhere else—if you run very fast for a long time, as we've been > doing." > -> "A slow sort of country!" said the Queen. "Now, here, you see, it takes all -> the running you can do, to keep in the same place. If you want to get +> "A slow sort of country!" said the Queen. "Now, here, you see, it takes all +> the running you can do, to keep in the same place. If you want to get > somewhere else, you must run at least twice as fast as that!" > > [Carroll, Lewis: Through the Looking-Glass, Chapter 2]( https://www.gutenberg.org/files/12/12-h/12-h.htm) -## About +# About + The Red Queen benchmark framework was created to facilitate the benchmarking algorithms used in quantum compilation. -The framework is tightly integrated into `pytest`. Therefore, to use it -effectively, you should know the basics of `pytest` first. Take a look at the +The framework is tightly integrated into `pytest`. Therefore, to use it +effectively, you should know the basics of `pytest` first. Take a look at the [introductory material](https://docs.pytest.org/en/latest/getting-started.html). -## Usage +# Usage + Red Queen is a framework for benchmarking quantum compilation algorithms. Since -it was not designed as a package, there is no notion of installation. Hence, you -must clone this repository to use it: +is is still in early development, you must clone this repository to use it: + ```bash git clone git@github.com:Qiskit/red-queen.git ``` -To run benchmarks, you must first go to the `red-queen` directory and install -the required packages: +Since Red Queen is a Python package is version control is very important, therefore using a virtual enviornment to run benchmarks is highly recomended. + +To get into Red Queen repository + ```bash cd red-queen -pip install -r requirements.txt ``` +To create a virtual environment + +```bash +python -m venv .virtualenv +``` + +After you create the virtual enviornment you need to activate it + +To activate the virtual enviornment + +[For MacOs and Linux] + +```bash +source .virtualenv/bin/activate +``` + +[For Windows] + +```bash +.virtualenv\Scripts\activate.bat +``` + +You need to install all of the neccessary dependencies for Red Queen + +```bash +pip install -e . +``` + +Lastly `cd` into `red_queen` so that you'll be able to use the `cli` (command line interface) + +```bash +cd red_queen +``` + +With all that you are ready to start using Red Queen CLI. + +Red Queen's `CLI` simplifies the execution of benchmarks by constructing the pytest scripts for you all you gotta do is specify the benchmarks that you'd like to run. + +You can find all available benchmark just by running this command + +```bash +red-queen --help +``` + +The general templete for the `CLI` is as follows: + +```bash +red-queen -c -b +``` + +> Side bar: The complier flag is optional. If you don't specify a complier the `cli` will just run `qiskit` + Now, suppose you want to run the mapping benchmarks using only `tweedledum`. -You must do it using `pytest` +You can do this via the `CLI` or with `pytest` + +[For MacOs and Linux] + +With `CLI` + ```bash -pytest red_queen/games/mapping/map_queko.py -m tweedledum --store +red-queen -c tweedledum -b map_queko.py ``` -To run pytest on Windows, you will have to use `python -m` in order to run the -`pytest` command. You will also need to add `-s` to your pytest call to disable -stdin handling. +With `pytest` + ```bash -python -m pytest -s red_queen/games/mapping/map_queko.py -m tweedledum --store +python -m pytest games/mapping/map_queko.py -m tweedledum --store ``` -The benchmark suite will consider all functions named `bench_*` in -`red_queen/games/mapping/map_queko.py`. Because we set the `-m` option, only the the ones -marked with `tweedledum` will be run. (We could easy do the same for `qiskit`). -If you don't define a `-m` option, all `bench_*` functions will be run. +[For Windows] + +With `CLI` +```bash +red-queen -c tweedledum -b map_queko.py +``` + +With `pytest` + +```bash +python -m pytest -s games/mapping/map_queko.py -m tweedledum --store +``` + +To run pytest or any python script on Windows, you will have to use `python -m` in order to run the +`pytest` command. You will also need to add `-s` to your pytest call to disable +stdin handling. + +```bash +python -m pytest -s games/mapping/map_queko.py -m tweedledum --store +``` + +The benchmark suite will consider all functions named `bench_*` in +`games/mapping/map_queko.py`. Because The `--store` option tells the framework to store the results in json file in the `results` directory. To see the results as a table, you can use the you can use: + ```bash python -m report.console_tables --storage results/0001_bench.json ``` ## Warning + This code is still under development. There are many razer sharp edges. For information of how execution works and other details about the framwork @@ -73,7 +153,7 @@ on the knowledge of the internals of the following established `pytest` plugins: ## License -This software is licensed under the Apache 2.0 licence (see +This software is licensed under the Apache 2.0 licence (see [LICENSE](https://github.com/Qiskit/red-queen/blob/main/LICENSE)) ## Contributing diff --git a/red_queen/cli.py b/red_queen/cli.py new file mode 100644 index 0000000..f23295d --- /dev/null +++ b/red_queen/cli.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +# ------------------------------------------------------------------------------ +# Part of Red Queen Project. This file is distributed under the Apache 2.0. +# See accompanying file /LICENSE for details. +# ------------------------------------------------------------------------------ + +"""The purpose of this code below is to make the user experience of +the Red Queen benchmark suite more steamlined + +The code achieves that by collecting all avaiable benchmark alongside their paths, +and uses this information to create exacutable pytest code that +will exacute the benchmarks for users with worrying about the +nuisances ofthe pytest framework. The scope of this cli will +grow with time. +""" +import os +import platform +import subprocess +import configparser +import click + +# This function retrives the paths for each benchmark and returns +# a hashmap with the benchmark's name and it's path +# It also retrives all available compliers for Red Queen +def benchmark_complier_retrieval(): + benchmark_dict = {} + list_of_benchmarks = [] + list_of_compliers = set() + benchmark_category_set = set() + benchmark_path = "games/" + for benchmark_category in os.scandir(benchmark_path): + if benchmark_category.is_dir(): + benchmark_category_set.add(benchmark_category.name) + name_path_pair = {} + for benchmark in os.scandir(f"{benchmark_path}{benchmark_category.name}"): + if ( + not benchmark.name.startswith("_") + and benchmark.name.endswith(".py") + and benchmark.is_file() + ): + name_path_pair[benchmark.name] = benchmark.path + benchmark_dict.update(name_path_pair) + + for benchmark_name, _ in benchmark_dict.items(): + list_of_benchmarks.append(benchmark_name) + + # Config helps us process the .ini file + config = configparser.ConfigParser() + config.read("../pytest.ini") + for complier in config["pytest"]["markers"].split("\n"): + if complier != "": + list_of_compliers.add(complier) + return benchmark_dict, list_of_benchmarks, list_of_compliers + + +# The function retrives the latest benchmark ran's results +def result_retrieval(): + results = {} + dir_path = "results" + for result in os.scandir(dir_path): + if result.is_file(): + filename = result.name + result_count = int(filename.split("_")[0]) + results[result_count] = {result.name: result.path} + return results + + +# This functions creates the pytest command and runs it +def run_benchmarks(pytest_paths: str, m_tag: str, compiler: str): + command_list = ["pytest"] + compiler_command = [m_tag, compiler, "--store"] + # Are you using a Windows Machine? + if platform.system() == "Windows": + command_list.insert(0, "-m") + command_list.insert(0, "python") + command_list.append("-s") + for _, string in enumerate(pytest_paths): + command_list.append(string) + for _, string in enumerate(compiler_command): + command_list.append(string) + subprocess.run( + command_list, + check=True, + ) + else: + command_list.insert(0, "-m") + command_list.insert(0, "python") + for _, string in enumerate(pytest_paths): + command_list.append(string) + for _, string in enumerate(compiler_command): + command_list.append(string) + subprocess.run( + command_list, + check=True, + ) + + +# This function displays the results +def show_result(): + results_dict = result_retrieval() + result_num = max(results_dict.keys()) + result_path = tuple(results_dict[result_num].values())[0] + command_list = ["python", "-m", "report.console_tables", "--storage"] + command_list.append(str(result_path)) + subprocess.run( + command_list, + check=True, + ) + click.echo("To view the table again:") + click.echo(" ".join(command_list) + "\n") + + +benchmark_hash, benchmarks_list, complier_list = benchmark_complier_retrieval() + + +# The code below deals with each flag of the cli +@click.command() +@click.option( + "-c", + "--compiler", + "compiler", + multiple=True, + type=click.Choice(complier_list), + help="enter a compliler here", +) +@click.option( + "-b", + "--benchmark", + "benchmark", + multiple=True, + type=click.Choice(benchmarks_list), + help="enter the specfic benchmark(s) here", +) +# The main function calls on the above helper functions to run the desired benchmarks +def main(compiler=None, benchmark=None): + benchmark_paths = [] + pytest_paths = "" + m_tag = "" + # Is there a specfic compiler listed? + if len(compiler) > 0: + m_tag = "-m" + compiler = compiler[0] + else: + compiler = "" + j = 0 + i = 0 + ### Has the user input benchmarks? + if len(benchmark) > 0: + ### Are the inputted benchmark(s) valid? + if set(benchmark).issubset(benchmarks_list): + for j, _ in enumerate(benchmark): + benchmark_paths.append(benchmark_hash[benchmark[j]]) + pytest_paths = benchmark_paths + run_benchmarks(pytest_paths, m_tag, compiler) + show_result() + i += 1 + else: + print("Please input benchmark") + print("Example:\nred-queen -b grovers.py") + + +if __name__ == "__main__": + main() diff --git a/requirements.txt b/requirements.txt index 58ace3f..4667510 100644 --- a/requirements.txt +++ b/requirements.txt @@ -6,4 +6,5 @@ qiskit-terra qiskit-aer pytket>1.0,<2.0 setproctitle -rich \ No newline at end of file +rich +click \ No newline at end of file diff --git a/setup.py b/setup.py index 8532d8e..f4f045c 100644 --- a/setup.py +++ b/setup.py @@ -59,5 +59,10 @@ "Documentation": "https://qiskit.org/documentation/", "Source Code": "https://github.com/Qiskit/red-queen", }, + entry_points={ + 'console_scripts': [ + "red-queen = red_queen.cli:main", + ], + }, zip_safe=False, ) \ No newline at end of file