Added parameter optimization and evaluation functionality for iotools benchmarks #11

DanteNiewenhuis · 2023-04-17T08:13:51Z

Added python files that can be used to optimize the parameters of reading and writing and RNTuple file.

The main functionality is evaluating single parameter configurations (see tutorials/evaluate_parameters.py),
and optimizing parameters for a specific benchmark (see tutorials/run_annealer.py).

Three different methods of optimizing parameters are provided in Algorithms.py:

Hill Climber: A simple algorithm that mutates the current parameter configuration, and accepts the new configuration when it results in higher performance.
Simulated Annealer: similar to a Hill Climber, but adds a small chance to accept negative mutations.
Random Walker: Similar to a Hill Climber, but every step is accepted. This is not useful for optimization, but can be used to better understand the solution space

To use the files, first the paths in variables.py have to be updated.

This update also adds two flags to the generator files for defining page and cluster size:

-p define page size in bytes (default 65536)
-x define cluster size in bytes (default 52428800)

jalopezg-git

Many thanks, @DanteNiewenhuis! 👍

I have attached some preliminary comments, but I guess it's up to @jblomer to finally approve and merge.

jalopezg-git · 2023-07-13T11:32:10Z

gen_atlas.cxx

+   size_t pagesize = 65536;
+   size_t clustersize = 52428800;


Suggested change

size_t pagesize = 65536;

size_t clustersize = 52428800;

size_t pageSize = (64 * 1024);

size_t clusterSize = (50 * 1000 * 1000);

Or even better, IMO, set it to 0 or -1U, and only call SetApproxUnzippedPageSize(), etc., if a value was actually set by the user.
If unset, importer should be using the default value from RNTupleWriteOptions.

gen_atlas.cxx

jalopezg-git · 2023-07-13T11:34:37Z

gen_cms.cxx

Same comments in gen_atlas.cxx apply here.

jalopezg-git · 2023-07-13T11:35:05Z

gen_h1.cxx

Same comments in gen_atlas.cxx apply here.

jalopezg-git · 2023-07-13T11:35:42Z

gen_lhcb.cxx

Same comment in gen_atlas.cxx apply here.

jalopezg-git · 2023-07-13T11:41:07Z

parameter_optimization/tutorials/add_path.py

+
+
+import sys
+sys.path.insert(0, "/home/dante-niewenhuis/Documents/CERN-parameters-new")


I think changing this path is not mentioned in the README.md. Also, what is it used for?

Sadly, Python cannot import packages that are in the parent folder of a file.
To fix this, I add the optimization_tools folder to the paths so Python can find it.
I have changed the string so it is clearer that it is something the user should to change.

jalopezg-git · 2023-07-13T11:51:27Z

parameter_optimization/Benchmarking/DataStructures/Configuration.py

+        page_sizes = [(16, "KB"), (32, "KB"), (64, "KB"), (128, "KB"), (256, "KB"), (512, "KB"),
+                      (1, "MB"), (2, "MB"), (4, "MB"), (8, "MB"), (16, "MB")]


Per the convertToByte() function, MB here refers to a power-of-two unit, i.e. a mebibyte.

However, we typically use the MB (1000 * 1000) bytes as the unit for on-disk data (see, e.g. the default value for the fApproxZippedClusterSize member in RNTupleWriteOptions).

jalopezg-git · 2023-07-13T11:53:52Z

parameter_optimization/Benchmarking/benchmark_utils.py

+    return False
+
+
+def convertToByte(value: int, form: str = "b") -> int:


This seems to duplicate this function in Parameters.py. I think we should get rid of one of them.

jalopezg-git · 2023-07-13T11:54:59Z

parameter_optimization/Benchmarking/benchmark_utils.py

+    """
+    for line in benchmark_output.split("\n"):
+        if target in line:
+


Suggested change

jalopezg-git · 2023-07-13T11:58:02Z

parameter_optimization/Benchmarking/benchmark_utils.py

+    for _ in tqdm(range(evaluations)):
+
+        # Reset cache
+        os.system('sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"')


The iotools repository already has clear_page_cache.c. The recipe in the Makefile also sets the suid bit, so that no sudo is required after

$ make clear_page_cache

jblomer

Very nice, thanks a lot!

Some minor comments.

Feel free to add your name and/or contact details to the README.

jblomer · 2023-07-25T14:47:48Z

.gitignore

+gen_h1
+gen_lhcb
+
+*.pyc


Missing newline at the end of the file

jblomer · 2023-07-25T14:51:42Z

gen_atlas.cxx

+   size_t pagesize = (64 * 1024);
+   size_t clustersize = (50 * 1000 * 1000);


Here and in the other generators: use zero to indicate the default page size should be used if the option is not set. I think also the file name modification should only be done if there is a deviation from the default.

jblomer · 2023-07-25T14:53:03Z

gen_cms.cxx

 {
-   std::cout << "Usage: " << progname << " -i <ttjet_13tev_june2019.root> -o <ntuple-path> -c <compression> [-m(t)]"
+   std::cout << "Usage: " << progname << " -o <ntuple-path> -c <compression> -p <page-size> -x <cluster-size> [-m(t)] <H1 root file>"


Suggested change

std::cout << "Usage: " << progname << " -o <ntuple-path> -c <compression> -p <page-size> -x <cluster-size> [-m(t)] <H1 root file>"

std::cout << "Usage: " << progname << " -o <ntuple-path> -c <compression> -p <page-size> -x <cluster-size> [-m(t)] <CMS root file>"

jblomer · 2023-07-25T14:54:00Z

optimization_tools/README.md

@@ -0,0 +1,12 @@
+This folder contains python files used to benchmark and optimize parameters based on the iotools benchmarks.
+
+To run the files, first the Paths in Benchmarking/variables.py need to be updated. 


Suggested change

To run the files, first the Paths in Benchmarking/variables.py need to be updated.

To run the files, first the paths in Benchmarking/variables.py need to be updated.

jblomer · 2023-07-25T14:56:27Z

optimization_tools/tutorials/add_path.py

+
+
+import sys
+sys.path.insert(0, "/home/dante-niewenhuis/Documents/CERN-parameters-new")


Suggested change

sys.path.insert(0, "/home/dante-niewenhuis/Documents/CERN-parameters-new")

sys.path.insert(0, "CERN-parameters-new")

This will not work in this case because "CERN-parameters-new" is a parent of the current working directory

jblomer · 2023-07-25T15:00:12Z

optimization_tools/Benchmarking/Algorithms.py

+# root [12] auto ntuple = ROOT::Experimental::RNTupleReader::Open("DecayTree" , "generated/B2HHH~none.ntuple")
+# root [13] ntuple->PrintInfo(ROOT::Experimental::ENTupleInfo::kStorageDetails)


Add to the comment what it means / why it is there

jblomer · 2023-07-25T15:06:59Z

optimization_tools/Benchmarking/benchmark_utils.py

+
+    for _ in tqdm(range(evaluations)):
+        # Reset cache
+        os.system('sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"')


For this we have the clean_page_cache utility

DanteNiewenhuis added 4 commits April 17, 2023 10:08

test push

8e7918b

Added flags for page and cluster size to gen files

8a2bdee

Added all files used for parameter optimization

e4d3cb2

changed default clustersize for generators

88653f5

DanteNiewenhuis changed the title ~~Flags for page and cluster size~~ Added parameter optimization and evaluation functionality for iotools benchmarks Jun 19, 2023

jalopezg-git reviewed Jul 13, 2023

View reviewed changes

DanteNiewenhuis added 6 commits July 25, 2023 10:36

updates for the PR

b3ff00e

Merge remote-tracking branch 'upstream/master'

537c9f4

Added flags for page and cluster size to gen files

0dfc990

changed default clustersize for generators

9af1614

removing merge conflicts

3a700b8

updated the convertToByte function

d4fc1a5

jblomer requested changes Jul 25, 2023

View reviewed changes

Updates based on the comments in the PR

8deaaa3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added parameter optimization and evaluation functionality for iotools benchmarks #11

Added parameter optimization and evaluation functionality for iotools benchmarks #11

DanteNiewenhuis commented Apr 17, 2023 •

edited

Loading

jalopezg-git left a comment

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

DanteNiewenhuis Jul 28, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jalopezg-git Jul 13, 2023

jblomer left a comment

jblomer Jul 25, 2023

jblomer Jul 25, 2023

jblomer Jul 25, 2023

jblomer Jul 25, 2023

jblomer Jul 25, 2023

DanteNiewenhuis Jul 28, 2023

jblomer Jul 25, 2023

jblomer Jul 25, 2023



		import sys
		sys.path.insert(0, "/home/dante-niewenhuis/Documents/CERN-parameters-new")

		page_sizes = [(16, "KB"), (32, "KB"), (64, "KB"), (128, "KB"), (256, "KB"), (512, "KB"),
		(1, "MB"), (2, "MB"), (4, "MB"), (8, "MB"), (16, "MB")]

		return False


		def convertToByte(value: int, form: str = "b") -> int:

		size_t pagesize = (64 * 1024);
		size_t clustersize = (50 * 1000 * 1000);

	std::cout << "Usage: " << progname << " -o <ntuple-path> -c <compression> -p <page-size> -x <cluster-size> [-m(t)] <H1 root file>"
	std::cout << "Usage: " << progname << " -o <ntuple-path> -c <compression> -p <page-size> -x <cluster-size> [-m(t)] <CMS root file>"

		@@ -0,0 +1,12 @@
		This folder contains python files used to benchmark and optimize parameters based on the iotools benchmarks.

		To run the files, first the Paths in Benchmarking/variables.py need to be updated.

	sys.path.insert(0, "/home/dante-niewenhuis/Documents/CERN-parameters-new")
	sys.path.insert(0, "CERN-parameters-new")

		# root [12] auto ntuple = ROOT::Experimental::RNTupleReader::Open("DecayTree" , "generated/B2HHH~none.ntuple")
		# root [13] ntuple->PrintInfo(ROOT::Experimental::ENTupleInfo::kStorageDetails)

Added parameter optimization and evaluation functionality for iotools benchmarks #11

Are you sure you want to change the base?

Added parameter optimization and evaluation functionality for iotools benchmarks #11

Conversation

DanteNiewenhuis commented Apr 17, 2023 • edited Loading

jalopezg-git left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jblomer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanteNiewenhuis commented Apr 17, 2023 •

edited

Loading