Releases: miranov25/RootInteractive
v0-01-11
Last Release with the Bokeh 2.x.y
- Last Release Before Transitioning to Bokeh 3.x.y
- Please note that the upcoming release will not be backward compatible.
What's Changed
- Development version development for the Domain-Specific Language with a fraction of the desired functionality (details below)
- Next Release Goals:
- M:N joins
- Lossy compression/decompression
- Enhanced user-friendliness through integration with the ROOT team
Details:
- Various client side lazy evaluation optimizations and bugfix by @pl0xz0rz in #336
- RDataFrame interface binary search, fix for BoolOp by @pl0xz0rz in #337
- Error handling for custom variables by @pl0xz0rz in #339
- Rolling window statistics - so far only sum, mean and std by @pl0xz0rz in #340
- Rolling median builtin for domain specific language with RDataFrame by @pl0xz0rz in #342
- RDataFrame rolling quantile by @pl0xz0rz in #343
- Optimization for simple joins on client by @pl0xz0rz in #344
- Removed uproot dependency by @pl0xz0rz in #347
Full Changelog: v1-01-10...v0-01-11
v0-01-10
List of the most important changes
-
For detailed description of the recent changes , see the description in the CHEP conference:
-
Templated parameterized functions (see details in proceedings above)
- The generation of configuration arrays, in particular figureArray, histogramArray, aliasArray, layout, widgetLay- out and parameterArray, together contribute to the construction and creation of interactive visualisations and provide a streamlined approach to the complex display and analysis of
data.
- The generation of configuration arrays, in particular figureArray, histogramArray, aliasArray, layout, widgetLay- out and parameterArray, together contribute to the construction and creation of interactive visualisations and provide a streamlined approach to the complex display and analysis of
-
mutliselect for the histograms ( array of histograms could be defined by user on the client)
-
Experimental Domain-specific language for RDataFrame code generation (Numpy,Python like syntax & joins)
-
Adding automatic description tables and documentation /tab (for variable description)
-
Better error handling
-
Optimization of compression
-
Optimization of machine learning RandoForest - fully parallelizing PredictStat
What's Changed
- Optimized compression by @pl0xz0rz in #292
- Compression bugfix by @pl0xz0rz in #294
- Rdataframe makeDefine by @pl0xz0rz in #298
- makeDefine fixes by @pl0xz0rz in #299
- Changed aliasDict in makeJSCallback hacky workaround to the intended way by @pl0xz0rz in #300
- makeDefine function signature and array size detection by @pl0xz0rz in #301
- makeDefine support for detecting output type -> #293 by @pl0xz0rz in #304
- Optimization for custom filter using text input by @pl0xz0rz in #305
- makeJSCallback optimization and bugfixes by @pl0xz0rz in #306
- makeJScallback removal by @pl0xz0rz in #307
- Div widget and table to describe data (parsing user input metadata) - for #309 by @pl0xz0rz in #311
- Fix for #313 by @pl0xz0rz in #314
- Replacing deprecated sklearn with scikit-learn by @ehellbar in #308
- Summary table for selection by @pl0xz0rz in #315
- descriptionTable change to bokeh DataTable by @pl0xz0rz in #316
- Added variable name to description table by @pl0xz0rz in #317
- Added parameter 'meta' to bokehDrawArray by @pl0xz0rz in #318
- Predict rf stat - parallelization (#320) by @pl0xz0rz in #321
- Predict rf stat by @pl0xz0rz in #322
- Linear fitter on the client by @pl0xz0rz in #325
- Cdsstack and multiselect variables for histograms by @pl0xz0rz in #326
- Multiselect varY / varYNorm template by @pl0xz0rz in #327
- Optimized multiselect weights by @pl0xz0rz in #328
- Changing diff function in template by @pl0xz0rz in #330
- Template bugfixes and support for function in namespace by @pl0xz0rz in #331
- Template switchable by @pl0xz0rz in #333
- #332 added better error handling for invalid widgets by @pl0xz0rz in #334
Full Changelog: v0-01-09...v0-01-10
v0-01-09 - preparation for new tutorial
v0-01-08 - 2D user defined parameterized axis transforms (e.g for rotation, function normalization)
What's Changed
- Color axis added to plot title by @pl0xz0rz in #273
- Parametrized transforms by @pl0xz0rz in #274
- Using parameters in expressions by @pl0xz0rz in #275
- 2D user defined axis transforms (e.g for rotation, function normalization) by @pl0xz0rz in #276
- Added support for the '1' alias by @pl0xz0rz in #277
- Change interface for palettes to be more consistent by @pl0xz0rz in #279
Full Changelog: v0-01-07...v0-01-08
v0-01-07 - release
What's Changed
- Parameterized (Anonymous) custom JS expressions/function on client (binary and unary operation, numpy functions, comparators) by @pl0xz0rz in #249
- Unbinned mean, standard deviation and quantile for histogram projections by @pl0xz0rz in #250
- Heatmap tuple interface by @pl0xz0rz in #251
- Asymmetric error bars by @pl0xz0rz in #252
- Spinner range fix (for #250) by @pl0xz0rz in #253
- Less verbose aliasArray interface by @pl0xz0rz in #255
- Cumulative histogram by @pl0xz0rz in #258
- Y-Axis transforms (#254, #247) by @pl0xz0rz in #256
- X axis transform by @pl0xz0rz in #260
- Spinner ranges formatting and step size --> #259 by @pl0xz0rz in #264
- Custom user defined function freezing bugfix in function dependencies (for #265) by @pl0xz0rz in #266
- Technical fix enabling using constants as an alias + optimization for aliases by @pl0xz0rz in #267
- Optimized bokehDrawSA by removing dead code (factor ~ 2) by @pl0xz0rz in #268
- Minor histogram tutorial update and enabling spinnerRange for controlling aliases by @pl0xz0rz in #270
Full Changelog: v0-01-06...v0-01-07
Release v0-01-06
What's Changed
- Lazy evaluation of histograms and partial implementation of lazy population of tabs by @pl0xz0rz in #222
- MultiSelect optimization/bugfix by @pl0xz0rz in #226
- Custom JS function text input by @pl0xz0rz in #228
- Heatmap with source by @bulukutlu in #230
- Histogram edge case with auto ranges fix by @pl0xz0rz in #234
- Histogram ranges / number of bins widgets by @pl0xz0rz in #237
- Toggleable widgets by @pl0xz0rz in #239
- Spinner range widget by @pl0xz0rz in #240
- Spinner range bugfix by @pl0xz0rz in #242
- Migration to bokeh 2.3.3 by @pl0xz0rz in #150
- Int64 compression bugfix by @pl0xz0rz in #245
New Contributors
- @bulukutlu made their first contribution in #230
Full Changelog: v-01-05...v0-01-06
Description
In the last months after the tutorial, Marian & Marian have implemented many improvements in RootInteractive:
- extended functionality
- parameterization of the spinner RANGE
- used for multiscale selection using text or exponential spinner
- histogram parametrization (nBins, ranges)
- user defined function
- used e.g to define selection for the data and reference data as custom javascript
return tpcNCl>120 & ITSrefit
- used e.g to define selection for the data and reference data as custom javascript
- Bitmasks (And, OR) logic
- Templates/dictionaries for graphics parameterisation
- Acceleration
- Documentation
- parameterization of the spinner RANGE
Many of these new features have already been used in the recent TPC data volume studies and TrackCombinator studies:
- tracks/cluster MC/data comparison IR scan
- track efficiency
- electron bremstrahlung correction in tracking:
Furthermore, we have opted for better documentation and not only used Python's online documentation, but also provided documents for each parameter as README files:
- https://github.com/miranov25/RootInteractive/blob/master/README.md
- https://github.com/miranov25/RootInteractive/blob/master/RootInteractive/InteractiveDrawing/bokeh/doc/
-
https://github.com/miranov25/RootInteractive/blob/master/RootInteractive/InteractiveDrawing/bokeh/doc/READMEwidgets.md#real-data-use-case---tpc-clusters-mcdata-comparison ...
In the readme file we refer to the test_py files as an example and also to current realistic use cases.
We plan to make a new release today. If you have time, give us your feedback. We plan to release it before the OFFLINE meeting.
The documentation still needs many improvements, which will be included in the next release.
In the near future, next Release, we would like to focus on the new RDataFrame <-> awkward interface recently released in the scikit-hep, to further speed up the preparation of input data.
The ONNX interface for machine learning functions on the client (for parameter variation) is on the wish list, but I hope we will also converge on this for next Release
v-01-05 Release
What's Changed
- Merge master to ErrPDF by @miranov25 in #165
- Added regular expressions to select subset of variables to histoTable by @pl0xz0rz in #195
- Err pdf by @miranov25 in #197
- Adding more documentation to bokehDraw standalone tutorials by @ehellbar in #198
- TextAreaInput widget fix by @pl0xz0rz in #202
- Adding parameters (widgets) for the legend visibility and number of points for renderer by @pl0xz0rz in #203
- Fixed error handling of nonexisting columns by @pl0xz0rz in #208
- Optimized the MultiSelect widget selection caching results by @pl0xz0rz in #212
- Multiselect bitmask by @pl0xz0rz in #214
- Widgets in layoutDesc can be specified also by names (before only index could be used) by @pl0xz0rz in #215
- MakeJSCallback index slider, performance optimizations by @pl0xz0rz in #220
This is prerelease
Significant speed-up of the histogram was achieved by bug fixes and finally by the option to select only a subset of the data. Further improvement of CPU consumption under preparation #221 for the next release
Full Changelog: v0-01-04...v-01-05
v0-01-04 - release - adding dependency trees
Pull request and issues solved:
#169 - bug fixes needed by previous pull request
#172 - Reduced widget description verbosity for select widgets
#177, #178 - enabling parallel pytest, regression test, export results to json output for later analysis using jq
#175 - client-side joins
#186 - Dependency graphs on server, refactoring, parameters can change X and Y axes, some bugfixes
#186 Dependency graphs on server, refactoring, parameters can change X and Y axes
This PR refactors a lot of the code, and fixes some bugs. The PR is not complete, as joins on the client are missing, as is putting tooltips on the dependency tree.
Relates to #180 (dependency trees are created as Python dictionaries, and checked for cyclic dependencies), #156 (parameters can now change X and Y axes, legend field and error bars)
New features:
The "source" option in figureArray in bokehDrawArray - sets the source to use for drawing, so that using a different data source from the main one is less verbose - also frees up using the dot for joins
Parameters can now swap the variable used for X and Y axes
Custom error bars for histograms
Bugfixes:
Fixed error bars being broken for sources other than the main one
...
Changes:
String parsing now uses Python parsing, should be smarter e.g. when identifying whether an expression is a column or a floating point number
#175 - client-side joins
New parameter to bokehDrawArray - sourceArray
Outside of specifying histograms as in histoArray, it also allows specifying secondary data sources and joins on the client
Use case:
df2 = pd.DataFrame(...)
sourceArray = [
{"name": "df2", "data": df2},
{"name": "join_df2", "left": None, "right":"df2", "left_on": ["bin_index"], "right_on": ["bin_index"], "how": "left"}
]
#177, #178 - enabling parallel pytest, regression test, export results to json output for later analysis using jq
- standard pytest too slow, parallel test reduce test time from O(15 min) to O(1 min)
- Example usage:
- source macor
Singularity> source $RootInteractive/tests/pytest.sh
init
pathJQ -> return expanded list of the keys
makePyTest -> make pytest store results and metadata in json files
uploadPyTest -> upload test on the server
- run make PyTest ad upload results on server
Singularity> makePyTest
makePyTest
Parameters:
1.) param - testWorkers - number of workers to test
2.) param - testName
3.) param - testPrefixServer
4.) param - testServer
Example usage:
makePyTest 6 test6 /lustre/alice/users/miranov/tests/RootInteractive alice
What's Changed
- PWGPP-581 performance fixes by @pl0xz0rz in #62
- Compatibility bokeh by @pl0xz0rz in #74
- PWGPP-618 initial commit by @pl0xz0rz in #75
- Initial integration of visJS graph3d within bokehDrawArray by @pl0xz0rz in #77
- Added downsampling to JS callback by @pl0xz0rz in #78
- Fixed downsample indexing being inconsistent by @pl0xz0rz in #79
- Refactored makeJSCallback and optimized it, added color mapper rescaling by @pl0xz0rz in #81
- Optimized bokehDrawArray by @pl0xz0rz in #83
- Fixed multiselect to work with floats, no longer works with strings by @pl0xz0rz in #85
- Fixed JS callback to work with booleans in select/MultiSelect by @pl0xz0rz in #87
- Added client side histogramming using JavaScript by @pl0xz0rz in #92
- Client-side histogram - improved the performance slightly by @pl0xz0rz in #93
- Changed indices to nullable array instead of bokeh property in histogram by @pl0xz0rz in #94
- bokehDrawArray no longer outputs cdsSel if nothing draws from it by @pl0xz0rz in #95
- histograms can be used with varX specified as a function by @pl0xz0rz in #97
- Added client-side 2D histogram by @pl0xz0rz in #98
- Changing client-side histogramming to new interface by @pl0xz0rz in #99
- #100 - make cleanup by @miranov25 in #101
- #100 - cleanup part 2 by @miranov25 in #102
- More cleanup by @pl0xz0rz in #105
- Added colZ visualization for clientside histograms by @pl0xz0rz in #106
- Added axis orientation and error bars to histograms by @pl0xz0rz in #107
- Added error bars to 1D histograms by @pl0xz0rz in #108
- Array compression as a pipeline of actions by @miranov25 in #109
- #96 - adding compression CDS -not WORKING yet by @miranov25 in #110
- Compression array - optional compression of CDS before sending to client (or saving) #96 by @miranov25 in #111
- Remove unused columns by @pl0xz0rz in #113
- Fixed a bug causing compression to not work with histograms present by @pl0xz0rz in #114
- Fixed a downsampling bug by @pl0xz0rz in #116
- Histogram auto ranges by @pl0xz0rz in #118
- Added histogram summary table, so far only uses histograms that are i… by @pl0xz0rz in #119
- Added tooltips to histograms by @pl0xz0rz in #120
- Histogram stats table formatting by @pl0xz0rz in #122
- Added quantiles to histoStatsCDS by @pl0xz0rz in #124
- #123 added sum in user defined ranges by @pl0xz0rz in #125
- Fixed bug with colorZvar not being sent to client by @pl0xz0rz in #132
- Fixed default markers by @pl0xz0rz in #134
- Histond clientside by @pl0xz0rz in #127
- Fixed a bug with auto axis titles causing crashes for some 1D histograms by @pl0xz0rz in #136
- Histond tutorial - fix for broken pull request by @pl0xz0rz in #140
- Fixes regression caused by #138 by @pl0xz0rz in #144
- Auto-rescaling color mappers ignore elements where isOK is false by @pl0xz0rz in #148
- Fixed sphinx documentation generation (duct tape fix) by @pl0xz0rz in #149
- Compression dtype check by @pl0xz0rz in #154
- Range sums and efficiencies by @pl0xz0rz in #153
- Color Z widget by @pl0xz0rz in #156
- Fix for figure tabs by @pl0xz0rz in #158
- MakeJSCallback cleanup - extracting downsampling to custom bokeh model by @pl0xz0rz in #159
- Added legend options to parameter array by @pl0xz0rz in #160
- Compression bugfix and footgun removal by @pl0xz0rz in #161
- Auto tooltips for ND histogram projections by @pl0xz0rz in #163
- Client side functions and efficiencies by @pl0xz0rz in #166
- Tutorials for bokehDrawSA and client side functions by @pl0xz0rz in #167
- Minor documentation fixes by @pl0xz0rz in #168
- Removed error bar spaghetti and fixed some warnings by @pl0xz0rz in #169
- Reduced widget description verbosity by @pl0xz0rz in #172
- Clientside joins by @pl0xz0rz in #175
- Pull request - PyTest - Regression test and json/xml for later processing #177 (part 0) by @miranov25 in #178
- Pytest adding pytest server to store performance data as json reports by @miranov25 in #181
- Dependency graphs on server, refactoring, parameters can change X and Y axes, some bugfixes by @pl0xz0rz in #186
- Tooltip bugfix by @pl0xz0rz in #191
Full Changelog: v0.00.21...v0-01-04
v0-01-03-rev1 - Revision fixing warnings
v0-01-03 - functions on client and derived aggregated function
Updated tutorials:
graphics, histogramming and function on client:
Function on client
- enabling derived varables e.g for cut selection
- derived aggrgated variable .e.g for the effieciency /ratio, resplution ratio ...
- #166
Bug fixes
- Compression to proper data types - types not stored -> the same type in client/server