LOG4MEX Use Cases
LOG4MEX provides rich metadata about machine learning experiments, both machine-readable and human-readable1 format at same time! Experiments now can be easily compared, regardless ML strategy. No need to use frameworks or workflow systems! Check it out!
An
Experimentis divided intonExperiment Configurations, each of that consisting into different dimensions of analysis (e.g.:Datasets,Hardware Environments,Features, etc.)
Each
Experiment Configurationcontains one or moreExecutions.
Following, we introduce some use cases which have been benefiting from its usage.
FOX is a highly accurate open-source framework for named entity recognition. We used LOG4MEX to export the ML outputs of its experiment, minimising the drawbacks of not having a more refined schemata. Instead of trawling through the directory structure (more than 20.000 files in about 5GB of uncompressed data) and make suppositions, SPARQL queries comes into play as a most elegant and fast solution to extract information.
Figure above depicts an excerpt of code for the generated metadata file. Researchers are able to intuitively understand and absorb the relation of the variables existing along experiment configurations.
Give me all algorithms which were measured by F-measure.
PREFIX mexcore: <http://mex.aksw.org/mex-core#>
PREFIX mexperf: <http://mex.aksw.org/mex-perf#>
PREFIX mexalgo: <http://mex.aksw.org/mex-algo#>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT DISTINCT ?ExecutionID ?Algorithm ?Performance ?fMeasure WHERE {
?execution prov:used ?alg;
prov:id ?ExecutionID.
?Performance prov:wasGeneratedBy ?execution.
?Performance mexperf:f1Measure ?fMeasure.
?alg a mexalgo:Algorithm.
?alg rdfs:label ?Algorithm.
}
ORDER BY DESC (?fMeasure)
LIMIT 30
Result:
| ExecutionID | Algorithm | Performance | fMeasure |
|---|---|---|---|
| "C0_MEX_EXEC_D44" | "BaggingJ48" | mea_clas_C0_MEX_EXEC_D44_cf_1_-568657719 | 0.9968 |
| "C0_MEX_EXEC_D24" | "Logistic Model Trees" | mea_clas_C0_MEX_EXEC_D24_cf_1_-568657719 | 0.9968 |
| "C0_MEX_EXEC_D16" | "Random Forest" | mea_clas_C0_MEX_EXEC_D16_cf_1_-568657719 | 0.9968 |
| "C0_MEX_EXEC_D64" | "Multilayer Perceptron" | mea_clas_C0_MEX_EXEC_D64_cf_1_-568657719 | 0.9967 |
| "C0_MEX_EXEC_D68" | "Decision Table" | mea_clas_C0_MEX_EXEC_D68_cf_1_-568657719 | 0.9966 |
| "C0_MEX_EXEC_D60" | "Functional Trees" | mea_clas_C0_MEX_EXEC_D60_cf_1_-568657719 | 0.9966 |
| "C0_MEX_EXEC_D40" | "J48" | mea_clas_C0_MEX_EXEC_D40_cf_1_-568657719 | 0.9965 |
| "C0_MEX_EXEC_D28" | "Ada BoostM1" | mea_clas_C0_MEX_EXEC_D28_cf_1_-568657719 | 0.9965 |
| "C0_MEX_EXEC_D56" | "Support Vector Machines" | mea_clas_C0_MEX_EXEC_D56_cf_1_-568657719 | 0.9964 |
| "C0_MEX_EXEC_D76" | "Logistic Regression" | mea_clas_C0_MEX_EXEC_D76_cf_1_-568657719 | 0.9962 |
| "C0_MEX_EXEC_D52" | "Additive Logistic Regression" | mea_clas_C0_MEX_EXEC_D52_cf_1_-568657719 | 0.996 |
| "C0_MEX_EXEC_D4" | "NER STANFORD" | mea_clas_C0_MEX_EXEC_D4_cf_1_-568657719 | 0.9959 |
| "C0_MEX_EXEC_D36" | "Class Level Voting" | mea_clas_C0_MEX_EXEC_D36_cf_1_-568657719 | 0.9957 |
| "C0_MEX_EXEC_D8" | "Sequential Minimal Optimization" | mea_clas_C0_MEX_EXEC_D8_cf_1_-568657719 | 0.9957 |
| "C0_MEX_EXEC_D12" | "Naive Bayes" | mea_clas_C0_MEX_EXEC_D12_cf_1_-568657719 | 0.9954 |
| "C0_MEX_EXEC_D60" | "Functional Trees" | mea_clas_C0_MEX_EXEC_D60_cf_1_-567674523 | 0.9945 |
| "C0_MEX_EXEC_D24" | "Logistic Model Trees" | mea_clas_C0_MEX_EXEC_D24_cf_1_-567674523 | 0.9945 |
| "C0_MEX_EXEC_D44" | "BaggingJ48" | mea_clas_C0_MEX_EXEC_D44_cf_1_-567674523 | 0.9945 |
| "C0_MEX_EXEC_D28" | "Ada BoostM1" | mea_clas_C0_MEX_EXEC_D28_cf_1_-567674523 | 0.9945 |
| "C0_MEX_EXEC_D40" | "J48" | mea_clas_C0_MEX_EXEC_D40_cf_1_-567674523 | 0.9945 |
| "C0_MEX_EXEC_D16" | "Random Forest" | mea_clas_C0_MEX_EXEC_D16_cf_1_-567674523 | 0.9944 |
| "C0_MEX_EXEC_D64" | "Multilayer Perceptron" | mea_clas_C0_MEX_EXEC_D64_cf_1_-567674523 | 0.9944 |
| "C0_MEX_EXEC_D68" | "Decision Table" | mea_clas_C0_MEX_EXEC_D68_cf_1_-567674523 | 0.9944 |
| "C0_MEX_EXEC_D60" | "Functional Trees" | mea_clas_C0_MEX_EXEC_D60_cf_1_-567554490 | 0.9943 |
| "C0_MEX_EXEC_D24" | "Logistic Model Trees" | mea_clas_C0_MEX_EXEC_D24_cf_1_-567554490 | 0.9943 |
| "C0_MEX_EXEC_D28" | "Ada BoostM1" | mea_clas_C0_MEX_EXEC_D28_cf_1_-567554490 | 0.9943 |
| "C0_MEX_EXEC_D40" | "J48" | mea_clas_C0_MEX_EXEC_D40_cf_1_-567554490 | 0.9943 |
| "C0_MEX_EXEC_D56" | "Support Vector Machines" | mea_clas_C0_MEX_EXEC_D56_cf_1_-567674523 | 0.9943 |
| "C0_MEX_EXEC_D44" | "BaggingJ48" | mea_clas_C0_MEX_EXEC_D44_cf_1_-567554490 | 0.9942 |
| "C0_MEX_EXEC_D64" | "Multilayer Perceptron" | mea_clas_C0_MEX_EXEC_D64_cf_1_-567554490 | 0.9942 |
Give me the best precision achieved by a given algorithm perfomed over dataset called News.*
PREFIX mexcore: <http://mex.aksw.org/mex-core#>
PREFIX mexperf: <http://mex.aksw.org/mex-perf#>
PREFIX mexalgo: <http://mex.aksw.org/mex-algo#>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT DISTINCT ?ExecutionID ?Algorithm ?precision ?execution WHERE {
?execution prov:wasInformedBy ?conf.
?execution prov:id ?ExecutionID.
?conf prov:used ?dataset.
?dataset dcterms:title "News*".
?perfn prov:wasGeneratedBy ?execution.
?perfn mexperf:precision ?precision.
?execution prov:used ?alg.
?alg a mexalgo:Algorithm.
?alg rdfs:label ?Algorithm.
?alg rdf:type ?algn.
}
ORDER BY DESC (?precision)
LIMIT 30
Result:
| ExecutionID | Algorithm | precision | execution |
|---|---|---|---|
| "C0_MEX_EXEC_D12" | "Naive Bayes" | 0.9842 | exe_12_cf_1_-568620203 |
| "C0_MEX_EXEC_D20" | "Simple Voting" | 0.9833 | exe_20_cf_1_-568620203 |
| "C0_MEX_EXEC_D32" | "NER ILLINOIS EXTENDED" | 0.9785 | exe_32_cf_1_-568620203 |
| "C0_MEX_EXEC_D16" | "Random Forest" | 0.9768 | exe_16_cf_1_-568620203 |
| "C0_MEX_EXEC_D44" | "BaggingJ48" | 0.9765 | exe_44_cf_1_-568620203 |
| "C0_MEX_EXEC_D64" | "Multilayer Perceptron" | 0.9752 | exe_64_cf_1_-568620203 |
| "C0_MEX_EXEC_D60" | "Functional Trees" | 0.9751 | exe_60_cf_1_-568620203 |
| "C0_MEX_EXEC_D24" | "Logistic Model Trees" | 0.9744 | exe_24_cf_1_-568620203 |
| "C0_MEX_EXEC_D28" | "Ada BoostM1" | 0.9743 | exe_28_cf_1_-568620203 |
| "C0_MEX_EXEC_D40" | "J48" | 0.974 | exe_40_cf_1_-568620203 |
| "C0_MEX_EXEC_D36" | "Class Level Voting" | 0.9731 | exe_36_cf_1_-568620203 |
| "C0_MEX_EXEC_D8" | "Sequential Minimal Optimization" | 0.9727 | exe_8_cf_1_-568620203 |
| "C0_MEX_EXEC_D56" | "Support Vector Machines" | 0.9722 | exe_56_cf_1_-568620203 |
| "C0_MEX_EXEC_D76" | "Logistic Regression" | 0.9716 | exe_76_cf_1_-568620203 |
| "C0_MEX_EXEC_D52" | "Additive Logistic Regression" | 0.9715 | exe_52_cf_1_-568620203 |
| "C0_MEX_EXEC_D72" | "NER BALIE" | 0.9571 | exe_72_cf_1_-568620203 |
| "C0_MEX_EXEC_D48" | "NER OPEN NLP" | 0.9565 | exe_48_cf_1_-568620203 |
| "C0_MEX_EXEC_D68" | "Decision Table" | 0.8743 | exe_68_cf_1_-568620203 |
| "C0_MEX_EXEC_D4" | "NER STANFORD" | 0.8279 | exe_4_cf_1_-568620203 |
| "C0_MEX_EXEC_D75" | "Logistic Regression" | 0.8262 | exe_75_cf_1_-568620203 |
| "C0_MEX_EXEC_D35" | "Class Level Voting" | 0.8034 | exe_35_cf_1_-568620203 |
| "C0_MEX_EXEC_D51" | "Additive Logistic Regression" | 0.8011 | exe_51_cf_1_-568620203 |
| "C0_MEX_EXEC_D63" | "Multilayer Perceptron" | 0.7956 | exe_63_cf_1_-568620203 |
| "C0_MEX_EXEC_D19" | "Logistic Regression" | 0.7789 | exe_19_cf_1_1523859061 |
| "C0_MEX_EXEC_D2" | "Additive Logistic Regression" | 0.7748 | exe_2_cf_1_1523859061 |
| "C0_MEX_EXEC_D47" | "NER OPEN NLP" | 0.7651 | exe_47_cf_1_-568620203 |
| "C0_MEX_EXEC_D7" | "BaggingJ48" | 0.7619 | exe_7_cf_1_1523859061 |
| "C0_MEX_EXEC_D59" | "Functional Trees" | 0.7609 | exe_59_cf_1_-568620203 |
| "C0_MEX_EXEC_D67" | "Decision Table" | 0.7559 | exe_67_cf_1_-568620203 |
| "C0_MEX_EXEC_D17" | "Functional Trees" | 0.7547 | exe_17_cf_1_1523859061 |
Give me the sampling method used by a given execution.
PREFIX mexcore: <http://mex.aksw.org/mex-core#>
PREFIX mexperf: <http://mex.aksw.org/mex-perf#>
PREFIX mexalgo: <http://mex.aksw.org/mex-algo#>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT DISTINCT ?execution ?sampling ?folds ?sequential ?testSize ?trainSize WHERE {
?execution prov:wasInformedBy ?conf.
?conf prov:used ?sampling.
?sampling mexcore:folds ?folds.
?sampling mexcore:sequential ?sequential.
?sampling mexcore:testSize ?testSize.
?sampling mexcore:trainSize ?trainSize.
}
ORDER BY ASC (?execution)
LIMIT 10
Result:
| execution | sampling | folds | sequential | testSize | trainSize |
|---|---|---|---|---|---|
| exe_10_cf_1_-567554490 | sm_cf_1_-567554490 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_-567674523 | sm_cf_1_-567674523 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_-568593294 | sm_cf_1_-568593294 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_-568620203 | sm_cf_1_-568620203 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_-568657719 | sm_cf_1_-568657719 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_1522910486 | sm_cf_1_1522910486 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_1522910578 | sm_cf_1_1522910578 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_1523859061 | sm_cf_1_1523859061 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_1523891668 | sm_cf_1_1523891668 | "10" | "true" | "0.1" | "0.9" |
| exe_10_cf_1_1524581609 | sm_cf_1_1524581609 | "10" | "true" | "0.1" | "0.9" |
SML-Bench is a benchmark framework for inductive learning tools. It currently supports ILP systems like Aleph, Progol and the GILPS implementations, as well as tools for learning on OWL-based ontologies, like the DL-Learner. SML-Bench is easily extensible with further systems or own prototypes and comes with a set of pre-defined learning scenarios. These scenarios define the dataset to learn on, positive and negative examples to inductively derive new knowledge from, and tool settings.
To make benchmark results better comparable and also to be able to load and share 'blueprints' for benchmark setups, a MEX import and output feature was added. The generation of RDF data describing a benchmark run is established by LOG4MEX annotations inside the SML-Bench Java code. As an alternative to a (flat) configuration file, an existing MEX benchmark description can be used to configure SML-Bench. A MEX description of a benchmark run comprises the specification of the assessed tools, their configured parameters and the algorithms they are based on using the mex-algo vocabulary. The configuration of the SML-Bench framework is done using elements from mex-core which describe aspects like the overall maximum execution time, the positive and negative examples used, the actual dataset and the utilized hardware. mex-perf was used to express the actual benchmarking results, e.g. the accuracy, F-score values achieved by the benchmarked inductive learning systems, as well as possible failures, e.g. due to timeouts. Following, we highlight an excerpt of SML integrated with LOG4MEX.
...
Configuration res = log.getValidationResults(scenarioSystem, fold);
if (res.containsKey(BenchmarkLog.tp) &&
res.containsKey(BenchmarkLog.fp) &&
res.containsKey(BenchmarkLog.tn) &&
res.containsKey(BenchmarkLog.fn)) {
int tp = res.getInt(BenchmarkLog.tp);
int fp = res.getInt(BenchmarkLog.fp);
int tn = res.getInt(BenchmarkLog.tn);
int fn = res.getInt(BenchmarkLog.fn);
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.TRUEPOSITIVE, tp);
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.FALSEPOSITIVE, fp);
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.TRUENEGATIVE, tn);
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.FALSENEGATIVE, fn);
// ----------------- accuracy -----------------
double sum = (tp + fp + tn + fn);
if (sum != 0) {
double acc = (tp + tn) / sum;
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.ACCURACY, acc);
}
// ----------------- F-score ------------------
double prec_denom = tp + fp;
double rec_denom = tp + fn;
if (prec_denom != 0 && rec_denom != 0) {
double precision = tp / (prec_denom);
double recall = tp /rec_denom;
if ((precision + recall) > 0) {
double fscore = 2 * ((precision * recall) / (precision + recall));
mex.Configuration(conf).Execution(exec).addPerformance(
EnumMeasures.F1MEASURE, fscore);
}
}
} else {
mex.Configuration(conf).Execution(exec).addPerformance(EnumMeasures.ERROR, -1);
}
...A query that collects results from more than one data sources is called federated query. The engines that allow executing federated queries are called federated engines. In SPARQL endpoint federation engines evaluation, the query runtime is the central performance measure. In addition, the number of data sources selected, the source selection time, the number of intermediate results, the precision and recall are also important measures to pinpoint the limitations of the federated engines.
We have two type of configurations to run the experiments: 1) we specify the specification (RAM, Hard disk, Processor, etc..) of all the machines running the SPARQL endpoints, 2) the specification of the machine running the federation engines. In this use case, we exported the [LargeRDFBench] (https://github.com/AKSW/largerdfbench) results in to mex machine readable format. The selected federated engines can be directly compared using simple SPARQL queries over mex results, optimizing the results analysis process.
Resource2Vec is a provider of knowledge graph embeddings of RDF datasets. The service comes with a self-contained RESTful API, which can be used online or deployed on a local machine. Given an RDF dataset and a configuration (e.g., desired rank of the output), the backend software transforms the original triple-based into a feature-based dataset in ARFF format and publishes it on OpenML, a portal for collaborative machine learning backed by a strong, active community.
Resource2Vec uses the mex vocabulary to describe the datasets containing embeddings.
The following is an excerpt of a JSON-LD specification of a Resource2Vec dataset.
{
"@id" : "this:MC0_MEX_EXEC_D1_1",
"@type" : [ "mexperf:StatisticalMeasure", "prov:Entity" ],
"mexperf:processingTime" : 3245,
"wasInformedBy" : "this:EXECC0_MEX_EXEC_D1"
}, {
"@id" : "this:PAR1",
"@type" : [ "mexcore:HyperParameter", "prov:Entity" ],
"identifier" : "rank",
"value" : "2"
}, {
"@id" : "this:rescal",
"@type" : [ "mexalgo:Algorithm", "prov:Entity" ],
"hasParameterCollection" : "this:PAR1",
"identifier" : "algorithm1",
"hasAlgorithmClass" : "mexalgo:TensorFactorization"
}, {
"@id" : "this:application",
"@type" : [ "mexcore:ApplicationContext", "prov:Organization", "prov:Person", "prov:Agent" ],
"dateCopyrighted" : "2016.05.01 01:31:54 CEST",
"givenName" : "Tommaso Soru",
"mbox" : "tsoru@informatik.uni-leipzig.de"
},
WASOTA Project Website
[1] - thanks to lodview