This live report demonstrates how CK can help create researchers reproducible and interactive article from reusable components


Here we provide details how to reproduce experiments from our papers via Collective Knowledge Framework and Repository (CK). All artifacts are shared via CK "reproduce-ck-paper" repository at GitHub. If you have CK installed (docs), you can download and install this repository via CMD simply as
    ck pull repo:reproduce-ck-paper
   
We would like to thank the community for validating various results and sharing unexpected behavior with us via our public CK repository!

Abstract

Empirical auto-tuning and machine learning techniques have been showing high potential to improve execution time, power consumption, code size, reliability and other important metrics of various applications for more than two decades. However, they are still far from widespread production use due to lack of native support for auto-tuning in an ever changing and complex software and hardware stack, large and multi-dimensional optimization spaces, excessively long exploration times, and lack of unified mechanisms for preserving and sharing of optimization knowledge and research material.

We present a possible collaborative approach to solve above problems using Collective Mind knowledge management system. In contrast with previous cTuning framework, this modular infrastructure allows to preserve and share through the Internet the whole auto-tuning setups with all related artifacts and their software and hardware dependencies besides just performance data. It also allows to gradually structure, systematize and describe all available research material including tools, benchmarks, data sets, search strategies and machine learning models. Researchers can take advantage of shared components and data with extensible meta-description to quickly and collaboratively validate and improve existing auto-tuning and benchmarking techniques or prototype new ones. The community can now gradually learn and improve complex behavior of all existing computer systems while exposing behavior anomalies or model mispredictions to an interdisciplinary community in a reproducible way for further analysis. We present several practical, collaborative and model-driven auto-tuning scenarios. We also decided to release all material at c-mind.org/repo to set up an example for a collaborative and reproducible research as well as our new publication model in computer engineering where experimental results are continuously shared and validated by the community.


Raising number of compiler flags in GCC

Reproducibility: CK scripts with CID=84e27ad9dd12e734:bbf4abb470b0271a

Note the trend of slowing down total number of boolean optimizations and rising number of parameteric optimizations.

Unified, reproducible, multi-objective, compiler flag autotuning examples powered by CK

We prepared Getting Started Guide about how to reproduce various autotuning scenarios (including from this paper) via CK in your environment (hardware and software). You can find it here.

Interactive graph of slambench v1.1 cpu autotuning

Reproduce/reuse/replay/discuss via CK (interactive graphs)

$#ck_include_start#${"cid":"2d41f89bcf32d4d4:3645af97abb448ab", "where":"div#ck_interactive_f6eb74c75ae8565f", "html":"2d_points_time_vs_size_with_pareto_interactive.html", "style":"2d_points_time_vs_size_with_pareto_interactive.style"}$#ck_include_stop#$
More examples of reproducible, collaborative and live slambench experiments via CK:

Non-interactive graph of image corner detection algorith (susan corners) autotuning

Reproduce/reuse/replay/discuss via CK (interactive graphs)



Non-interactive graph of analysis of execution time variation of image/neuron filter kernel versus different frequency

CK scripts: reproduce.py
Density graph

Histogram graph


Note: CK allows the community validate the results and share unexpected behavior in public cknowledge.org/repo here.

Note: we are gradually converting all the code and data related to this paper from the deprecated Collective Mind Format to the new Collective Knowledge Framework.