Check the preview of 2nd version of this platform being developed by the open MLCommons taskforce on automation and reproducibility as a free, open-source and technology-agnostic on-prem platform.
program:generate-target-latency (v3.0.0)
Share
Copyright: See copyright in the source repository
License: See license in the source repository
Creation date: 2021-03-04
Source: GitHub
cID: b0ac08fe1d3c2615:372a4a6ca5cef5ba

Don't hesitate to get in touch if you encounter any issues or would like to discuss this community project!
Please report if this CK component works: 1 ▲ or fails: 0 ▲
Sign up to be notified when artifacts are shared or updated!

Description  

This portable workflow is our attempt to provide a common CLI with Python JSON API and a JSON meta description to automatically detect or install required components (models, data sets, libraries, frameworks, tools), and then build, run, validate, benchmark and auto-tune the associated method (program) across diverse models, datasets, compilers, platforms and environments. Our on-going project is to make the onboarding process as simple as possible via this platform. Please check this CK white paper and don't hesitate to contact us if you have suggestions or feedback!
  • Automation framework: CK
  • Development repository: ck-ml
  • Source: GitHub
  • Available command lines:
    • ck run program:generate-target-latency --cmd_key=default (META)
  • Support for host OS: any
  • Support for target OS: any
  • Tags: program,lang-python,mlperf-inference,mlperf
  • How to get the stable version via the client:
    pip install cbench
    cb download program:generate-target-latency --version=3.0.0 --all
    ck run program:generate-target-latency
  • How to get the development version:
    pip install ck
    ck pull repo:ck-ml
    ck run program:generate-target-latency
  • CLI and Python API: module:program

    • Dependencies    

    ReadMe  

    Description

    This program performs "performance ranging" by extracting the mean latency over a fixed number of queries. This latency can then be used for proper performance runs. This is especially useful for families of models such as MobileNet and EfficientNet where the performance can range by an order of magnitude.

    Usage

    Store

    Run cmdgen:benchmark.* with --scenario=range_singlestream --max_query_count=<...> arguments to store performance ranging results to the local CK repo.

    Example

    $ ck run cmdgen:benchmark.image-classification.tflite-loadgen --library=tflite-v2.4.1-ruy \
--model:=`ck list_variations misc --query_module_uoa=package --tags=model,tflite,effnet --variation_prefix=lite --separator=:` \
--model_extra_tags,=non-quantized,quantized \
--mode=performance --scenario=range_singlestream --max_query_count=256 \
--sut=rpi4coral --fan_mode=on

    Extract

    Use this script to extract the ranging results from the CK repo. All arguments are optional.

    $ ck run ck-mlperf:program:generate-target-latency \
  --env.CK_MLPERF_SUBMISSION_REPO=local \
  --env.CK_MLPERF_SUBMISSION_TAGS=foo,bar \
  --env.CK_MLPERF_SUBMISSION_OUT="$PWD"/target_latency.txt

    Alternatively:

    $ cd $(ck find program:generate-target-latency) && ./run.py \
  --repo-uoa local \
  --tags foo,bar \
  --out $(pwd)/target_latency.txt

    Example

    $ $(ck find program:generate-target-latency)/run.py --tags=inference_engine.tflite | sort | tee $(ck find program:image-classification-tflite-loadgen)/target_latency.rpi4coral.txt
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite0-non-quantized   70 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite0-quantized   34 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite1-non-quantized   99 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite1-quantized   49 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite2-non-quantized  136 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite2-quantized   67 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite3-non-quantized  203 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite3-quantized  101 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite4-non-quantized  342 # max_query_count=256
rpi4coral,tflite-v2.4.1-ruy,efficientnet-lite4-quantized  169 # max_query_count=256

    Tags

    The tags are related to experiments, i.e. the following query matches the same experiments as ./run.py --tags=$tags:

    $ ck search experiment --tags=mlperf,scenario.range_singlestream.$tags`

    You can use the following one-liner to list tags for particular ranging experiments:

     $ for i in $(ck search experiment --tags=mlperf,scenario.range_singlestream); do echo $i; ck list_tags $i; echo; done

    Use

    Run cmdgen:benchmark.* with --target_latency_file=<...> instead of --target_latency=<...>.

    Example

    $ ck run cmdgen:benchmark.image-classification.tflite-loadgen --library=tflite-v2.4.1-ruy \
--model:=`ck list_variations misc --query_module_uoa=package --tags=model,tflite,effnet --variation_prefix=lite --separator=:` \
--model_extra_tags,=non-quantized,quantized \
--mode=performance --scenario=singlestream --sut=rpi4coral --fan_mode=on \
--target_latency_file=$(ck find program:image-classification-tflite-loadgen)/target_latency.rpi4coral.txt \
--power=yes --power_server_port=4951 --power_server_ip=192.168.0.3

    Versions  

    Files  

    Comments  

    Please log in to add your comments!
    If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!