Collective Knowledge solution

Object detection • MLPerf inference • TensorFlow CPU • COCO • 50 images validation • Raspberry Pi • benchmark • Portable Workflows

solution:demo-obj-detection-coco-tf-cpu-benchmark-rpi-portable-workflows (v1.3.0)

Portable solution description

Install and run this solution on your platform in several simple steps. Our goal is to make it simpler to reproduce results from research papers, participate in crowd-benchmarking, and enable "live" papers.

What is the portable CK solution: docs
Reproduced paper: MLPerf Inference Benchmark
Reproduced SOTA results (crowd-benchmarking): sota-mlperf-object-detection-v0.5-crowd-benchmarking
Auto-tuning results: (Pareto-efficient ML/SW/HW co-design)
Number of collaborative runs by the community: 164
Reproducibility checklist: ACM/cTuning
Automation framework: CK
CK repository: A collection of all AI and ML workflows and packages in the CK format
Benchmarking pipeline (build, run, validate): program:object-detection-tf-py-benchmark
Workflow CLI and Python API (compile, run, benchmark, validate): module:program
Workflow dependencies (model, dataset, framework): View
Workflow params: --repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=50 --env.CK_METRIC_TYPE=COCO --no_state_check --skip_print_timers
Other platforms: Linux (50 images) Linux (5000 images) MacOS (in progress) Android Docker (Windows/Linux)

Don't hesitate to get in touch if you encounter any issues or would like to discuss this community project!

Check the prerequisites for your system

Install manually from the command line (to be automated in the future):

# Tested on Raspberry Pi 4

# These dependencies are needed to rebuild COCO API:

 sudo apt update
 sudo apt install git wget libz-dev curl cmake
 sudo apt install gcc g++ autoconf autogen libtool
 sudo apt install libfreetype6-dev
 sudo apt install python3.7-dev
 sudo apt install -y libsm6 libxext6 libxrender-dev

Install cBench (docs)

Install cBench from the command line (a small Python library to manage CK solutions):

pip3 install cbench
 or 
python3 -m pip install cbench
 or
pip install cbench

Note that you may need to add the --user flag if you install in your user space, i.e. "python3 -m pip install cbench --user"

Init this solution with the portable workflow on your machine

Run manually from your command line (cBench will attempt to automatically adapt this workflow to your system - you may need to press Enter several times to select default answers for some questions):

cb init demo-obj-detection-coco-tf-cpu-benchmark-rpi-portable-workflows

Start cBench (status: disconnected)

cb start

Participate in collaborative crowd-benchmarking of this solution

or start crowd-benchmarking manually from the command line:

cb benchmark demo-obj-detection-coco-tf-cpu-benchmark-rpi-portable-workflows

Run this workflow locally

or start local run manually from the command line:

cb run demo-obj-detection-coco-tf-cpu-benchmark-rpi-portable-workflows

  # Note that the following CK program pipeline will be executed:
  ck compile program:object-detection-tf-py-benchmark --cmd_key=default --speed
  ck run program:object-detection-tf-py-benchmark --cmd_key=default

Successfully tested configuration

Host OS: linux-32 (Raspbian GNU/Linux 10 (buster))
Target OS: linux-32 (Raspbian GNU/Linux 10 (buster))
Target machine: Raspberry (Raspberry Pi 4 Model B Rev 1.1)
Target CPU: BCM2835
Target CPUs:

Python min version: 3.6
Python max version: 3.7.99

Dependencies

Reused CK components

These components are automatically installed by cBench from this portal:

cb download --version=1.0.0 --force package:model-tf-mlperf-ssd-mobilenet
cb download --version=1.0.0 --force soft:model.tensorflow.object-detection
cb download --version=1.0.0 --force script:install-package-tensorflowmodel-object-detection

cb download --version=1.0.0 --force package:lib-python-numpy
cb download --version=1.0.0 --force soft:lib.python.numpy

cb download --version=1.0.0 --force package:lib-python-scipy
cb download --version=1.0.0 --force soft:lib.python.scipy

cb download --version=1.0.0 --force package:lib-python-matplotlib
cb download --version=1.0.0 --force soft:lib.python.matplotlib

cb download --version=1.0.0 --force package:lib-python-pillow
cb download --version=1.0.0 --force soft:lib.python.pillow

cb download --version=1.0.0 --force package:lib-python-cython
cb download --version=1.0.0 --force soft:lib.python.cython

cb download --version=1.0.0 --force package:lib-python-cv2
cb download --version=1.0.0 --force soft:lib.python.cv2

cb download --version=1.0.0 --force package:tool-coco-codereef
cb download --version=1.0.0 --force soft:tool.coco

cb download --version=1.0.1 --force package:dataset-coco-2017-val-small
cb download --version=1.0.0 --force soft:dataset.coco.2017.val

cb download --version=1.0.2 --force package:lib-tensorflow-1.1.0-cpu
cb download --version=1.0.0 --force package:lib-tensorflow-1.4.0-cpu
cb download --version=1.0.0 --force package:lib-tensorflow-1.13.1-cpu
cb download --version=1.0.0 --force package:lib-tensorflow-1.14.0-cpu
cb download --version=1.0.0 --force soft:lib.tensorflow

cb download --version=1.0.1 --force package:lib-tensorflow-pip
cb download --version=1.0.0 --force soft:lib.python.tensorflow

cb download --version=1.0.0 --force package:labelmap-coco
cb download --version=1.0.0 --force soft:labelmap.object-detection

cb download --version=1.0.0 --force soft:compiler.gcc

cb download --version=1.0.0 --force package:tensorflowmodel-api-object-detection
cb download --version=1.0.0 --force soft:model.tensorflow-models-api

cb download --version=1.0.0 --force package:lib-protobuf-3.0.0-host
cb download --version=1.0.0 --force soft:lib.protobuf.host

cb download --version=1.0.0 --force script:process-compiler-for-cmake

cb download --version=1.0.0 --force soft:tool.cmake

cb download --version=1.0.0 --force soft:lib.protobuf.host

cb download --version=1.0.1 --force module:program
cb download --version=1.0.0 --force module:dataset
cb download --version=1.0.0 --force module:pipeline
cb download --version=1.0.0 --force module:choice
cb download --version=1.0.0 --force module:experiment
cb download --version=1.0.0 --force module:math.variation

cb download --version=1.4.0 --force program:object-detection-tf-py-benchmark




#ck install package:lib-tensorflow-pip --env.PACKAGE_VERSION=1.14.0 # Doens't install PIL needed further (1.13.1 package installs it)
#ck install package:lib-tensorflow-1.13.1-cpu --env.VIA_PYPI --env.ALLOW_ALL_BITS
ck install package:lib-tensorflow-1.14.0-cpu --env.VIA_PYPI --env.ALLOW_ALL_BITS

ck install package --tags=model,tf,object-detection,mlperf,ssd-mobilenet,non-quantized

ck install package --tags=lib,python-package,numpy
ck install package --tags=lib,python-package,scipy --force_version=1.2.1
ck install package --tags=lib,python-package,matplotlib
ck install package --tags=lib,python-package,pillow
ck install package --tags=lib,python-package,cython
ck install package --tags=lib,python-package,cv2,opencv-contrib-python --force_version=3.4.3.18

ck install package:tool-coco-codereef

ck install package:dataset-coco-2017-val-small

Comments

Please log in to add your comments!

If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!