Collective Knowledge solution

Object detection • MLPerf inference v0.5 • OpenVINO • SSD MobileNet v1 • COCO • 500 images validation • Linux • benchmark • portable workflows

solution:mlperf-inference-v0.5-detection-openvino-ssd-mobilenet-coco-500-linux (v1.0.0)

Portable solution description

Install and run this solution on your platform in several simple steps. Our goal is to make it simpler to reproduce results from research papers, participate in crowd-benchmarking, and enable "live" papers.

What is the portable CK solution: docs
Reproduced paper: MLPerf Inference Benchmark
Reproduced SOTA results (crowd-benchmarking): mlperf-inference-v0.5-classification
Auto-tuning results: (Pareto-efficient ML/SW/HW co-design)
Number of collaborative runs by the community: 12
Reproducibility checklist: ACM/cTuning
Automation framework: CK
CK repository: A collection of all AI and ML workflows and packages in the CK format
Benchmarking pipeline (build, run, validate): program:mlperf-inference-v0.5
Workflow CLI and Python API (compile, run, benchmark, validate): module:program
Workflow dependencies (model, dataset, framework): View
Workflow params: --repetitions=1 --no_state_check --skip_print_timers --env.CK_LOADGEN_SCENARIO=Offline --env.CK_LOADGEN_MODE=Accuracy --env.CK_LOADGEN_DATASET_SIZE=500 --env.CK_OPENVINO_NTHREADS=$NPROCS --env.CK_OPENVINO_NSTREAMS=$NPROCS --env.CK_OPENVINO_NIREQ=$NPROCS

Don't hesitate to get in touch if you encounter any issues or would like to discuss this community project!

Check the prerequisites for your system

Install manually from the command line (to be automated in the future):

# Tested on Ubuntu 18.04 
# (including Docker and Windows 10 Sybsystem for Linux)

# These dependencies are needed to rebuild COCO API:

 sudo apt update

 sudo apt install git wget libz-dev zip curl 
 sudo apt install python3 python3-pip
 sudo apt install gcc g++ autoconf autogen libtool cmake

Install cBench (docs)

Install cBench from the command line (a small Python library to manage CK solutions):

pip3 install cbench
 or 
python3 -m pip install cbench
 or
pip install cbench

Note that you may need to add the --user flag if you install in your user space, i.e. "python3 -m pip install cbench --user"

Init this solution with the portable workflow on your machine

Run manually from your command line (cBench will attempt to automatically adapt this workflow to your system - you may need to press Enter several times to select default answers for some questions):

cb init mlperf-inference-v0.5-detection-openvino-ssd-mobilenet-coco-500-linux

Start cBench (status: disconnected)

cb start

Participate in collaborative crowd-benchmarking of this solution

or start crowd-benchmarking manually from the command line:

cb benchmark mlperf-inference-v0.5-detection-openvino-ssd-mobilenet-coco-500-linux

Run this workflow locally

or start local run manually from the command line:

cb run mlperf-inference-v0.5-detection-openvino-ssd-mobilenet-coco-500-linux

  # Note that the following CK program pipeline will be executed:
  ck compile program:mlperf-inference-v0.5 --cmd_key=object-detection --speed
  ck run program:mlperf-inference-v0.5 --cmd_key=object-detection

Successfully tested configuration

Host OS: linux-64 (Ubuntu 16.04.6 LTS)
Target OS: linux-64 (Ubuntu 16.04.6 LTS)
Target machine: Hewlett-Packard (HP Z640 Workstation)
Target CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
Target CPUs:

Python min version: 3.6
Python max version: 3.7.99

Dependencies

Reused CK components

These components are automatically installed by cBench from this portal:

# Get stable components for the CK solution
cbench download package:lib-boost-1.67.0-without-python --force
cbench download package:mlperf-inference-source-ck --force
cbench download package:lib-openvino-ck --force


# Pull CK repositories (including ck-mlperf, ck-env, ck-autotuning, ck-tensorflow, ck-docker).
ck pull repo:ck-openvino

# Use generic Linux settings with dummy frequency setting scripts.
ck detect platform.os --platform_init_uoa=generic-linux-dummy

# Detect C/C++ compiler (gcc).
ck detect soft:compiler.gcc

# Detect CMake build tool.
#ck detect soft --tags=cmake --full_path=`which cmake`


# Install the latest Python package installer (pip) and some dependencies.
python -m pip install --ignore-installed pip setuptools



#-----------------------------------------------------------------------------#
# Step 1. Install Python dependencies (for Model Optimizer and LoadGen).
#-----------------------------------------------------------------------------#
# OpenVINO pre-release strictly requires TensorFlow < 2.0 and NetworkX < 2.4.
ck install package --tags=lib,python-package,tensorflow --force_version=1.15.2
ck install package --tags=lib,python-package,networkx --force_version=2.3.0
ck install package --tags=lib,python-package,defusedxml
# Cython is an implicit dependency of NumPy.
ck install package --tags=lib,python-package,cython
ck install package --tags=lib,python-package,numpy
# test-generator is an implicit dependency of Model Optimizer (not in requirements.txt).
ck install package --tags=lib,python-package,test-generator
# Abseil is a LoadGen dependency.
ck install package --tags=lib,python-package,absl


#-----------------------------------------------------------------------------#
# Step 2. Install C++ dependencies (for Inference Engine and MLPerf program).
#-----------------------------------------------------------------------------#
ck install package --tags=channel-stable,opencv,v3.4.3
ck install package:lib-boost-1.67.0-without-python
# Install LoadGen from a branch reconstructed according to Intel's README.
ck install package --tags=ck-mlperf,inference,source,dividiti.v0.5-intel
ck install package --tags=lib,loadgen,static
#-----------------------------------------------------------------------------#


#-----------------------------------------------------------------------------#
# Step 3. Install the OpenVINO "pre-release" used for MLPerf Inference v0.5.
#-----------------------------------------------------------------------------#
ck install package --tags=lib,ck-openvino,pre-release
ck compile ck-openvino:program:mlperf-inference-v0.5
#-----------------------------------------------------------------------------#


#-----------------------------------------------------------------------------#
# Step 4. Install the first 500 images of the ImageNet 2012 validation dataset.
# TODO: Create a calibration dataset.
#-----------------------------------------------------------------------------#
ck install package --tags=dataset,imagenet,val,min --no_tags=resized
ck install package --tags=dataset,imagenet,aux
# The OpenVINO program expects to find val_map.txt in the dataset directory.
head -n 500 `ck locate env --tags=aux`/val.txt > `ck locate env --tags=val`/val_map.txt
# Install misc Python dependencies required for calibration.
python  -m pip install nibabel pillow progress py-cpuinfo pyyaml shapely sklearn tqdm xmltodict yamlloader
# Install "headless" OpenCV (which doesn't need libsm6, libxext6, libxrender-dev).
ck install package --tags=lib,python-package,cv2,opencv-python-headless
#-----------------------------------------------------------------------------#


#-----------------------------------------------------------------------------#
# Step 7. Install the official SSD-MobileNet model for MLPerf Inference v0.5
# and convert it into the OpenVINO format.
#-----------------------------------------------------------------------------#
ck install package --tags=model,tf,ssd-mobilenet,quantized,for.openvino
ck install package --tags=model,openvino,ssd-mobilenet --safe
#-----------------------------------------------------------------------------#


#-----------------------------------------------------------------------------#
# Step 8. Install the COCO 2017 validation dataset (5,000 images).
#-----------------------------------------------------------------------------#
# Download the dataset to the default path. Remove all training annotations (~765 MB).
echo | ck install package --tags=object-detection,dataset,coco.2017,val,original,full && ck virtual env --tags=object-detection,dataset,coco.2017,val,original,full --shell_cmd='rm $CK_ENV_DATASET_COCO_LABELS_DIR/*train2017.json'
# Install Python COCO API.
#ck install package --tags=lib,python-package,cython # already installed
#ck install package --tags=lib,python-package,numpy # already installed
ck install package --tags=lib,python-package,matplotlib
ck install package --tags=tool,coco,api
#-----------------------------------------------------------------------------#

Comments

Please log in to add your comments!

If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!