This portal has been archived. Explore the next generation of this technology.
package:model-tflite-mlperf-ssd-mobilenet (v3.0.0)
Please report if this CK component works: 1 ▲
or fails: 0 ▲
Sign up to be notified when artifacts are shared or updated!
This meta package is our attempt to provide a unified Python API, CLI
and JSON meta description
for different package managers and building tools
to automatically download and install different components (models, data sets, libraries, frameworks, tools)
necessary to run portable program pipelines
across evolving platforms.
Our on-going project is to make the onboarding process as simple as possible via this platform.
Please check this CK white paper
and don't hesitate to contact us if you have suggestions or feedback!
Step 1:
Step 2: from
Option 1: from
Option 2: from
If you notice any inapropriate content that should not be here, please report us as soon as possible and we will try to remove it within 48 hours!
Copyright: See copyright in the source repository
License: See license in the source repository
Creation date: 2019-03-27
Source: GitHub
cID: 1dc07ee0f4742028:ad9565c4d07b3d1d
Don't hesitate to get in touch if you encounter any issues or would like to discuss this community project!
License: See license in the source repository
Creation date: 2019-03-27
Source: GitHub
cID: 1dc07ee0f4742028:ad9565c4d07b3d1d
Description
- Automation framework: CK
- Development repository: ck-ml
- Source: GitHub
- Tags: model,tflite,object-detection,mlperf,ssd-mobilenet-v1,ssd-mobilenet,ssd,mobilenet,mobilenet-v1,nhwc,vcoco,non-quantized,downloaded
- Support for host OS: any
- Support for target OS: any
- Version: reference
- Software detection plugin: model.tensorflow.py
- How to get the stable version from this portal via the CK client:
pip install cbench
cb download package:model-tflite-mlperf-ssd-mobilenet --version=3.0.0 --all
ck install package:model-tflite-mlperf-ssd-mobilenet
ck help package
- How to get the development version:
pip install ck
ck pull repo:ck-ml
ck install package:model-tflite-mlperf-ssd-mobilenet
ck help package
- Useful CK automation actions (CLI with the Python CK API and JSON IO):
ck show env
ck find env:{ID from above}
ck virtual env:{ID from above}
- CLI and Python API: module:package
- Variations:
- from-zenodo
- from-dropbox
- Useful CK automation actions (CLI with the Python CK API and JSON IO):
ReadMe
MLPerf Inference - Object Detection - SSD-MobileNet (TFLite)
This model was converted to TFLite from the original TF model in two steps, by adapting instructions from Google's blog.
Step 1: model.ckpt.* to tflite_graph.pb
We tested this step with TensorFlow v1.11-v1.13, either prebuilt or built from source.
NB: On 25/Apr/2019 we informed Google of a bug in their converter, which can be fixed e.g. as follows:
anton@diviniti:~/CK_TOOLS/tensorflowmodel-api-master/models/research$ git diff
diff --git a/research/object_detection/export_tflite_ssd_graph.py b/research/object_detection/export_tflite_ssd_graph.py
index b7ed428..1b52335 100644
--- a/research/object_detection/export_tflite_ssd_graph.py
+++ b/research/object_detection/export_tflite_ssd_graph.py
@@ -136,7 +136,7 @@ def main(argv):
export_tflite_ssd_graph_lib.export_tflite_graph(
pipeline_config, FLAGS.trained_checkpoint_prefix, FLAGS.output_directory,
FLAGS.add_postprocessing_op, FLAGS.max_detections,
- FLAGS.max_classes_per_detection, FLAGS.use_regular_nms)
+ FLAGS.max_classes_per_detection, FLAGS.detections_per_class, FLAGS.use_regular_nms)
if __name__ == '__main__':
This was fixed upstream on 31/May/2019 (albeit in a somewhat less elegant way).
Manual instructions
Install TensorFlow
$ python -m pip install tensorflow --user
$ python -c "import tensorflow as tf; print(tf.__version__)"
1.13.1
Install TensorFlow Object Detection API
$ export TF_MODEL_API=...
Install TensorFlow SSD-MobileNet model
$ export $TMP_DIR=/tmp && cd ${TMD_DIR}
$ wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz
$ tar xvzf ssd_mobilenet_v1_coco_2018_01_28.tar.gz
$ export TF_MODEL_DIR=${PWD}/ssd_mobilenet_v1_coco_2018_01_28
$ ls -la ${TF_MODEL_DIR}
total 58176
drwxr-xr-x 3 anton anton 4096 Feb 1 2018 .
drwxrwxrwt 18 root root 36864 Apr 26 11:42 ..
-rw-r--r-- 1 anton anton 77 Feb 1 2018 checkpoint
-rw-r--r-- 1 anton anton 29103956 Feb 1 2018 frozen_inference_graph.pb
-rw-r--r-- 1 anton anton 27380740 Feb 1 2018 model.ckpt.data-00000-of-00001
-rw-r--r-- 1 anton anton 8937 Feb 1 2018 model.ckpt.index
-rw-r--r-- 1 anton anton 3006546 Feb 1 2018 model.ckpt.meta
-rw-r--r-- 1 anton anton 4138 Feb 1 2018 pipeline.config
drwxr-xr-x 3 anton anton 4096 Feb 1 2018 saved_model
Convert
$ cd ${TF_MODEL_API}/research
$ export PYTHONPATH=.:./slim:$PYTHONPATH
$ export TFLITE_MODEL_DIR=${TF_MODEL_DIR}
$ python object_detection/export_tflite_ssd_graph.py \
--input_type image_tensor \
--pipeline_config_path ${TF_MODEL_DIR}/pipeline.config \
--trained_checkpoint_prefix ${TF_MODEL_DIR}/model.ckpt \
--output_directory ${TFLITE_MODEL_DIR} \
--add_postprocessing_op=true \
--use_regular_nms=true \
--config_override " \
model { \
ssd { \
post_processing { \
batch_non_max_suppression { \
score_threshold: 0.3 \
iou_threshold: 0.6 \
max_detections_per_class: 100 \
max_total_detections: 100 \
} \
} \
} \
} \
"
Semi-automated instructions
Install TensorFlow
$ ck install package --tags=lib,tensorflow,v1.13,vcpu
More than one package or version found:
0) lib-tensorflow-1.13.1-src-cpu Version 1.13.1 (333b554fb5b0e443)
1) lib-tensorflow-1.13.1-cpu Version 1.13.1 (88ad16f0bcfb4ae2)
Please select the package to install [ hit return for "0" ]:
Option 1 is faster, but option 0 can be used for Step 2 (where source code is needed).
Install TensorFlow Object Detection API
$ ck install package --tags=model,tensorflow,api
Install TensorFlow SSD-MobileNet model
$ ck install package --tags=model,tensorflow,mlperf,ssd-mobilenet,non-quantized
Descend into virtual environments one by one
TensorFlow
$ ck virtual env --tags=lib,tensorflow,v1.13,vcpu
$ ${CK_ENV_COMPILER_PYTHON_FILE} -c "import tensorflow as tf; print(tf.__version__)"
NB: Using ${CK_ENV_COMPILER_PYTHON_FILE} should ensure that the same version of
Python that was used to install TensorFlow and its dependencies (e.g. /usr/bin/python3.6)
will be used to run the conversion script.
TensorFlow Object Detection API
$ ck virtual env --tags=model,tensorflow,api
$ echo ${CK_ENV_TENSORFLOW_MODELS_OBJ_DET_DIR}
/home/anton/CK_TOOLS/tensorflowmodel-api-master/models/research/object_detection
TensorFlow SSD-MobileNet model
$ ck virtual env --tags=model,tensorflow,mlperf,ssd-mobilenet,non-quantized
$ echo ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE}
/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet/model.ckpt
$ echo "$(dirname ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE})"
/home/anton/CK_TOOLS/model-tf-mlperf-ssd-mobilenet
TODO: Need to introduce an environment variable for the model directory,
so not having to use the $(dirname ...) idiom all the time.
Convert
$ ${CK_ENV_COMPILER_PYTHON_FILE} \
${CK_ENV_TENSORFLOW_MODELS_OBJ_DET_DIR}/export_tflite_ssd_graph.py \
--input_type image_tensor \
--pipeline_config_path "$(dirname ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE})"/pipeline.config \
--trained_checkpoint_prefix "$(dirname ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE})"/model.ckpt \
--output_directory "$(dirname ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE})" \
--add_postprocessing_op=true \
--use_regular_nms=true \
--config_override " \
model { \
ssd { \
post_processing { \
batch_non_max_suppression { \
score_threshold: 0.3 \
iou_threshold: 0.6 \
max_detections_per_class: 100 \
max_total_detections: 100 \
} \
} \
} \
} \
"
$ grep -A 3 use_regular_nms "$(dirname ${CK_ENV_TENSORFLOW_MODEL_WEIGHTS_FILE})"/tflite_graph.pbtxt
key: "use_regular_nms"
value {
b: true
}
Step 2: from tflite_graph.pb to detect*.tflite
TODO: Update with manual and semi-automatic instructions.
We tested this step with (the source of) TensorFlow v1.11-v1.13 and Bazel v0.20.0.
Option 1: from tflite_graph.pb to detect.tflite
$ bazel run -c opt tensorflow/contrib/lite/toco:toco -- \
--input_file=${TFLITE_MODEL_DIR}/tflite_graph.pb \
--output_file=${TFLITE_MODEL_DIR}/detect.tflite \
--input_shapes=1,300,300,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--inference_type=FLOAT \
--allow_custom_ops
...
INFO: Invocation ID: b49af84b-4ef0-44c9-adf0-236700f8cd86
INFO: Analysed target //tensorflow/lite/toco:toco (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //tensorflow/lite/toco:toco up-to-date:
bazel-bin/tensorflow/lite/toco/toco
INFO: Elapsed time: 0.247s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tensorflow/lite/toco/toco '--input_file=/home/ivan/Downloads/tflite_ssd_mobilenet_v1_coco_2018_01_28/tflite_graph.pb' '--output_file=/home/ivan/Downloads/tflite_ssd_mobilenet_v1_coco_2018_01_28/_n2_detect.tflite' '--input_shapes=1,300,300,3' '--input_arrays=normalized_input_image_tensor' '--output_arrays=TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3' '--infeINFO: Build completed successfully, 1 total action
2019-04-22 09:11:44.344086: I tensorflow/lite/toco/import_tensorflow.cc:1324] Converting unsupported operation: TFLite_Detection_PostProcess
2019-04-22 09:11:44.354655: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 500 operators, 754 arrays (0 quantized)
2019-04-22 09:11:44.366436: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 500 operators, 754 arrays (0 quantized)
2019-04-22 09:11:44.401719: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 64 operators, 176 arrays (0 quantized)
2019-04-22 09:11:44.402990: I tensorflow/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 64 operators, 176 arrays (0 quantized)
2019-04-22 09:11:44.405046: I tensorflow/lite/toco/allocate_transient_arrays.cc:345] Total transient array allocated size: 11520000 bytes, theoretical optimal value: 8640000 bytes.
2019-04-22 09:11:44.405323: I tensorflow/lite/toco/toco_tooling.cc:399] Estimated count of arithmetic ops: 2.49483 billion (note that a multiply-add is counted as 2 ops).
2019-04-22 09:11:44.405706: W tensorflow/lite/toco/tflite/operator.cc:1407] Ignoring unsupported type in list attribute with key '_output_types'
Option 2: from tflite_graph.pb to detect_cut.tflite
$ bazel run -c opt tensorflow/contrib/lite/toco:toco -- \
--input_file=${TFLITE_MODEL_DIR}/tflite_graph.pb \
--output_file=${TFLITE_MODEL_DIR}/detect_cut.tflite \
--input_shapes=1,300,300,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='raw_outputs/box_encodings','raw_outputs/class_predictions' \
--inference_type=FLOAT \
--allow_custom_ops
Reference accuracy
Regular NMS
$ ck benchmark program:object-detection-tflite --env.USE_NMS=regular \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=5000 --env.CK_METRIC_TYPE=COCO \
--record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-accuracy \
--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy \
--skip_print_timers --skip_stat_analysis --process_multi_keys
...
Summary:
-------------------------------
Graph loaded in 0.000000s
All images loaded in 0.000000s
All images detected in 0.000000s
Average detection time: 0.000000s
mAP: 0.22349680978666922
Recall: 0.2550505369422975
--------------------------------
Fast NMS
$ ck benchmark program:object-detection-tflite --env.USE_NMS=fast \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=5000 --env.CK_METRIC_TYPE=COCO \
--record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-accuracy \
--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy \
--skip_print_timers --skip_stat_analysis --process_multi_keys
...
Summary:
-------------------------------
Graph loaded in 0.000000s
All images loaded in 0.000000s
All images detected in 0.000000s
Average detection time: 0.000000s
mAP: 0.21859688835124763
Recall: 0.24801510024502602
--------------------------------
Fast NMS graph with custom model settings
You can reproduce the regular NMS behaviour even with the fast NMS graph by requesting to use custom model settings:
$ ck benchmark program:object-detection-tflite --env.USE_NMS=fast --env.CUSTOM_MODEL_SETTINGS=yes \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=5000 --env.CK_METRIC_TYPE=COCO \
--record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-accuracy \
--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy \
--skip_print_timers --skip_stat_analysis --process_multi_keys
...
Summary:
-------------------------------
Graph loaded in 0.000000s
All images loaded in 0.000000s
All images detected in 0.000000s
Average detection time: 0.000000s
mAP: 0.22349680978666922
Recall: 0.2550505369422975
--------------------------------
For example, you can lower the NMS score threshold for more accurate albeit slower detection:
$ ck benchmark program:object-detection-tflite --env.USE_NMS=fast \
--env.CUSTOM_MODEL_SETTINGS=yes --env.NMS_SCORE_THRESHOLD=0.00001 \
--repetitions=1 --env.CK_BATCH_SIZE=1 --env.CK_BATCH_COUNT=5000 --env.CK_METRIC_TYPE=COCO \
--record --record_repo=local --record_uoa=mlperf-object-detection-ssd-mobilenet-tflite-accuracy \
--tags=mlperf,object-detection,ssd-mobilenet,tflite,accuracy \
--skip_print_timers --skip_stat_analysis --process_multi_keys
See here for more details on custom model settings.
Files
- .cm/desc.json (3 bytes)
- .cm/info.json (338 bytes)
- .cm/meta.json (2689 bytes)
- README.md (13407 bytes)