近几年，设备上的深度学习应用越来越普遍。在应用中部署深度学习模型给开发者带来挑战。开发者们需要选择一个合适的框架，选择性地利用量化压缩技术与模型精度进行权衡，最终将模型部署到设备上。对比测试这些框架，并从中选择是一个繁琐耗时的工作。
MobileAIBench是小米开源的一个端到端的测试工具，用于评测同一模型在不同框架上运行的性能表现，希望测评结果可以提供给开发者一些指导。
每日评测结果请查看最新的CIPipeline页面中的_benchmark_步骤的运行结果。
准备环境MobileAIBench现在支持多种框架(MACE、SNPE、ncnn以及TensorFlowLite)，需要安装以下的依赖：
依赖安装命令验证可用的版本Python2.7ADBapt-getinstallandroid-tools-adbRequiredbyAndroidrun,>=1.0.32AndroidNDKNDKinstallationguideRequiredbyAndroidbuild,r15cBazelbazelinstallationguide0.13.0CMakeapt-getinstallcmake>=3.11.3FileLockpipinstall-Ifilelock==3.0.0RequiredbyAndroidrunPyYamlpipinstall-Ipyyaml==3.123.12.0shpipinstall-Ish==1.12.141.12.14SNPE(可选)下载并解压1.15.0备注:鉴于SNPE的许可不允许第三方再分发,目前BazelWORKSPACE配置中的链接只能在CIServer中访问。如果想测评SNPE(通过--frameworks指定all或者显式指定了SNPE)，需从官方地址下载并解压，然后修改WORKSPACE文件如下。
#new_http_archive(# name = "snpe",# build_file = "third_party/snpe/snpe.BUILD",# sha256 = "b11780e5e7f591e916c69bdface4a1ef75b0c19f7b43c868bd62c0f3747d3fbb",# strip_prefix = "snpe-1.15.0",# urls = [# "https://cnbj1-fds.api.xiaomi.net/aibench/third_party/snpe-1.15.0.zip",# ],#)new_local_repository( name = "snpe", build_file = "third_party/snpe/snpe.BUILD", path = "/path/to/snpe-1.15.0",)数据结构+-----------------+ +------------------+ +---------------+| Benchmark | | BaseExecutor | <--- | MaceExecutor |+-----------------+ +------------------+ +---------------+| - executor |-------> | - framework || - model_name | | - runtime | +---------------+| - model_file | | | <--- | SnpeExecutor || - input_names | +------------------+ +---------------+| - input_files | | + Init() || - input_shapes | | + Prepare() | +---------------+| - output_names | | + Run() | <--- | NcnnExecutor || - output_shapes | | + Finish() | +---------------++-----------------+ +------------------+ | - Register() | +---------------+| - Run() | <--- | TfLiteExecutor|+-----------------+ +---------------+如何使用测试所有模型在所有框架上的性能python tools/benchmark.py --output_dir=output --frameworks=all \ --runtimes=all --model_names=all \ --target_abis=armeabi-v7a,arm64-v8a运行时间可能比较长，如果只想测试指定模型和框架，可以添加如下选项：
optiontypedefaultexplanation--output_dirstroutputBenchmarkoutputdirectory.--frameworksstrallFrameworks(MACE/SNPE/NCNN/TFLITE),commaseparatedlistorall.--runtimesstrallRuntimes(CPU/GPU/DSP),commaseparatedlistorall.--target_abisstrarmeabi-v7aTargetABIs(armeabi-v7a,arm64-v8a),commaseparatedlist.--model_namesstrallModelnames(InceptionV3,MobileNetV1…),commaseparatedlistorall.--run_intervalint10Runintervalbetweenbenchmarks,seconds.--num_threadsint4Thenumberofthreads.在已有框架中添加新模型评测注册模型
在aibench/benchmark/benchmark_main.cc中添加:
#ifdef AIBENCH_ENABLE_YOUR_FRAMEWORK std::unique_ptr<aibench::YourFrameworkExecutor> your_framework_executor(new aibench::YourFrameworkExecutor()); AIBENCH_BENCHMARK(your_framework_executor.get(), MODEL_NAME, FRAMEWORK_NAME, RUNTIME, MODEL_FILE, (std::vector<std::string>{INPUT_NAME}), (std::vector<std::string>{INPUT_FILE}), (std::vector<std::vector<int64_t>>{INPUT_SHAPE}), (std::vector<std::string>{OUTPUT_NAME}), (std::vector<std::vector<int64_t>>{OUTPUT_SHAPE})); #endife.g.
AIBENCH_BENCHMARK(mobilenetv1_mace_cpu_executor.get(), MobileNetV1, MACE, CPU, mobilenet_v1, (std::vector<std::string>{"input"}), (std::vector<std::string>{"dog.npy"}), (std::vector<std::vector<int64_t>>{{1, 224, 224, 3}}), (std::vector<std::string>{ "MobilenetV1/Predictions/Reshape_1"}), (std::vector<std::vector<int64_t>>{{1, 1001}}));在tools/model_list.py中注册模型名称
配置模型文件和输入文件
在tools/model_and_input.yml中配置MODEL_FILE和INPUT_FILE。
运行测试
python tools/benchmark.py --output_dir=output --frameworks=MACE \ --runtimes=CPU --model_names=MobileNetV1 \ --target_abis=armeabi-v7a,arm64-v8a查看结果
cat output/report.csv加入新的AI框架定义executor并实现其接口:
class YourFrameworkExecutor : public BaseExecutor { public: YourFrameworkExecutor() : BaseExecutor(FRAMEWORK_NAME, RUNTIME) {} // Init method should invoke the initializing process for your framework // (e.g. Mace needs to compile OpenCL kernel once per target). It will be // called only once when creating framework engine. virtual Status Init(const char *model_name, int num_threads); // Load model and prepare to run. It will be called only once before // benchmarking the model. virtual Status Prepare(const char *model_name); // Run the model. It will be called more than once. virtual Status Run(const std::map<std::string, BaseTensor> &inputs, std::map<std::string, BaseTensor> *outputs); // Unload model and free the memory after benchmarking. It will be called // only once. virtual void Finish();};在aibench/benchmark/benchmark_main.cc中包含头文件:
#ifdef AIBENCH_ENABLE_YOUR_FRAMEWORK#include "aibench/executors/your_framework/your_framework_executor.h"#endif添加依赖third_party/your_framework,aibench/benchmark/BUILD和WORKSPACE.
测试模型
在已有框架中添加新模型评测
评论