近几年，设备上的深度学习应用越来越普遍。在应用中部署深度学习模型给开发者带来挑战。开发者们需要选择一个合适的框架，选择性地利用量化压缩技术与模型精度进行权衡，最终将模型部署到设备上。对比测试这些框架，并从中选择是一个繁琐耗时的工作。
MobileAIBech是小米开源的一个端到端的测试工具，用于评测同一模型在不同框架上运行的性能表现，希望测评结果可以提供给开发者一些指导。
每日评测结果请查看最新的CIPipelie页面中的_bechmark_步骤的运行结果。
准备环境MobileAIBech现在支持多种框架(MACE、SNPE、c以及TesorFlowLite)，需要安装以下的依赖：
依赖安装命令验证可用的版本Pytho2.7ADBapt-getistalladroid-tools-adbRequiredbyAdroidru,>=1.0.32AdroidNDKNDKistallatioguideRequiredbyAdroidbuild,r15cBazelbazelistallatioguide0.13.0CMakeapt-getistallcmake>=3.11.3FileLockpipistall-Ifilelock==3.0.0RequiredbyAdroidruPyYamlpipistall-Ipyyaml==3.123.12.0shpipistall-Ish==1.12.141.12.14SNPE(可选)下载并解压1.15.0备注:鉴于SNPE的许可不允许第三方再分发,目前BazelWORKSPACE配置中的链接只能在CIServer中访问。如果想测评SNPE(通过--frameworks指定all或者显式指定了SNPE)，需从官方地址下载并解压，然后修改WORKSPACE文件如下。
#ew_http_archive(# ame = "spe",# build_file = "third_party/spe/spe.BUILD",# sha256 = "b11780e5e7f591e916c69bdface4a1ef75b0c19f7b43c868bd62c0f3747d3fbb",# strip_prefix = "spe-1.15.0",# urls = [# "https://cbj1-fds.api.xiaomi.et/aibech/third_party/spe-1.15.0.zip",# ],#)ew_local_repository( ame = "spe", build_file = "third_party/spe/spe.BUILD", path = "/path/to/spe-1.15.0",)数据结构+-----------------+ +------------------+ +---------------+| Bechmark | | BaseExecutor | <--- | MaceExecutor |+-----------------+ +------------------+ +---------------+| - executor |-------> | - framework || - model_ame | | - rutime | +---------------+| - model_file | | | <--- | SpeExecutor || - iput_ames | +------------------+ +---------------+| - iput_files | | + Iit() || - iput_shapes | | + Prepare() | +---------------+| - output_ames | | + Ru() | <--- | NcExecutor || - output_shapes | | + Fiish() | +---------------++-----------------+ +------------------+ | - Register() | +---------------+| - Ru() | <--- | TfLiteExecutor|+-----------------+ +---------------+如何使用测试所有模型在所有框架上的性能pytho tools/bechmark.py --output_dir=output --frameworks=all \ --rutimes=all --model_ames=all \ --target_abis=armeabi-v7a,arm64-v8a运行时间可能比较长，如果只想测试指定模型和框架，可以添加如下选项：
optiotypedefaultexplaatio--output_dirstroutputBechmarkoutputdirectory.--frameworksstrallFrameworks(MACE/SNPE/NCNN/TFLITE),commaseparatedlistorall.--rutimesstrallRutimes(CPU/GPU/DSP),commaseparatedlistorall.--target_abisstrarmeabi-v7aTargetABIs(armeabi-v7a,arm64-v8a),commaseparatedlist.--model_amesstrallModelames(IceptioV3,MobileNetV1…),commaseparatedlistorall.--ru_itervalit10Ruitervalbetweebechmarks,secods.--um_threadsit4Theumberofthreads.在已有框架中添加新模型评测注册模型
在aibech/bechmark/bechmark_mai.cc中添加:
#ifdef AIBENCH_ENABLE_YOUR_FRAMEWORK std::uique_ptr<aibech::YourFrameworkExecutor> your_framework_executor(ew aibech::YourFrameworkExecutor()); AIBENCH_BENCHMARK(your_framework_executor.get(), MODEL_NAME, FRAMEWORK_NAME, RUNTIME, MODEL_FILE, (std::vector<std::strig>{INPUT_NAME}), (std::vector<std::strig>{INPUT_FILE}), (std::vector<std::vector<it64_t>>{INPUT_SHAPE}), (std::vector<std::strig>{OUTPUT_NAME}), (std::vector<std::vector<it64_t>>{OUTPUT_SHAPE})); #edife.g.
AIBENCH_BENCHMARK(mobileetv1_mace_cpu_executor.get(), MobileNetV1, MACE, CPU, mobileet_v1, (std::vector<std::strig>{"iput"}), (std::vector<std::strig>{"dog.py"}), (std::vector<std::vector<it64_t>>{{1, 224, 224, 3}}), (std::vector<std::strig>{ "MobileetV1/Predictios/Reshape_1"}), (std::vector<std::vector<it64_t>>{{1, 1001}}));在tools/model_list.py中注册模型名称
配置模型文件和输入文件
在tools/model_ad_iput.yml中配置MODEL_FILE和INPUT_FILE。
运行测试
pytho tools/bechmark.py --output_dir=output --frameworks=MACE \ --rutimes=CPU --model_ames=MobileNetV1 \ --target_abis=armeabi-v7a,arm64-v8a查看结果
cat output/report.csv加入新的AI框架定义executor并实现其接口:
class YourFrameworkExecutor : public BaseExecutor { public: YourFrameworkExecutor() : BaseExecutor(FRAMEWORK_NAME, RUNTIME) {} // Iit method should ivoke the iitializig process for your framework // (e.g. Mace eeds to compile OpeCL kerel oce per target). It will be // called oly oce whe creatig framework egie. virtual Status Iit(cost char *model_ame, it um_threads); // Load model ad prepare to ru. It will be called oly oce before // bechmarkig the model. virtual Status Prepare(cost char *model_ame); // Ru the model. It will be called more tha oce. virtual Status Ru(cost std::map<std::strig, BaseTesor> &iputs, std::map<std::strig, BaseTesor> *outputs); // Uload model ad free the memory after bechmarkig. It will be called // oly oce. virtual void Fiish();};在aibech/bechmark/bechmark_mai.cc中包含头文件:
#ifdef AIBENCH_ENABLE_YOUR_FRAMEWORK#iclude "aibech/executors/your_framework/your_framework_executor.h"#edif添加依赖third_party/your_framework,aibech/bechmark/BUILD和WORKSPACE.
测试模型
在已有框架中添加新模型评测
评论