Flutter目标检测插件ultralytics_yolo的使用

简介

ultralytics_yolo 是一个Flutter插件，用于将Ultralytics YOLO计算机视觉模型集成到移动应用中。该插件支持Android和iOS平台，并提供对象检测和图像分类的API。

特性

在进一步操作或报告新问题之前，请确保您已彻底阅读本文档。

使用方法

先决条件

导出Ultralytics YOLO模型

在使用Ultralytics YOLO之前，必须导出所需的模型。导出的模型文件格式为.tflite和.mlmodel，然后将其包含在您的应用中。使用Ultralytics YOLO CLI导出模型。

Android

yolo export format=tflite model=yolov8n imgsz=320 int8

iOS

yolo export format=mlmodel model=yolov8n imgsz=[320, 192] half nms

安装

导出模型后，您会得到.tflite和.mlmodel文件。将这些文件包含在您的应用的assets文件夹中。

权限

确保您有访问相机和存储的必要权限。

Android

在AndroidManifest.xml中添加以下权限：

<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

iOS

在Info.plist中添加以下权限：

<key>NSCameraUsageDescription</key>
<string>Camera permission is required for object detection.</string>
<key>NSPhotoLibraryUsageDescription</key>
<string>Storage permission is required for object detection.</string>

在Podfile中添加以下配置：

post_install do |installer|
  installer.pods_project.targets.each do |target|
    flutter_additional_ios_build_settings(target)

    # Start of the permission_handler configuration
    target.build_configurations.each do |config|
      config.build_settings['GCC_PREPROCESSOR_DEFINITIONS'] ||= [
        '$(inherited)',
        
        ## dart: PermissionGroup.camera
        'PERMISSION_CAMERA=1',

        ## dart: PermissionGroup.photos
        'PERMISSION_PHOTOS=1',
      ]
    end 
    # End of the permission_handler configuration
  end
end

使用示例

创建预测器对象

使用LocalYoloModel类创建预测器对象。此类需要以下参数：

final model = LocalYoloModel(
  id: id,
  task: Task.detect /* 或 Task.classify */,
  format: Format.tflite /* 或 Format.coreml*/,
  modelPath: modelPath,
  metadataPath: metadataPath,
);

对象检测器

final objectDetector = ObjectDetector(model: model);
await objectDetector.loadModel();

图像分类器

final imageClassifier = ImageClassifier(model: model);
await imageClassifier.loadModel();

相机预览

使用UltralyticsYoloCameraPreview小部件显示相机预览和预测结果。

final _controller = UltralyticsYoloCameraController();

UltralyticsYoloCameraPreview(
  predictor: predictor, // 您的预测模型数据
  controller: _controller, // Ultralytics相机控制器
  loadingPlaceholder: Center(
    child: Wrap(
      direction: Axis.vertical,
      crossAxisAlignment: WrapCrossAlignment.center,
      children: [
        const CircularProgressIndicator(
          color: Colors.white,
          strokeWidth: 2,
        ),
        const SizedBox(height: 20),
        Text(
          'Loading model...',
          style: theme.typography.base.copyWith(
            color: Colors.white,
            fontSize: 14,
          ),
        ),
      ],
    ),
  ),
)

图像检测与分类

使用detect或classify方法获取图像预测结果。

objectDetector.detect(imagePath: imagePath);
// 或
imageClassifier.classify(imagePath: imagePath);

示例代码

以下是完整的示例代码，展示了如何在Flutter应用中使用ultralytics_yolo插件进行对象检测。

import 'dart:io' as io;
import 'package:flutter/material.dart';
import 'package:path/path.dart';
import 'package:path_provider/path_provider.dart';
import 'package:flutter/services.dart';
import 'package:permission_handler/permission_handler.dart';
import 'package:ultralytics_yolo/ultralytics_yolo.dart';
import 'package:ultralytics_yolo/yolo_model.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatefulWidget {
  const MyApp({super.key});

  @override
  State<MyApp> createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  final controller = UltralyticsYoloCameraController();

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: FutureBuilder<bool>(
          future: _checkPermissions(),
          builder: (context, snapshot) {
            final allPermissionsGranted = snapshot.data ?? false;

            return !allPermissionsGranted
                ? Container()
                : FutureBuilder<ObjectDetector>(
                    future: _initObjectDetectorWithLocalModel(),
                    builder: (context, snapshot) {
                      final predictor = snapshot.data;

                      return predictor == null
                          ? Container()
                          : Stack(
                              children: [
                                UltralyticsYoloCameraPreview(
                                  controller: controller,
                                  predictor: predictor,
                                  onCameraCreated: () {
                                    predictor.loadModel(useGpu: true);
                                  },
                                ),
                                StreamBuilder<double?>(
                                  stream: predictor.inferenceTime,
                                  builder: (context, snapshot) {
                                    final inferenceTime = snapshot.data;

                                    return StreamBuilder<double?>(
                                      stream: predictor.fpsRate,
                                      builder: (context, snapshot) {
                                        final fpsRate = snapshot.data;

                                        return Times(
                                          inferenceTime: inferenceTime,
                                          fpsRate: fpsRate,
                                        );
                                      },
                                    );
                                  },
                                ),
                              ],
                            );
                    },
                  );
          },
        ),
        floatingActionButton: FloatingActionButton(
          child: const Icon(Icons.abc),
          onPressed: () {
            controller.toggleLensDirection();
          },
        ),
      ),
    );
  }

  Future<ObjectDetector> _initObjectDetectorWithLocalModel() async {
    final modelPath = await _copy('assets/yolov8n_int8.tflite');
    final metadataPath = await _copy('assets/metadata.yaml');
    final model = LocalYoloModel(
      id: '',
      task: Task.detect,
      format: Format.tflite,
      modelPath: modelPath,
      metadataPath: metadataPath,
    );

    return ObjectDetector(model: model);
  }

  Future<ImageClassifier> _initImageClassifierWithLocalModel() async {
    final modelPath = await _copy('assets/yolov8n-cls.mlmodel');
    final model = LocalYoloModel(
      id: '',
      task: Task.classify,
      format: Format.coreml,
      modelPath: modelPath,
    );

    return ImageClassifier(model: model);
  }

  Future<String> _copy(String assetPath) async {
    final path = '${(await getApplicationSupportDirectory()).path}/$assetPath';
    await io.Directory(dirname(path)).create(recursive: true);
    final file = io.File(path);
    if (!await file.exists()) {
      final byteData = await rootBundle.load(assetPath);
      await file.writeAsBytes(byteData.buffer
          .asUint8List(byteData.offsetInBytes, byteData.lengthInBytes));
    }
    return file.path;
  }

  Future<bool> _checkPermissions() async {
    List<Permission> permissions = [];

    var cameraStatus = await Permission.camera.status;
    if (!cameraStatus.isGranted) permissions.add(Permission.camera);

    var storageStatus = await Permission.storage.status;
    if (!storageStatus.isGranted) permissions.add(Permission.storage);

    if (permissions.isEmpty) {
      return true;
    } else {
      try {
        Map<Permission, PermissionStatus> statuses =
            await permissions.request();
        return statuses[Permission.camera] == PermissionStatus.granted &&
            statuses[Permission.storage] == PermissionStatus.granted;
      } on Exception catch (_) {
        return false;
      }
    }
  }
}

class Times extends StatelessWidget {
  const Times({
    super.key,
    required this.inferenceTime,
    required this.fpsRate,
  });

  final double? inferenceTime;
  final double? fpsRate;

  @override
  Widget build(BuildContext context) {
    return SafeArea(
      child: Align(
        alignment: Alignment.bottomCenter,
        child: Container(
            margin: const EdgeInsets.all(20),
            padding: const EdgeInsets.all(20),
            decoration: const BoxDecoration(
              borderRadius: BorderRadius.all(Radius.circular(10)),
              color: Colors.black54,
            ),
            child: Text(
              '${(inferenceTime ?? 0).toStringAsFixed(1)} ms  -  ${(fpsRate ?? 0).toStringAsFixed(1)} FPS',
              style: const TextStyle(color: Colors.white70),
            )),
      ),
    );
  }
}

通过上述步骤，您可以成功地在Flutter应用中集成Ultralytics YOLO模型进行对象检测和图像分类。希望这个指南对您有所帮助！