Flutter媒体处理插件flutter_mediapipe的使用

flutter_mediapipe 是一个用于在Flutter中集成Mediapipe功能的插件，主要用于Android平台。以下是详细的步骤和示例代码，帮助您快速上手。

设备支持

• [OK] Android
• [NG] iOS（尚未实现）

在Android上使用flutter_mediapipe

Docker环境搭建

首先，克隆Mediapipe仓库并构建Docker镜像：

git clone https://github.com/google/mediapipe.git
cd mediapipe
docker build --tag=mediapipe .
docker run -it --name mediapipe mediapipe:latest

再次进入Docker容器：

docker start [your container ID]
docker exec -i -t mediapipe bash

设置和准备

在容器内运行以下命令进行设置：

apt update
apt install vim
apt install zip
bash ./setup_android_sdk_and_ndk.sh

创建必要的目录和文件：

mkdir -p mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe
vi mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/BUILD

编辑 BUILD 文件内容如下：

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
    name = "flutter_mediapipe",
    calculators = ["//mediapipe/graphs/face_mesh:mobile_calculators"],
)

构建

构建jniLibs和binary graph：

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:BUILD --linkopt="-s"
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:flutter_mediapipe --linkopt="-s"

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:BUILD
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:facemeshgpu

创建项目结构

创建必要的目录：

mkdir flutter_mediapipe/
mkdir flutter_mediapipe/android
mkdir flutter_mediapipe/android/libs
mkdir flutter_mediapipe/android/src
mkdir flutter_mediapipe/android/src/main
mkdir flutter_mediapipe/android/src/main/assets
mkdir flutter_mediapipe/android/src/main/jniLibs
mkdir flutter_mediapipe/protos

复制生成的文件到项目结构中：

cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/libflutter_mediapipe_android_lib.jar flutter_mediapipe/android/libs

cp mediapipe/modules/face_detection/face_detection_front.tflite flutter_mediapipe/android/src/main/assets
cp mediapipe/modules/face_landmark/face_landmark.tflite flutter_mediapipe/android/src/main/assets
cp bazel-out/k8-opt/bin/mediapipe/graphs/face_mesh/face_mesh_mobile_gpu.binarypb flutter_mediapipe/android/src/main/assets

mkdir work
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/flutter_mediapipe.aar work/aar.zip
cd work/
unzip aar.zip
cd ..
cp -r work/jni/* flutter_mediapipe/android/src/main/jniLibs/

cp mediapipe/framework/formats/landmark.proto flutter_mediapipe/protos/

压缩项目文件：

zip -r flutter_mediapipe.zip flutter_mediapipe

从容器中复制生成的文件：

docker cp 3730fdc4d319:/mediapipe/flutter_mediapipe.zip .
unzip flutter_mediapipe.zip

然后将 flutter_mediapipe 目录复制到您的Flutter插件项目中。

构建APK

flutter build apk --split-per-abi

示例代码

以下是一个简单的Flutter应用示例，展示如何使用 flutter_mediapipe 插件：

import 'package:flutter/material.dart';
import 'package:flutter_mediapipe/flutter_mediapipe.dart';
import 'package:flutter_mediapipe/gen/landmark.pb.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatefulWidget {
  [@override](/user/override)
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  [@override](/user/override)
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(
          title: const Text('Flutter Mediapipe'),
        ),
        body: Container(
            child: NativeView(
                onViewCreated: (FlutterMediapipe c) => setState(() {
                      c.landMarksStream.listen(_onLandMarkStream);
                      c.platformVersion.then((content) => print(content));
                    }))),
      ),
    );
  }

  void _onLandMarkStream(NormalizedLandmarkList landmarkList) {
    landmarkList.landmark.asMap().forEach((int i, NormalizedLandmark value) {
      print('Index: $i \n' + '$value');
    });
  }
}