Flutter音视频通信插件sora_flutter_sdk的使用

发布于 1周前 作者 itying888 来自 Flutter

Flutter音视频通信插件sora_flutter_sdk的使用

关于时雨堂的开源软件

在使用之前，请阅读时雨堂的开源软件说明。

概要

Sora Flutter SDK 是一个用于开发基于 WebRTC SFU Sora 的 Flutter 客户端应用程序的库。

文档

文档正在准备中。

支持的平台

  • Windows x86_64
  • macOS arm64
  • iOS arm64
  • iPadOS arm64
  • Android arm64
  • Ubuntu x86_64

计划支持的平台

支持的硬件编码器/解码器

每个平台都支持相应的硬件加速器。

  • NVIDIA VIDEO CODEC SDK (NVENC / NVDEC)
    • Windows / Linux
    • VP9 / H.264
  • Apple macOS / iOS / iPadOS Video Toolbox
    • H.264
  • Google Android HWA
    • VP8 / VP9 / H.264
  • Intel oneVPL (Intel Media SDK 后继者)
    • Windows / Linux
    • VP9 / AV1 / H.264

计划支持的硬件编码器/解码器

  • NVIDIA Jetson Video HWA
    • Linux
    • VP9 / AV1 / H.264

许可证

Sora Flutter SDK 使用 Apache License 2.0 版权许可。

Copyright 2022-2023, Wandbox LLC (Original Author)
Copyright 2022-2023, SUZUKI Tetsuya (Original Author)
Copyright 2022-2023, Shiguredo Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

示例代码

example/lib/main.dart

import 'dart:async';
import 'dart:io';

import 'package:collection/collection.dart';
import 'package:flutter/material.dart';
import 'package:sora_flutter_sdk/sora_flutter_sdk.dart';

import 'environment.dart';

void main() async {
  SoraClientConfig.flutterVersion = Environment.flutterVersion;

  // 获取视频捕获设备列表
  WidgetsFlutterBinding.ensureInitialized();
  final devices = await DeviceList.videoCapturers();
  for (final device in devices) {
    print('设备 => $device');
  }
  final frontCamera = await DeviceList.frontCamera();
  final backCamera = await DeviceList.backCamera();
  print('前置摄像头 => $frontCamera');
  print('后置摄像头 => $backCamera');

  runApp(const MyApp());
}

class MyApp extends StatefulWidget {
  const MyApp({super.key});

  [@override](/user/override)
  State<MyApp> createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  SoraClient? _soraClient;
  var _isConnected = false;
  List<DeviceName> _capturers = List<DeviceName>.empty();
  var _capturerNum = 0;

  // 连接时使用的相机，或当前使用的相机
  DeviceName? _connectDevice;
  var _video = true;
  var _audio = true;
  var _audioCodec = SoraAudioCodecType.opus;

  [@override](/user/override)
  void initState() {
    super.initState();

    // initState 不能是异步的，因此使用闭包来执行异步操作
    () async {
      final capturers = await DeviceList.videoCapturers();
      setState(() {
        _capturers = capturers;
        _connectDevice = _capturers.firstOrNull;
      });
    }();
  }

  [@override](/user/override)
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        body: Builder(builder: _buildMain),
      ),
    );
  }

  Widget _buildMain(BuildContext context) {
    final screenSize = MediaQuery.of(context).size;
    return Center(
      child: Column(
        children: [
          SizedBox(
            height: screenSize.height * 0.7,
            child: VideoGroupView(
              soraClient: _soraClient,
            ),
          ),
          SizedBox(
            height: screenSize.height * 0.3,
            child: Column(
              mainAxisAlignment: MainAxisAlignment.center,
              children: [
                DeviceListDropdownButton(
                    connectDevice: _connectDevice,
                    capturers: _capturers,
                    onChanged: (device) {
                      setState(() {
                        _connectDevice = device;
                        if (_soraClient?.switchingVideoDevice == true) {
                          _setCamera(device);
                        }
                      });
                    }),
                AudioCodecDropdownButton(
                  codec: _audioCodec,
                  onChanged: (codec) {
                    setState(() {
                      _audioCodec = codec ?? SoraAudioCodecType.opus;
                    });
                  },
                ),
                ConnectButtons(
                  onConnect: _connect,
                  onDisconnect: _disconnect,
                  canSwitchCamera: _canSwitchCamera,
                  onSwitchCamera: _switchCamera,
                ),
                const SizedBox(height: 8),
                MuteButtons(
                  enabled: _isConnected,
                  video: _video,
                  audio: _audio,
                  onChanged: (video, audio) {
                    setState(() {
                      _video = video;
                      _audio = audio;
                      _soraClient?.setVideoEnabled(_video);
                      _soraClient?.setAudioEnabled(_audio);
                    });
                  },
                ),
              ],
            ),
          ),
        ],
      ),
    );
  }

  Future<void> _connect() async {
    if (_isConnected) {
      return;
    }

    await _soraClient?.dispose();

    final config = SoraClientConfig(
      signalingUrls: Environment.urlCandidates,
      channelId: Environment.channelId,
      role: SoraRole.sendrecv,
    )
      ..audioCodecType = _audioCodec
      ..metadata = Environment.signalingMetadata
      ..audioStreamingLanguageCode = "ja-JP"
      ..videoDeviceName = _connectDevice;

    final soraClient = await SoraClient.create(config)
      ..onDisconnect = (String errorCode, String message) {
        print("断开连接: 错误码=$errorCode 消息=$message");
        _disconnect();
      }
      ..onSetOffer = (String offer) {
        print("设置Offer: $offer");
      }
      ..onNotify = (String text) {
        print("通知: $text");
      }
      ..onSwitchTrack = (SoraVideoTrack track) {
        setState(() {/* 切换摄像头后的处理 */});
      }
      ..onAddTrack = (SoraVideoTrack track) {
        setState(() {/* 轨道数量变化后的处理 */});
      }
      ..onRemoveTrack = (SoraVideoTrack track) {
        setState(() {/* 轨道数量变化后的处理 */});
      };

    try {
      await soraClient.connect();

      setState(() {
        _soraClient = soraClient;
        _isConnected = true;
      });
    } on Exception catch (e) {
      print('连接失败 => $e');
    }
  }

  Future<void> _disconnect() async {
    if (!_isConnected && _soraClient == null) {
      return;
    }
    print('断开连接');

    try {
      await _soraClient?.dispose();
    } on Exception catch (e) {
      print('释放资源失败 => $e');
    }

    setState(() {
      _soraClient = null;
      _isConnected = false;
      _video = true;
      _audio = true;
    });
  }

  Future<void> _setCamera(DeviceName? name) async {
    if (name == null) {
      return;
    }

    if (_soraClient != null) {
      // 已连接则切换摄像头
      await _doSwitchCamera(name);
    } else {
      // 未连接则设置连接参数
      setState(() {
        _connectDevice = name;
      });
    }
  }

  bool get _canSwitchCamera {
    if ((Platform.isIOS || Platform.isAndroid) && _soraClient != null && !_soraClient!.switchingVideoDevice) {
      return true;
    } else {
      return false;
    }
  }

  Future<void> _switchCamera() async {
    if (_soraClient == null || _capturers.isEmpty) {
      return;
    }

    // 选择下一个要使用的摄像头
    var next = _capturerNum + 1;
    if (next >= _capturers.length) {
      next = 0;
    }
    final name = _capturers[next];
    final result = await _doSwitchCamera(name);
    if (result) {
      setState(() {
        _capturerNum = next;
      });
    }
  }

  Future<bool> _doSwitchCamera(DeviceName name) async {
    print('切换到 => $name');

    // 切换过程中禁用切换按钮
    // 先异步调用 switchVideoDevice 方法再更新界面
    // 切换过程中 _soraClient.switchingVideoDevice 为 true
    final future = _soraClient!.switchVideoDevice(name: name);

    // 更新界面使切换按钮无效
    setState(() {});

    // 等待切换完成并继续后续操作
    final result = await future;
    setState(() {
      if (result) {
        print('切换设备成功 => $name, ${_soraClient!.switchingVideoDevice}');
        _connectDevice = name;
      } else {
        print('切换失败');
      }
    });
    return result;
  }
}

class DeviceListDropdownButton extends StatelessWidget {
  DeviceListDropdownButton({
    super.key,
    required this.connectDevice,
    required this.capturers,
    required this.onChanged,
  });

  final DeviceName? connectDevice;
  final List<DeviceName> capturers;
  final void Function(DeviceName?) onChanged;

  [@override](/user/override)
  Widget build(BuildContext context) => Row(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          const Text('摄像头: '),
          DropdownButton(
            value: connectDevice,
            items: capturers
                .map((DeviceName device) => DropdownMenuItem(
                      child: Text('${device.name}'),
                      value: device,
                    ))
                .toList(),
            // 切换过程中禁用按钮
            onChanged: onChanged,
          ),
        ],
      );
}

class AudioCodecDropdownButton extends StatelessWidget {
  AudioCodecDropdownButton({
    super.key,
    required this.codec,
    required this.onChanged,
  });

  final SoraAudioCodecType codec;
  final void Function(SoraAudioCodecType?) onChanged;

  [@override](/user/override)
  Widget build(BuildContext context) => Row(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          const Text('音频编解码器: '),
          DropdownButton(
            value: codec,
            items: SoraAudioCodecType.values
                .map((codec) => DropdownMenuItem(
                      child: Text(codec.name),
                      value: codec,
                    ))
                .toList(),
            onChanged: onChanged,
          ),
        ],
      );
}

class ConnectButtons extends StatelessWidget {
  ConnectButtons({
    super.key,
    required this.onConnect,
    required this.onDisconnect,
    required this.canSwitchCamera,
    required this.onSwitchCamera,
  });

  final void Function() onConnect;
  final void Function() onDisconnect;
  final bool canSwitchCamera;
  final void Function() onSwitchCamera;

  [@override](/user/override)
  Widget build(BuildContext context) => Row(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          ElevatedButton(
            onPressed: onConnect,
            child: const Text('连接'),
          ),
          const SizedBox(width: 20),
          ElevatedButton(
            onPressed: onDisconnect,
            child: const Text('断开'),
          ),
          const SizedBox(width: 20),
          SwitchCameraButton(
            enabled: canSwitchCamera,
            onPressed: onSwitchCamera,
          ),
        ],
      );
}

class MuteButtons extends StatelessWidget {
  MuteButtons({
    super.key,
    required this.enabled,
    required this.video,
    required this.audio,
    required this.onChanged,
  });

  final bool enabled;
  final bool video;
  final bool audio;
  final void Function(bool video, bool audio) onChanged;

  [@override](/user/override)
  Widget build(BuildContext context) => Row(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          const Text('静音:'),
          const SizedBox(width: 20),
          const Text('视频'),
          Switch(
            value: video,
            onChanged: enabled
                ? (flag) {
                    onChanged(flag, audio);
                  }
                : null,
          ),
          const SizedBox(width: 8),
          const Text('音频'),
          Switch(
            value: audio,
            onChanged: enabled
                ? (flag) {
                    onChanged(video, flag);
                  }
                : null,
          ),
        ],
      );
}

class SwitchCameraButton extends StatelessWidget {
  SwitchCameraButton({
    super.key,
    required this.enabled,
    required this.onPressed,
  });

  final bool enabled;
  final void Function() onPressed;

  [@override](/user/override)
  Widget build(BuildContext context) => ElevatedButton(
        onPressed: enabled ? onPressed : null,
        child: Icon(Icons.flip_camera_ios),
      );
}

class VideoGroupView extends StatelessWidget {
  VideoGroupView({
    super.key,
    required this.soraClient,
  });

  final SoraClient? soraClient;

  [@override](/user/override)
  Widget build(BuildContext context) {
    var renderers = List<SoraRenderer>.empty();
    if (soraClient != null) {
      renderers = soraClient!.tracks
          .map((track) => SoraRenderer(
                width: 320,
                height: 240,
                track: track,
              ))
          .toList();
    }

    return SingleChildScrollView(
      child: Center(
        child: GridView.count(
          shrinkWrap: true,
          physics: const NeverScrollableScrollPhysics(),
          crossAxisCount: 2,
          children: renderers,
        ),
      ),
    );
  }
}

更多关于Flutter音视频通信插件sora_flutter_sdk的使用的实战系列教程也可以访问 https://www.itying.com/category-92-b0.html

1 回复

更多关于Flutter音视频通信插件sora_flutter_sdk的使用的实战系列教程也可以访问 https://www.itying.com/category-92-b0.html


当然，下面是一个关于如何在Flutter项目中使用sora_flutter_sdk插件进行音视频通信的示例代码。这个插件通常用于集成Shinobi Sora的音视频通信功能。

前提条件

  1. 确保你已经在Flutter环境中安装了sora_flutter_sdk插件。
  2. 拥有一个有效的Shinobi Sora服务账号和相应的信令服务器URL。

安装sora_flutter_sdk

首先，在你的pubspec.yaml文件中添加依赖：

dependencies:
  flutter:
    sdk: flutter
  sora_flutter_sdk: ^最新版本号  # 请替换为实际的最新版本号

然后运行flutter pub get来安装依赖。

Flutter项目中的使用示例

1. 导入必要的包

在你的Dart文件中导入sora_flutter_sdk：

import 'package:sora_flutter_sdk/sora_flutter_sdk.dart';

2. 配置Sora客户端

在main.dart中配置Sora客户端：

void main() {
  WidgetsFlutterBinding.ensureInitialized();
  runApp(MyApp());
}

class MyApp extends StatefulWidget {
  @override
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  late SoraClient soraClient;

  @override
  void initState() {
    super.initState();

    // 配置Sora客户端
    soraClient = SoraClient(
      channelId: 'your_channel_id',  // 替换为你的频道ID
      role: SoraRole.sendrecv,       // 发送和接收音视频
      signalingUrl: 'your_signaling_url',  // 替换为你的信令服务器URL
      multiStream: true,             // 是否支持多路流
      simulcast: true,               // 是否启用Simulcast
      audio: SoraAudioConfig(
        bitrate: 64000,
      ),
      video: SoraVideoConfig(
        bitrate: 1500000,
        width: 640,
        height: 480,
        framerate: 30,
      ),
    );

    // 连接Sora服务器
    soraClient.connect().then((_) {
      print('Connected to Sora server');
    }).catchError((error) {
      print('Failed to connect to Sora server: $error');
    });
  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: SoraVideoViewPage(soraClient: soraClient),
    );
  }

  @override
  void dispose() {
    // 断开连接并释放资源
    soraClient.disconnect();
    super.dispose();
  }
}

3. 创建视频视图页面

创建一个新的页面来显示本地和远程的视频流：

import 'package:flutter/material.dart';
import 'package:sora_flutter_sdk/sora_flutter_sdk.dart';

class SoraVideoViewPage extends StatefulWidget {
  final SoraClient soraClient;

  SoraVideoViewPage({required this.soraClient});

  @override
  _SoraVideoViewPageState createState() => _SoraVideoViewPageState();
}

class _SoraVideoViewPageState extends State<SoraVideoViewPage> {
  late SoraLocalVideoView localVideoView;
  late SoraRemoteVideoView remoteVideoView;

  @override
  void initState() {
    super.initState();

    // 初始化本地视频视图
    localVideoView = SoraLocalVideoView(
      soraClient: widget.soraClient,
    );

    // 初始化远程视频视图
    remoteVideoView = SoraRemoteVideoView(
      soraClient: widget.soraClient,
      remoteStreamId: 'remote_stream_id',  // 这里可以动态设置远程流的ID
    );

    // 开始发送本地视频流
    widget.soraClient.publish().then((_) {
      print('Started publishing local video stream');
    }).catchError((error) {
      print('Failed to publish local video stream: $error');
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Sora Video Communication'),
      ),
      body: Column(
        children: <Widget>[
          Expanded(child: localVideoView),
          Expanded(child: remoteVideoView),
        ],
      ),
    );
  }

  @override
  void dispose() {
    // 释放资源
    localVideoView.dispose();
    remoteVideoView.dispose();
    super.dispose();
  }
}

注意事项

  1. 权限管理：确保在AndroidManifest.xml和Info.plist中添加了必要的音视频和摄像头权限。
  2. 错误处理：在实际应用中，应添加更多的错误处理逻辑，以确保在各种情况下都能正确处理。
  3. UI设计：上述示例仅用于演示基本的音视频通信功能，实际项目中应根据需求设计更友好的用户界面。

这个示例展示了如何使用sora_flutter_sdk插件在Flutter应用中实现音视频通信。你可以根据具体需求进一步扩展和优化代码。

回到顶部