Flutter文档分析插件document_analysis的使用

Flutter文档分析插件document_analysis的使用

简介

document_analysis 是一个用于文档分析处理的集合。

开始使用

在您的 Dart（或 Flutter）项目中的 pubspec.yaml 文件中添加依赖：

dependencies:
  ...
  document_analysis: ^0.1.2

向量距离测量

由于这是基于文档的分析，距离测量必须在 0-1 范围内（归一化，不同于欧几里得距离）。当前可用的距离测量方法包括：

• Jaccard
• Cosine

调用：jaccardDistance(vector1, vector2)

输入向量是 List<double>

示例代码：

// 定义两个向量
List<double> vector1 = [0, 1, 1.5, 3, 2, 0.5];
List<double> vector2 = [1, 3, 3.5, 4, 0.5, 0];

// 计算Jaccard距离和Cosine距离
print("Jaccard: ${jaccardDistance(vector1, vector2)}"); // 输出约0.333...
print("Cosine: ${cosineDistance(vector1, vector2)}");   // 输出约0.156...

文档相似度

当前可用的文档相似度函数基于以下方法：

• 词频
• 词频-逆文档频率（TF-IDF）
• 混合 TF-IDF

调用：wordFrequencySimilarity(doc1, doc2, distanceFunction: jaccardDistance)

• doc1, doc2: 输入文档（字符串）
• distanceFunction: 向量距离测量 (vector1, vector2)=>double

示例代码：

// 定义两个文档
String doc1 = "Report: Xiaomi topples Fitbit and Apple as world's largest wearables vendor";
String doc2 = "Xiaomi topples Fitbit and Apple as world's largest wearables vendor: Strategy Analytics";

// 使用Jaccard和Cosine计算文档相似度
print("${wordFrequencySimilarity(doc1, doc2, distanceFunction: jaccardDistance)}"); // 输出约0.769...
print("${wordFrequencySimilarity(doc1, doc2, distanceFunction: cosineDistance)}");  // 输出约0.870...

矩阵创建

从一组文档创建词向量矩阵，当前可用的方法包括：

• 词频
• TF-IDF
• 混合 TF-IDF

调用：wordFrequencyMatrix([doc1, doc2])

• [...]: 所有文档，List<String>

示例代码：

// 定义两个文档
String doc1 = "Report: Xiaomi topples Fitbit and Apple as world's largest wearables vendor";
String doc2 = "Xiaomi topples Fitbit and Apple as world's largest wearables vendor: Strategy Analytics";

// 创建词频矩阵
print(wordFrequencyMatrix([doc1, doc2]));
// 输出矩阵[[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0], [0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]]

文档分词器

将文档（字符串）分成多个指标：

调用：documentTokenizer(List<String> documentList, {minLen = 1, String Function(String) stemmer, List<String> stopwords})

• documentList: 所有文档在一个列表中
• minLen: 在分词过程中考虑的单词出现的最小次数
• stemmer: 词干提取函数
• stopwords: 应该在文档分析中忽略的一组常见词

示例代码：

// 定义三个文档
String doc1 = "Report: Xiaomi topples Fitbit and Apple as world's largest wearables vendor";
String doc2 = "Xiaomi topples Fitbit and Apple as world's largest wearables vendor: Strategy Analytics";
String doc3 = "Apple releases new iPhone model";

// 对文档进行分词
var tokenizationOutput = documentTokenizer([doc1, doc2, doc3]);

// 输出TokenizationOutput对象的内容
print(tokenizationOutput.bagOfWords);
print(tokenizationOutput.wordInDocumentOccurrence);
print(tokenizationOutput.documentBOW);
print(tokenizationOutput.documentTotalWord);
print(tokenizationOutput.numberOfDistintWords);
print(tokenizationOutput.totalNumberOfWords);

输出 TokenizationOutput 对象：

class TokenizationOutput{
  ///每个单词在所有文档中的计数
  Map<String, double> bagOfWords = {};
  ///某个单词在所有文档中出现的频率（每篇文档最多一次）
  Map<String, double> wordInDocumentOccurrence = {};
  ///每个文档的“词袋”列表
  List<Map<String, double>> documentBOW = [];
  ///每篇文档中的总单词数
  List<int> documentTotalWord = [];
  ///所有文档中的不同单词总数
  int numberOfDistintWords = 0;
  ///所有文档中的单词总数
  int totalNumberOfWords = 0;
}

Flutter 示例

以下是完整的 Flutter 示例，展示如何在 Flutter 应用程序中使用 document_analysis 插件来检查两个文档/字符串之间的相似性。

import 'package:flutter/material.dart';
import 'package:document_analysis/document_analysis.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  [@override](/user/override)
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Document Analysis'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key key, this.title}) : super(key: key);

  final String title;

  [@override](/user/override)
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  String doc1 = "Report: Xiaomi topples Fitbit and Apple as world's largest wearables vendor";
  String doc2 = "Xiaomi topples Fitbit and Apple as world's largest wearables vendor: Strategy Analytics";

  double jaccardSim;
  double cosineSim;

  [@override](/user/override)
  void initState() {
    super.initState();

    // 计算相似度
    jaccardSim = wordFrequencySimilarity(doc1, doc2, distanceFunction: jaccardDistance);
    cosineSim = wordFrequencySimilarity(doc1, doc2, distanceFunction: cosineDistance);
  }

  [@override](/user/override)
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            Text(
              'Jaccard Similarity:',
            ),
            Text(
              '$jaccardSim',
              style: Theme.of(context).textTheme.headline4,
            ),
            SizedBox(height: 20),
            Text(
              'Cosine Similarity:',
            ),
            Text(
              '$cosineSim',
              style: Theme.of(context).textTheme.headline4,
            ),
          ],
        ),
      ),
    );
  }
}