Rust爱尔兰语言模型库lingua-irish-language-model使用指南

简介

lingua-irish-language-model是一个专门为爱尔兰语(Gaeilge)设计的高性能自然语言处理库，基于Rust语言开发。该库提供了爱尔兰语文本分析、语言检测、分词、词性标注等功能，特别适合需要处理爱尔兰语文本的应用程序。

主要特性

• 高性能爱尔兰语文本处理
• 准确的语言检测
• 爱尔兰语分词功能
• 词性标注支持
• 轻量级且内存高效

安装

在Cargo.toml中添加依赖：

[dependencies]
lingua-irish-language-model = "0.1"

基本使用方法

1. 语言检测

use lingua_irish_language_model::LanguageDetector;

fn main() {
    let detector = LanguageDetector::new();
    let text = "Tá an lá go hálainn inniu";
    let result = detector.detect_language(text);
    
    println!("检测到的语言: {:?}", result);
    // 输出: 检测到的语言: Irish
}

2. 文本分词

use lingua_irish_language_model::IrishTokenizer;

fn main() {
    let tokenizer = IrishTokenizer::new();
    let text = "Conas atá tú inniu? Tá mé go maith!";
    let tokens = tokenizer.tokenize(text);
    
    println!("分词结果: {:?}", tokens);
    // 输出: ["Conas", "atá", "tú", "inniu", "?", "Tá", "mé", "go", "maith", "!"]
}

3. 词性标注

use lingua_irish_language_model::IrishTagger;

fn main() {
    let tagger = IrishTagger::new();
    let text = "Tá an ghrian ag soilsiú";
    let tagged = tagger.tag(text);
    
    println!("词性标注结果:");
    for (word, pos) in tagged {
        println!("{}: {}", word, pos);
    }
    // 示例输出:
    // Tá: VB
    // an: DET
    // ghrian: N
    // ag: ADP
    // soilsiú: VN
}

高级用法

自定义词典

use lingua_irish_language_model::{IrishTagger, Dictionary};

fn main() {
    let mut dictionary = Dictionary::default();
    // 添加自定义词汇
    dictionary.add_word("ríomhaire", "N"); // 计算机
    dictionary.add_word("clárú", "V");    // 注册
    
    let tagger = IrishTagger::with_dictionary(dictionary);
    let text = "Tá mé ag clárú an ríomhaire nua";
    let tagged = tagger.tag(text);
    
    println!("{:?}", tagged);
}

批量处理文本

use lingua_irish_language_model::{IrishTokenizer, IrishTagger};

fn process_irish_text(texts: Vec<&str>) -> Vec<Vec<(String, String)>> {
    let tokenizer = IrishTokenizer::new();
    let tagger = IrishTagger::new();
    
    texts.into_iter()
        .map(|text| {
            let tokens = tokenizer.tokenize(text);
            tagger.tag_tokens(&tokens)
        })
        .collect()
}

fn main() {
    let texts = vec![
        "Dia dhuit",
        "Cad is ainm duit?",
        "Go raibh maith agat"
    ];
    
    let results = process_irish_text(texts);
    for result in results {
        println!("{:?}", result);
    }
}

完整示例代码

下面是一个综合使用该库所有主要功能的完整示例：

use lingua_irish_language_model::{
    LanguageDetector, 
    IrishTokenizer,
    IrishTagger,
    Dictionary
};
use std::sync::Arc;

fn main() {
    // 1. 语言检测示例
    let detector = LanguageDetector::new();
    let sample_text = "Tá an lá go hálainn inniu";
    let detection_result = detector.detect_language(sample_text);
    println!("语言检测结果: {:?}", detection_result);

    // 2. 创建线程安全的分词器和标注器
    let tokenizer = Arc::new(IrishTokenizer::new());
    let tagger = Arc::new(IrishTagger::new());

    // 3. 文本处理函数
    let process_text = |text: &str| {
        let tokens = tokenizer.tokenize(text);
        let tagged = tagger.tag_tokens(&tokens);
        (tokens, tagged)
    };

    // 4. 处理多个文本
    let texts = vec![
        "Dia dhuit, a chara",
        "Conas atá tú?",
        "Tá mé go maith, go raibh maith agat"
    ];

    for text in texts {
        println!("\n处理文本: {}", text);
        let (tokens, tagged) = process_text(text);
        println!("分词结果: {:?}", tokens);
        println!("词性标注:");
        for (word, pos) in tagged {
            println!("  {}: {}", word, pos);
        }
    }

    // 5. 使用自定义词典
    let mut custom_dict = Dictionary::default();
    custom_dict.add_word("leabhar", "N"); // 书
    custom_dict.add_word("léamh", "V");   // 读
    
    let custom_tagger = IrishTagger::with_dictionary(custom_dict);
    let custom_text = "Tá mé ag léamh leabhar";
    let custom_tagged = custom_tagger.tag(custom_text);
    
    println!("\n自定义词典处理结果:");
    for (word, pos) in custom_tagged {
        println!("  {}: {}", word, pos);
    }
}

性能优化建议

1. 对于大量文本处理，考虑使用IrishTokenizer和IrishTagger的Arc包装版本以实现线程安全
2. 对于已知纯爱尔兰语文本，可以跳过语言检测步骤
3. 重用Tokenizer和Tagger实例以避免重复初始化开销

注意事项

• 该库主要针对现代标准爱尔兰语优化
• 对于方言变体可能需要额外处理
• 某些古老或诗歌形式的爱尔兰语可能识别不准确

总结

lingua-irish-language-model为Rust开发者提供了处理爱尔兰语文本的强大工具，特别适合需要高性能爱尔兰语NLP处理的应用程序。通过简单的API，开发者可以快速集成爱尔兰语处理功能到自己的项目中。