Rust停用词过滤库stop-words使用指南

stop-words是一个用于Rust的停用词过滤库，可以帮助你高效地从自然语言文本中移除常见无意义词汇（停用词），这在文本处理、搜索和自然语言处理任务中非常有用。

安装

在Cargo.toml中添加依赖：

[dependencies]
stop-words = "0.7"

基本用法

1. 使用内置停用词列表

use stop_words::{get, LANGUAGE};

fn main() {
    // 获取英语停用词列表
    let stop_words = get(LANGUAGE::English);
    
    let text = "This is a sample sentence that contains some stop words";
    let words: Vec<&str> = text.split_whitespace().collect();
    
    // 过滤停用词
    let filtered: Vec<&str> = words
        .into_iter()
        .filter(|word| !stop_words.contains(&word.to_lowercase()))
        .collect();
    
    println!("Filtered: {:?}", filtered);
    // 输出: Filtered: ["sample", "sentence", "contains", "stop", "words"]
}

2. 支持多种语言

use stop_words::{get, LANGUAGE};

fn main() {
    // 获取法语停用词列表
    let french_stop_words = get(LANGUAGE::French);
    
    let text = "Ceci est un exemple de phrase en français";
    let words: Vec<&str> = text.split_whitespace().collect();
    
    let filtered: Vec<&str> = words
        .into_iter()
        .filter(|word| !french_stop_words.contains(&word.to_lowercase()))
        .collect();
    
    println!("Filtered: {:?}", filtered);
}

高级用法

1. 自定义停用词列表

use stop_words::StopWords;

fn main() {
    // 创建自定义停用词集合
    let mut custom_stop_words = StopWords::new();
    custom_stop_words.insert("custom".to_string());
    custom_stop_words.insert("words".to_string());
    
    let text = "This is a text with custom words to be filtered";
    let words: Vec<&str> = text.split_whitespace().collect();
    
    let filtered: Vec<&str> = words
        .into_iter()
        .filter(|word| !custom_stop_words.contains(&word.to_lowercase()))
        .collect();
    
    println!("Filtered: {:?}", filtered);
}

2. 处理标点符号和大小写

use stop_words::{get, LANGUAGE};

fn main() {
    let stop_words = get(LANGUAGE::English);
    let text = "The quick, brown fox jumps over the lazy dog. It's amazing!";
    
    // 移除标点并转为小写后过滤
    let filtered: Vec<String> = text
        .split_whitespace()
        .map(|word| word.trim_matches(|c: char| !c.is_alphabetic()).to_lowercase())
        .filter(|word| !word.is_empty() && !stop_words.contains(word))
        .collect();
    
    println!("Filtered: {:?}", filtered);
    // 输出: Filtered: ["quick", "brown", "fox", "jumps", "lazy", "dog", "amazing"]
}

性能提示

1. 对于重复处理，建议将停用词集合缓存起来而不是每次都重新创建
2. 对于大量文本处理，考虑使用并行处理（如Rayon库）

use stop_words::{get, LANGUAGE};
use rayon::prelude::*;

fn main() {
    let stop_words = get(LANGUAGE::English);
    let texts = vec![
        "This is the first text",
        "And here is another one",
        "The more the merrier"
    ];
    
    let filtered: Vec<Vec<&str>> = texts
        .par_iter()
        .map(|text| {
            text.split_whitespace()
                .filter(|word| !stop_words.contains(&word.to_lowercase()))
                .collect()
        })
        .collect();
    
    println!("Filtered: {:?}", filtered);
}

完整示例

以下是一个结合多个特性的完整示例：

use stop_words::{get, LANGUAGE, StopWords};
use rayon::prelude::*;

fn main() {
    // 1. 使用内置英语停用词
    let english_stop_words = get(LANGUAGE::English);
    
    // 2. 创建自定义停用词
    let mut custom_stop_words = StopWords::new();
    custom_stop_words.insert("special".to_string());
    custom_stop_words.insert("example".to_string());
    
    // 合并停用词
    let all_stop_words: StopWords = english_stop_words.union(&custom_stop_words).cloned().collect();
    
    // 要处理的文本
    let documents = vec![
        "This is a special example document with stop words.",
        "Another example to demonstrate parallel processing.",
        "The quick brown fox jumps over the lazy dog."
    ];
    
    // 并行处理
    let results: Vec<Vec<String>> = documents
        .par_iter()
        .map(|doc| {
            doc.split_whitespace()
                .map(|word| word.trim_matches(|c: char| !c.is_alphabetic()).to_lowercase())
                .filter(|word| !word.is_empty() && !all_stop_words.contains(word))
                .collect()
        })
        .collect();
    
    // 输出结果
    for (i, result) in results.iter().enumerate() {
        println!("Document {} filtered: {:?}", i + 1, result);
    }
}

这个完整示例展示了：

1. 使用内置英语停用词
2. 创建自定义停用词
3. 合并多个停用词集合
4. 处理标点符号和大小写
5. 使用Rayon进行并行处理
6. 处理多个文档

stop-words库支持多种语言，包括英语、法语、德语、西班牙语等，完整列表可以参考库文档。