Rust数据处理插件库rgwml的使用：高效机器学习与工作流自动化工具

概述

RGWML是一个专注于简化数据科学、机器学习和人工智能操作的Rust库。它被设计为优雅、高效且易于使用，主要特点包括：

• 提供类似Python Pandas的CSV处理功能
• 支持多种机器学习算法（XGBoost和各种聚类算法）
• 与多种数据库集成（MySQL、MSSQL、Clickhouse、Google Big Query等）
• 支持Google Sheets和H5文件处理
• 提供OpenAI API集成

安装依赖

首先需要安装系统依赖：

sudo apt-get update
sudo apt-get install python3-pip libxgboost-dev libhdf5-dev

然后安装Python依赖：

pip3 install google-cloud-bigquery clickhouse-driver pandas xgboost scikit-learn numpy h5py tables dask dask[dataframe] dask[distributed]

CSV处理示例

基本CSV操作

use rgwml::csv_utils::CsvBuilder;

// 创建新CSV
let builder = CsvBuilder::new()
    .set_header(&["Column1", "Column2", "Column3"])
    .add_rows(&[&["Row1-1", "Row1-2", "Row1-3"], &["Row2-1", "Row2-2", "Row2-3"]])
    .save_as("/path/to/your/file.csv");

// 从现有CSV加载
let builder = CsvBuilder::from_csv("/path/to/existing/file.csv");

// 从Google Sheets加载
use tokio::runtime::Runtime;
let rt = Runtime::new().unwrap();
rt.block_on(async {
    let csv_builder = CsvBuilder::from_publicly_viewable_google_sheet(
        "https://docs.google.com/spreadsheets/d/1U9ozNFwV__c15z4Mp_EWorGwOv6mZPaQ9dmYtjmCPow/edit#gid=272498272"
    ).await;
});

高级CSV操作

use rgwml::csv_utils::{Exp, ExpVal, CsvBuilder};

let _ = CsvBuilder::from_csv("/path/to/your/file.csv")
    .rename_columns vec![("OLD_COLUMN", "NEW_COLUMN")])
    .drop_columns(vec!["UNUSED_COLUMN"])
    .where_(
        vec![
            ("Exp1", Exp {
                column: "customer_type".to_string(),
                operator: "==".to_string(),
                compare_with: ExpVal::STR("REGULAR".to_string()),
                compare_as: "TEXT".to_string()
            }),
            ("Exp2", Exp {
                column: "invoice_data".to_string(),
                operator: ">".to_string(),
                compare_with: ExpVal::STR("2023-12-31 23:59:59".to_string()),
                compare_as: "TEXT".to_string()
            }),
        ],
        "Exp1 && Exp2"
    )
    .save_as("/path/to/modified/file.csv");

机器学习示例

XGBoost模型训练与预测

use rgwml::csv_utils::CsvBuilder;
use rgwml::xgb_utils::XgbConfig;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let mut builder = CsvBuilder::from_csv("/path/to/your/training_data.csv");
    
    let (builder, report) = builder.create_xgb_model(
        "feature1,feature2",  // 特征列
        "target_column",     // 目标列
        "PREDICTION",       // 预测列名
        "/model/directory", // 模型保存目录
        "my_model",         // 模型名称
        XgbConfig::default() // 使用默认配置
    ).await;
});

聚类分析

use rgwml::csv_utils::CsvBuilder;
use rgwml::clustering_utils::ClusteringConfig;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let mut builder = CsvBuilder::from_csv("/path/to/your/data.csv");
    
    let clustering_config = ClusteringConfig {
        operation: "KMEANS".to_string(),
        optimal_n_cluster_finding_method: "ELBOW".to_string(),
        ..Default::default()
    };
    
    builder.append_clustering_column(
        "feature1,feature2",  // 特征列
        "CLUSTER",           // 聚类结果列名
        clustering_config
    ).await;
});

数据库操作示例

查询MySQL数据库

use rgwml::csv_utils::CsvBuilder;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let result = CsvBuilder::from_mysql_query(
        "username", 
        "password", 
        "server", 
        "database", 
        "SELECT * FROM your_table"
    ).await;
});

查询Google BigQuery

use rgwml::csv_utils::CsvBuilder;
use tokio::runtime::Runtime;

let rt = Runtime::new().unwrap();
rt.block_on(async {
    let result = CsvBuilder::from_google_big_query_query(
        "path/to/credentials.json",
        "SELECT * FROM your_table"
    ).await;
});

完整示例：端到端机器学习工作流

use rgwml::csv_utils::{CsvBuilder, Exp, ExpVal};
use rgwml::xgb_utils::XgbConfig;
use rgwml::clustering_utils::ClusteringConfig;
use tokio::runtime::Runtime;

#[tokio::main]
async fn main() {
    // 1. 从数据库加载数据
    let mut builder = CsvBuilder::from_mysql_query(
        "username",
        "password",
        "server",
        "database",
        "SELECT * FROM customer_data"
    ).await.expect("Failed to load data from MySQL");
    
    // 2. 数据清洗和预处理
    builder
        .drop_columns(vec!["id", "unused_column"])
        .replace_all(vec!["*"], vec![("NA", "0"), ("null", "0")])
        .clean_or_test_clean_by_eliminating_rows_subject_to_column_parse_rules(
            "age:IS_NUMERICAL_VALUE;income:IS_POSITIVE_NUMERICAL_VALUE"
        );
    
    // 3. 特征工程
    builder
        .append_derived_category_column(
            "INCOME_GROUP",
            vec![
                ("LOW", vec![
                    ("Exp1", Exp {
                        column: "income".to_string(),
                        operator: "<".to_string(),
                        compare_with: ExpVal::STR("30000".to_string()),
                        compare_as: "NUMBERS".to_string()
                    }),
                ], "Exp1"),
                ("MEDIUM", vec![
                    ("Exp1", Exp {
                        column: "income".to_string(),
                        operator: ">=".to_string(),
                        compare_with: ExpVal::STR("30000".to_string()),
                        compare_as: "NUMBERS".to_string()
                    }),
                    ("Exp2", Exp {
                        column: "income".to_string(),
                        operator: "<".to_string(),
                        compare_with: ExpVal::STR("80000".to_string()),
                        compare_as: "NUMBERS".to_string()
                    }),
                ], "Exp1 && Exp2"),
                ("HIGH", vec![
                    ("Exp1", Exp {
                        column: "income".to_string(),
                        operator: ">=".to_string(),
                        compare_with: ExpVal::STR("80000".to_string()),
                        compare_as: "NUMBERS".to_string()
                    }),
                ], "Exp1")
            ]
        );
    
    // 4. 训练XGBoost模型
    let (builder, _) = builder.create_xgb_model(
        "age,income,spending_score",
        "churn",
        "CHURN_PREDICTION",
        "/models",
        "churn_model",
        XgbConfig::default()
    ).await;
    
    // 5. 聚类分析
    builder.append_clustering_column(
        "age,income,spending_score",
        "CLUSTER",
        ClusteringConfig {
            operation: "KMEANS".to_string(),
            optimal_n_cluster_finding_method: "ELBOW".to_string(),
            ..Default::default()
        }
    ).await;
    
    // 6. 保存结果
    builder.save_as("/output/final_results.csv");
    
    println!("Analysis completed successfully!");
}

总结

RGWML提供了一套全面的工具，用于：

1. 数据获取和预处理（从CSV、数据库、Google Sheets等）
2. 数据清洗和转换
3. 特征工程
4. 机器学习模型训练和预测
5. 聚类分析
6. 结果保存和可视化

通过链式调用和方法组合，可以构建复杂的数据处理流水线，同时保持代码的可读性和可维护性。