golang低延迟机器学习特征快速处理插件库go-featureprocessing的使用
golang低延迟机器学习特征快速处理插件库go-featureprocessing的使用
go-featureprocessing是一个快速、简单的类似sklearn的特征处理库，专为Go语言设计。
主要特性
- 不跨越cgo边界
- 无内存分配
- 不使用反射
- 方便的序列化
- 生成的代码有100%测试覆盖率和基准测试
- 支持拟合
- 支持UTF-8
- 并行批量转换
- 在批量模式下比sklearn更快
使用示例
基本用法
//go:generate go run github.com/nikolaydubina/go-featureprocessing/cmd/generate -struct=Employee
type Employee struct {
Age int `feature:"identity"`
Salary float64 `feature:"minmax"`
Kids int `feature:"maxabs"`
Weight float64 `feature:"standard"`
Height float64 `feature:"quantile"`
City string `feature:"onehot"`
Car string `feature:"ordinal"`
Income float64 `feature:"kbins"`
Description string `feature:"tfidf"`
SecretValue float64
}
employee := Employee{
Age: 22,
Salary: 1000.0,
Kids: 2,
Weight: 85.1,
Height: 160.0,
City: "Pangyo",
Car: "Tesla",
Income: 9000.1,
SecretValue: 42,
Description: "large text fields is not a problem neither, tf-idf can help here too! more advanced NLP will be added later!",
}
var fp EmployeeFeatureTransformer
config, _ := ioutil.ReadAll("employee_feature_processor.json")
json.Unmarshal(config, &fp)
features := fp.Transform(&employee)
// []float64{22, 1, 0.5, 1.0039999999999998, 1, 1, 0, 0, 0, 1, 5, 0.7674945674619879, 0.4532946552278861, 0.4532946552278861}
names := fp.FeatureNames()
// []string{"Age", "Salary", "Kids", "Weight", "Height", "City_Pangyo", "City_Seoul", "City_Daejeon", "City_Busan", "Car", "Income", "Description_text", "Description_problem", "Description_help"}
拟合数据
fp := EmployeeFeatureTransformer{}
fp.Fit([]Employee{...})
config, _ := json.Marshal(data)
_ = ioutil.WriteFile("employee_feature_processor.json", config, 0644)
手动初始化
fp := EmployeeFeatureTransformer{
Salary: MinMaxScaler{Min: 500, Max: 900},
Kids: MaxAbsScaler{Max: 4},
Weight: StandardScaler{Mean: 60, STD: 25},
Height: QuantileScaler{Quantiles: []float64{20, 100, 110, 120, 150}},
City: OneHotEncoder{Mapping: map[string]uint{"Pangyo": 0, "Seoul": 1, "Daejeon": 2, "Busan": 3}},
Car: OrdinalEncoder{Mapping: map[string]uint{"Tesla": 1, "BMW": 90000}},
Income: KBinsDiscretizer{QuantileScaler: QuantileScaler{Quantiles: []float64{1000, 1100, 2000, 3000, 10000}}},
Description: TFIDFVectorizer{
NumDocuments: 2,
DocCount: []uint{1, 2, 2},
CountVectorizer: CountVectorizer{Mapping: map[string]uint{"text": 0, "problem": 1, "help": 2}, Separator: " "},
},
}
性能基准
对于典型使用场景，这个结构体编码器可以在单个样本上获得约100ns的处理时间。性能基准如下：
0 - C++ FlatBuffers decode
...
200ps - 4.6GHz single cycle time
1ns - L1 cache latency
10ns - L2/L3 cache SRAM latency
20ns - DDR4 CAS, first byte from memory latency
20ns - C++ raw hardcoded structs access
80ns - C++ FlatBuffers decode/traverse/dealloc
----------> 100ns - go-featureprocessing typical processing
150ns - PCIe bus latency
171ns - Go cgo call boundary, 2015
200ns - some High Frequency Trading FPGA claims
800ns - Go Protocol Buffers Marshal
837ns - Go json-iterator/go json decode
1µs - Go Protocol Buffers Unmarshal
1µs - High Frequency Trading FPGA
3µs - Go JSON Marshal
7µs - Go JSON Unmarshal
9µs - Go XML Marshal
10µs - PCIe/NVLink startup time
17µs - Python JSON encode or decode times
30µs - UNIX domain socket, eventfd, fifo pipes latency
30µs - Go XML Unmarshal
100µs - Redis intrinsic latency
100µs - AWS DynamoDB + DAX
100µs - KDB+ queries
100µs - High Frequency Trading direct market access range
200µs - 1GB/s network air latency
200µs - Go garbage collector latency 2018
500µs - NGINX/Kong added latency
10ms - AWS DynamoDB
10ms - WIFI6 "air" latency
15ms - AWS Sagemaker latency
30ms - 5G "air" latency
100ms - typical roundtrip from mobile to backend
200ms - AWS RDS MySQL/PostgreSQL or AWS Aurora
10s - AWS Cloudfront 1MB transfer time
反射版本（beta）
如果你不能使用go:gencode版本，可以尝试基于反射的版本。但需要注意，反射版本会引入额外的开销，特别是当你的结构体有很多字段时。
go test -timeout=1h -bench=. -benchtime=10s -benchmem ./...
基准测试结果：
goos: darwin
goarch: amd64
// reflection
pkg: github.com/nikolaydubina/go-featureprocessing/structtransformer
BenchmarkStructTransformerTransform_32fields-4 1732573 2079 ns/op 512 B/op 2 allocs/op
// non-reflection
pkg: github.com/nikolaydubina/go-featureprocessing/cmd/generate/tests
BenchmarkWith32FieldsFeatureTransformer_Transform-8 31678317 116 ns/op 256 B/op 1 allocs/op
BenchmarkWith32FieldsFeatureTransformer_Transform_Inplace-8 80729049 43 ns/op 0 B/op 0 allocs/op
更多关于golang低延迟机器学习特征快速处理插件库go-featureprocessing的使用的实战教程也可以访问 https://www.itying.com/category-94-b0.html
1 回复
更多关于golang低延迟机器学习特征快速处理插件库go-featureprocessing的使用的实战系列教程也可以访问 https://www.itying.com/category-94-b0.html
go-featureprocessing: 低延迟机器学习特征处理库
go-featureprocessing 是一个专为 Golang 设计的轻量级、高性能机器学习特征处理库，特别适合需要低延迟预测服务的场景。下面我将详细介绍它的使用方法和示例代码。
核心特性
- 低延迟处理：优化特征转换速度
- 简单API：易于集成到现有ML pipeline
- 内存高效：最小化内存占用
- 类型安全：严格的Golang类型检查
安装
go get github.com/nikolaydubina/go-featureprocessing
基本使用示例
1. 数值特征标准化
package main
import (
"fmt"
"github.com/nikolaydubina/go-featureprocessing/transformers"
)
func main() {
// 创建标准化转换器(均值=10, 标准差=2)
scaler := transformers.StandardScaler{Mean: 10, Scale: 2}
// 转换单个特征
scaledValue := scaler.Transform(12)
fmt.Printf("标准化后的值: %.2f\n", scaledValue) // 输出: 1.00
// 批量转换
values := []float64{8, 10, 12, 14}
scaledValues := make([]float64, len(values))
for i, v := range values {
scaledValues[i] = scaler.Transform(v)
}
fmt.Printf("批量标准化结果: %v\n", scaledValues) // 输出: [-1 0 1 2]
}
2. 类别特征编码
package main
import (
"fmt"
"github.com/nikolaydubina/go-featureprocessing/transformers"
)
func main() {
// 创建OneHot编码器
encoder := transformers.OneHotEncoder{
Categories: [][]string{
{"red", "green", "blue"}, // 第一个特征的可能值
{"small", "large"}, // 第二个特征的可能值
},
}
// 编码样本
sample := []string{"green", "large"}
encoded := encoder.Transform(sample)
fmt.Printf("OneHot编码结果: %v\n", encoded)
// 输出: [0 1 0 0 1] (绿色+大号)
}
3. 完整特征流水线示例
package main
import (
"fmt"
"github.com/nikolaydubina/go-featureprocessing"
)
type Sample struct {
Age int `feature:"scale-standard"`
Income float64 `feature:"scale-minmax"`
Education string `feature:"one-hot"`
HasLoan bool `feature:"binary"`
Temperature float64 // 无tag，不会被处理
}
func main() {
// 创建特征处理器
processor, err := featureprocessing.NewFeatureProcessorFromStruct(Sample{})
if err != nil {
panic(err)
}
// 训练处理器(通常在离线阶段完成)
processor.Fit([]Sample{
{Age: 25, Income: 50000, Education: "bachelor", HasLoan: true},
{Age: 30, Income: 80000, Education: "master", HasLoan: false},
{Age: 35, Income: 120000, Education: "phd", HasLoan: true},
})
// 转换新样本
sample := Sample{
Age: 28,
Income: 65000,
Education: "master",
HasLoan: false,
}
features := processor.Transform(sample)
fmt.Printf("处理后的特征向量: %v\n", features)
// 输出类似: [-0.5 0.375 0 1 0 0]
}
性能优化技巧
- 复用转换器对象：避免重复创建
- 预分配切片：减少内存分配
- 批量处理：利用CPU缓存局部性
- 并行处理：对独立特征使用goroutine
// 并行特征处理示例
func processConcurrently(samples []Sample, processor *featureprocessing.FeatureProcessor) [][]float64 {
result := make([][]float64, len(samples))
var wg sync.WaitGroup
for i := range samples {
wg.Add(1)
go func(idx int) {
defer wg.Done()
result[idx] = processor.Transform(samples[idx])
}(i)
}
wg.Wait()
return result
}
适用场景
- 实时预测服务
- 边缘设备上的ML推理
- 高吞吐量数据处理
- 需要与Golang服务集成的ML应用
go-featureprocessing 通过简洁的API和高效的实现，为Golang开发者提供了强大的特征处理能力，特别适合对延迟敏感的应用场景。
