LoreonLabsPlatform

Overview

Intelligence

Markets
Builders
Research
Narratives
Ecosystems
Launchpads

Discover

Search
Sources

Other

tokenizers

Go bindings for Tiktoken & HuggingFace Tokenizer

OtherEmerging

Stars

—

Forks

—

Contributors

7

Last push

10mo ago

Recent commits

Latest commits.

fix: handle impartial utf-8 decoding
1045692Daulet Zhanguzin11mo ago
fix: link time version errors instead of runtime checks
f3bea18Daulet Zhanguzin11mo ago
fix: bubble up errors
c7bb9afDaulet Zhanguzin11mo ago
chore: update example to latest release
ad3e9cdDaulet Zhanguzin11mo ago
fix: handle non-utf8 input, lossy for now
99d8a14Daulet Zhanguzin11mo ago

fix: handle utf8 and add missing reserved tokens

62d5d7dDaulet Zhanguzin11mo ago

fix: handle older tokenizer configs

d6cd62aDaulet Zhanguzin11mo ago

chore: update example with tiktoken use

e7956e5Daulet Zhanguzin11mo ago

Top contributors

Builders behind this project.

riccardopinosio