LoreonLabsPlatform
DocsHome
  • Overview

Intelligence

  • Markets
  • Builders
  • Research
  • Ecosystems
  • Launchpads
  • Search
Ecosystems

Other

tokenizers

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

OtherEmerging
GitHubWebsite
Stars
—
Forks
—
Contributors
8
Last push
37mo ago

Recent commits

Latest commits.

  • Improve error for truncation with too high stride (#1275)
    8c9cfb0Connor Boyle37mo ago
  • [doc build] Use secrets (#1273)
    348ed70Mishig37mo ago
  • Update README.md - Broken link (#1272)
    5d70f15Santosh Bhavani37mo ago
  • Update Cargo.toml (#1266)
    f85e846Chris Ha37mo ago
  • fix documentation regarding regex (#1264)
    cb8d4deChris Ha37mo ago
Fixing broken link. (#1268)
c7102c4Nicolas Patry37mo ago
  • Update trainer.rs (#1257)
    cb81972Chris Ha37mo ago
  • Parallelize unigram trainer (#976)
    fc76ad4Mishig37mo ago

    • Top contributors

    Builders behind this project.

    n1t0
    1K commits
    Narsil
    214 commits
    Pierrci
    145 commits
    epwalsh
    50 commits
    mfuntowicz
    35 commits
    mishig25
    28 commits
    sebpuetz
    28 commits
    dependabot[bot]
    21 commits