LoreonLabsPlatform

Overview

Intelligence

Markets
Builders
Research
Narratives
Ecosystems
Launchpads

Discover

Search
Sources

PHP

tokenizers-php

Fast, pure-PHP tokenizer library compatible with Hugging Face tokenizers for encoding and decoding text

PHPEmergingphptokenizationtokenizers

Stars

18

Forks

—

Contributors

1

Last push

14h ago

Recent commits

Latest commits.

fix: swap encode/decode to read correct vocab direction
fb2b30dKyrian Obikwelu14h ago
feat: make Encoding class implement Countable
83e9f6dKyrian Obikwelu4mo ago
refactor: remove redundant modelMaxLength and expose token arrays
7ade08bKyrian Obikwelu4mo ago
feat: implement getConfig for tokenizer and all components
c907ffdKyrian Obikwelu4mo ago
feat: add FixedLengthPreTokenizer for fixed-length tokenization
637bca7Kyrian Obikwelu4mo ago

chore: code style fixes

e5db652Kyrian Obikwelu4mo ago

feat: use Hugging Face PHP for Hub loading

760f14bKyrian Obikwelu4mo ago

Merge pull request #2 from CodeWithKyrian/chore/php-8-2-and-readonly-classes

e7d79feKyrian Obikwelu4mo ago

Top contributors

Builders behind this project.