Fast tiktoken BPE tokenizer for PostgreSQL implemented in C
| Package | pg_tiktoken_c |
|---|---|
| Version | 1.1 |
| PostgreSQL versions | 14 · 15 · 16 · 17 · 18 |
| Repository | PIGSTY |
| Extension type | Standard |
| Requires CREATE EXTENSION | Yes |
| Requires preload | No |
| Trusted | No |
| Relocatable | No |
| Shared libraries | $libdir/pg_tiktoken_c |
| Stars | 3 |
| Latest commit | 2026-05-25 |
| Catalogue updated | 2026-08-30 |
| pg_tiktoken | tiktoken tokenizer for use with OpenAI models in postgres |
| pg_tokenizer | Tokenizers for full-text search |
| pg_jieba | Chinese full-text search parser based on cppjieba |
| pg_cjk_parser | CJK bigram parser derived from PostgreSQL full-text search |
| zhparser | a parser for full-text search of Chinese |
| pg_bigm | create 2-gram (bigram) index for faster full text search. |
| pgroonga | Use Groonga as index, fast full text search platform for all languages! |
| dict_xsyn | text search dictionary template for extended synonym processing |