Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer

Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer(github.com)

2 points by zurawiki 3 years ago | 1 comment

zurawiki 3 years ago |

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models

zurawiki 3 years ago |

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models