@lenml/tokenizer-gemini

a tokenizer.

based on @lenml/tokenizers

Usage

import { fromPreTrained } from "@lenml/tokenizer-gemini";

const tokenizer = fromPreTrained();
console.log(
  "encode()",
  tokenizer.encode("Hello, my dog is cute", null, {
    add_special_tokens: true,
  })
);
console.log("_encode_text", tokenizer._encode_text("Hello, my dog is cute"));

Full Tokenizer API

Complete api parameters and usage can be found in transformer.js tokenizers document

License

Apache-2.0