SentencePieceProcessorLike

class penzai.toolshed.token_visualization.SentencePieceProcessorLike

Bases: Protocol

Protocol defining the methods we need from a tokenizer.

Sentencepiece tokenizers conform to this interface, but anything else that implements it will also work.

Methods

GetPieceSize()	Returns the number of tokens in the vocabulary.
IdToPiece(id)	Decodes a token ID to a string.
IsControl(id)	Identifies whether a token is a control token.
__init__(*args, **kwargs)

GetPieceSize() → int

Returns the number of tokens in the vocabulary.

IdToPiece(id: int) → str

Decodes a token ID to a string.

IsControl(id: int) → bool

Identifies whether a token is a control token.