token_visualization

Helpers to visualize tokens, token probabilities, and token sequences.

Classes

SentencePieceProcessorLike

Protocol defining the methods we need from a tokenizer.

Functions

show_token_array(tokens, vocab[, ...])	Renders an array of tokens.
show_token_scores(tokens, scores, vocab[, ...])	Renders an array of token scores, but with the scored tokens as words.