show_token_array#
- penzai.toolshed.token_visualization.show_token_array(tokens: pz.nx.NamedArrayBase | jax.Array | np.ndarray, vocab: SentencePieceProcessorLike, axis_name_order: Sequence[str] | None = None)[source]#
Renders an array of tokens.
- Parameters:
tokens – An array of token IDs.
vocab – Vocabulary, usually a SentencePiece tokenizer.
axis_name_order – Optional ordering of the axis names. The last axis name will be laid out horizontally. Inferred if not given.
- Returns:
A visualization of the token array.