EmbeddingLookup

class penzai.nn.embeddings.EmbeddingLookup

Bases: Layer

Looks up token IDs in an embedding table.

This layer can be used to look up token embeddings in an embedding table. It is usually the first layer in a language model.

This lookup layer does not include any rescaling of the embeddings. If you would like to scale the embeddings after retrieval, consider adding a pz.nn.ConstantRescale layer after the EmbeddingLookup.

Variables:

table (EmbeddingTable) – The embedding table to look up embeddings in.

Methods

__init__(table)
input_structure()
output_structure()
__call__(token_index)	Retrieves tokens from the embedding table.

Attributes

table

Inherited Methods

(expand to view inherited methods)

attributes_dict()	Constructs a dictionary with all of the fields in the class.
from_attributes(**field_values)	Directly instantiates a struct given all of its fields.
key_for_field(field_name)	Generates a JAX PyTree key for a given field name.
select()	Wraps this struct in a selection, enabling functional-style mutations.
tree_flatten()	Flattens this tree node.
tree_flatten_with_keys()	Flattens this tree node with keys.
tree_unflatten(aux_data, children)	Unflattens this tree node.
treescope_color()	Computes a CSS color to display for this object in treescope.

__call__(token_index: named_axes.NamedArray) → named_axes.NamedArray

Retrieves tokens from the embedding table.

Parameters:

token_index – A named array of token indices. Axis names of this array must be disjoint from the axis names in the embedding table.

Returns:

A named array of embeddings, which includes all named axes of the input along with the (non-vocabulary) named axes of the embedding table.