EmbeddingLookup

EmbeddingLookup#

class penzai.deprecated.v1.nn.embeddings.EmbeddingLookup[source]#

Bases: Layer

Looks up token IDs in an embedding table.

This layer can be used to look up token embeddings in an embedding table. It is usually the first layer in a language model.

This lookup layer does not include any rescaling of the embeddings. If you would like to scale the embeddings after retrieval, consider adding a pz.nn.ConstantRescale layer after the EmbeddingLookup.

Variables:

table (EmbeddingTable) – The embedding table to look up embeddings in.

Methods

__init__(table)

input_structure()

output_structure()

__call__(token_index)

Retrieves tokens from the embedding table.

Attributes

table

Inherited Methods

(expand to view inherited methods)

attributes_dict()

Constructs a dictionary with all of the fields in the class.

from_attributes(**field_values)

Directly instantiates a struct given all of its fields.

key_for_field(field_name)

Generates a JAX PyTree key for a given field name.

select()

Wraps this struct in a selection, enabling functional-style mutations.

tree_flatten()

Flattens this tree node.

tree_flatten_with_keys()

Flattens this tree node with keys.

tree_unflatten(aux_data, children)

Unflattens this tree node.

treescope_color()

Computes a CSS color to display for this object in treescope.

__call__(token_index: named_axes.NamedArray) named_axes.NamedArray[source]#

Retrieves tokens from the embedding table.

Parameters:

token_index – A named array of token indices. Axis names of this array must be disjoint from the axis names in the embedding table.

Returns:

A named array of embeddings, which includes all named axes of the input along with the (non-vocabulary) named axes of the embedding table.