EmbeddingDecode

class penzai.nn.embeddings.EmbeddingDecode

Bases: Layer

Uses an embedding table to map embeddings back to token scores.

This layer can be used to map a model’s output embedding to the logits for a distribution over output tokens. It is usually the last layer in a language model.

The primary purpose of this layer is to simplify sharing weights between the embedding lookup and decode layers. It functions similarly to a Linear layer, but retrieves its parameter from an EmbeddingTable.

Variables:

table (EmbeddingTable) – The embedding table to look up embeddings in.

Methods

__init__(table)
__call__(out_embeddings, **_unused_side_inputs)	Retrieves tokens from the embedding table.

Attributes

table

Inherited Methods

(expand to view inherited methods)

attributes_dict()	Constructs a dictionary with all of the fields in the class.
bind_variables(variables[, allow_unused])	Convenience function to bind variables to a layer.
from_attributes(**field_values)	Directly instantiates a struct given all of its fields.
key_for_field(field_name)	Generates a JAX PyTree key for a given field name.
select()	Wraps this struct in a selection, enabling functional-style mutations.
stateless_call(variable_values, argument, /, ...)	Calls a layer with temporary variables, without modifying its state.
tree_flatten()	Flattens this tree node.
tree_flatten_with_keys()	Flattens this tree node with keys.
tree_unflatten(aux_data, children)	Unflattens this tree node.
treescope_color()	Computes a CSS color to display for this object in treescope.

__call__(out_embeddings: named_axes.NamedArray, **_unused_side_inputs) → named_axes.NamedArray

Retrieves tokens from the embedding table.

Parameters:

• out_embeddings – The output embeddings that should be mapped to token logits. Should be a named array that includes the same axes as the embedding table, except for the vocabulary axis, and may also include additional batch axes.

• **_unused_side_inputs – Side inputs (unused).

Returns:

A named array of logits, which includes all batch axes of the input along with the vocabulary axis of the embedding table.