generation/cache

Cache classes used by generate() to preserve and reuse decoder past key/value tensors across calls. Pass a DynamicCache through generation options when you need to persist KV state between turns.

Type Definitions

DynamicCache

Mutable cache of decoder past key/value tensors used by generate().

Pass a DynamicCache through generation options when you need to preserve cache tensors across calls or inspect them from return_dict_in_generate.

Type: Record<string, Tensor>

DynamicCache.constructor([entries])

Create a DynamicCache, optionally pre-populated with entries.

Parameters

• entries (Record<string, Tensor>) optional — Initial name→Tensor mappings.

DynamicCache.get_seq_length()

Get the cached sequence length. This requires at least one attention cache entry to be present.

Returns: number — The past sequence length.

DynamicCache.update(newEntries)

Update the cache in-place with new entries, disposing replaced GPU tensors.

Parameters

• newEntries (Record<string, Tensor>) — The new name → Tensor mappings.

DynamicCache.dispose()

Dispose all contained tensors whose data resides on the GPU. Returns a promise that resolves when all disposals are complete.

Returns: Promise<void> — Promise that resolves when all GPU tensors are disposed.