Demystifying Anthropic's J-Space: A Mathematical Primer

In Verbalizable Representations Form a Global Workspace in Language Models, Anthropic researchers introduce the J-space — an LLM analogue (demonstrated on Claude Opus 4.6) to the global workspace in the human cortex, the neural circuit hypothesized to underpin access consciousness. The authors present extensive empirical evidence validating this connection and demonstrate the utility of the J-space as an auditing tool for LLM alignment.

Our goal in this brief article is not to contribute to that broader conceptual debate. Instead, more modestly, we re-examine the formal definition and mathematical properties of the J-space and provide precise justifications for the surgical interventions performed by Anthropic. Because the original paper’s concise style can leave key mechanics implicit, we aim to provide a clear, step-by-step account for readers who value clear definitions and explicit derivations—delivered directly through an applied mathematics lens, without unnecessary jargon.

J-space as a Union of Cones

To keep this article self-contained, we first recall the basic definitions. The semantic representation of a token at position t in layer 𝓁 within the residual stream of a Transformer:

We then define the average causal impact of a variation in this representation on the final layer L of the Transformer as an expectation over a set of prompts p, across all token positions t, and over all future context positions t':

Converting this impact to logit scores is achieved by multiplying on the left by the unembedding matrix. The rows of the resulting matrix are by definition the J-lens vectors (omitting the normalization step for simplicity):

Thus, there are as many J-lens vectors as there are tokens in the LLM's vocabulary, this for each layer 𝓁 (we drop the index 𝓁 hereafter to avoid clutter). Because

these J-lens vectors form an overcomplete frame in the representation space.

The average impact of a representation in layer 𝓁 on the Transformer's propensity to output token i at step t is simply given by the inner product

As a side note, assuming these vectors are distributed approximately uniformly on the sphere, a classical result from high-dimensional geometry (Lévy's concentration of measure on the sphere) guarantees that pair-wise inner products between distinct vectors are close to zero on average

except for semantically related vectors (such as "king" and "emperor" for instance). For a standard model dimension of 4096, this inner product is on the order of 0.015.

The J-space itself is then formally defined as the set of k-sparse linear combinations of J-lens vectors with non-negative coefficients. More precisely, for any subset of indices

we define the cone

Finally, the J-space is defined as the set of all points expressible as a k-sparse non-negative combination of J-lens vectors. Geometrically, it corresponds to the union of all such cones:

which is a union of cones, rather than a linear subspace.

Verbalizable and Non-Verbalizable Components

The verbalizable component of a representation is defined as its orthogonal projection onto the J-space—or, framed differently, as an approximate k-sparse expansion over the J-lens vectors:

The sparsity constraint selects J-lens vectors in this sum that are approximately orthogonal and thus semantically distant. Since inner products between such vectors are near zero, the only tokens receiving high scores in (4) belong to semantic clusters close to the k J-lens vectors in the decomposition.

The non-verbalizable component of a representation is simply the residual of this projection:

Standard results in convex optimization establish that the residual of a projection onto a closed convex cone is orthogonal to the projection itself. Consequently, the inner product

is identically zero—a property that will prove useful in the next section.

Defining J-Space Capacity and Selecting the Sparsity k

A key question is how to select the sparsity parameter k in (8). To address this, we introduce the fraction of variance explained (FVE) by the projection onto the J-space:

where the expectation is on prompts p and positions t. The second equality follows from the orthogonality relation (10). To isolate signal from noise—given that k random vectors would naturally account for a fraction of the variance—we consider the excess variance explained by k J-lens vectors, defined as:

where the second term in (12) is defined identically to the first, but replaces the k J-lens vectors with k randomly selected vectors. Empirically, this excess variance plateaus beyond k = 25. The interpretation is that k≈25 represents a natural elbow: up to this threshold, projecting onto the J-space captures significantly more information than projecting onto an equivalent number of random vectors, after which excess variance plateaus.

Numerically, this excess variance is roughly 10%, which is instructive to compare against the variance explained by 25 random vectors. For d = 4096 and k = 25, the expected fraction of variance explained by random vectors would be on the order of 25 / 4096 < 1% assuming the components are independent. This stark imbalance confirms the hypothesis that verbalizable representations lie substantially within a very low-dimensional manifold (25 ≪ 4096).

A note on computational complexity: Computing the exact projection in (9) onto the J-space requires solving a non-negative sparse coding problem, which is NP-hard in general. This exact calculation is likely only necessary when evaluating the FVE in (11), which is then performed offline. Using an approximate method—such as the regression heuristic discussed below—would cause correlated J-lens vectors to double-count the variance.

Interventions in J-Space

To define interventions on J-space representations that are computationally tractable in practice, Anthropic researchers rely on a simple linear regression framework, avoiding the NP-hard exact projection onto the J-space. Let us examine this in detail.

The practical objective of an intervention is to swap an active concept in the J-space with another. We define a two-column matrix containing the source and target vectors:

Computing the orthogonal projection of a representation h onto this two-dimensional subspace reduces to solving a linear regression problem:

with the resulting projection given by:

This projection is computed explicitly using the Moore-Penrose pseudo-inverse:

Substituting one concept for another in the J-space then simply amounts to permuting the components of c:

By replacing the J-space which is conical manifold with a 2D linear subspace, expensive optimization is replaced by a closed-form expression—ideal for targeted causal manipulations.

It remains to show that this intervention indeed approximates swapping the exact coefficients from the sparse decomposition in (9).

Assume the indices 's' and 't' belong to the active set S. Using (13), we can write without any approximation:

We thus need to establish that the regression coefficients for the two swapped vectors closely approximate the exact coefficients:

From (16) and (18), we have:

First, we prove the strict equality for the last term in (20):

The argument is geometric. Because the vectors associated with the source and target concepts are active, their coefficients in the conical projection (9) are strictly positive. A small perturbation along either direction thus remains inside the cone. Formally, since:

it follows that:

Since the projection minimizes the distance between the original representation and the points of the cone, differentiating this distance along the source or target vector must yield zero:

This gives the exact vanishing condition claimed in (21) when using (16):

As for the second term in (20), its vanishing is only approximate, relying on (5) via Lévy's concentration lemma:

which establishes (19), confirming that using linear regression used to swap two concepts in the J-space provides a sound approximation to an exact coefficient permutation.

This reasoning assumes, however, that the source and target vectors correspond to reasonably distinct concepts. Otherwise, the vectors become nearly collinear, rendering the pseudoinverse in (16) ill-posed as the Gram matrix approaches singularity:

which destabilizes the coefficients c and may even yield negative values, causing the resulting vector to escape outside of the J-space cone.

This concludes our brief excursion into the linear algebra of the J-space.