GAT: Graph Attention Networks

A node weights each neighbor by a learned score, not by node degree.

A graph attention layer scores every edge from the two nodes' features, normalizes the scores with a softmax inside each neighborhood, and averages neighbor features with those weights. Nothing in the layer depends on the graph it was trained on.

Two learned vectors and a softmax turn a graph convolution's fixed neighbor weights into weights computed from the features, edge by edge.

The task is node classification. A graph has $N$ nodes, each carrying a feature vector $\vec h_i \in \mathbb{R}^F$, and a few nodes carry a label. On the Cora citation graph that is 2,708 papers, a 1,433-dimensional bag-of-words vector per paper, 5,429 citation edges, 7 topic classes, and labels on 140 papers, 20 per class. The model is scored on 1,000 of the remaining papers. A graph convolutional network (GCN) does this by replacing each node's features with a weighted sum over the node and its neighbors, where the weight on edge $(i, j)$ is $1/\sqrt{d_i d_j}$, a number set by the two degrees (self-loop included) before training starts. A paper cited by a closely related paper and a paper cited by an unrelated one get the same weight if the degrees match.

Graph attention networks replace that fixed weight with a number computed from the two nodes' features by a small function shared across every edge, then normalized across the neighborhood so the weights into any node sum to one. The paper lists what that buys: the work is parallel across edges and across nodes, a node with three neighbors and a node with three hundred use the same parameters, and because no parameter has a size tied to the graph, a trained layer runs unchanged on a graph it never saw. On the protein-protein interaction (PPI) benchmark the test graphs are held out of training entirely, and the model reaches a micro-averaged F1 score (precision and recall pooled over every node and label) of 0.973, against 0.768 for the strongest GraphSAGE the authors could configure.

The layer takes six equations: a score per edge, (1) and (3); a masked softmax that turns scores into weights, (2); a weighted sum, (4); and two ways of combining several attention heads, (5) and (6). After those, this page walks a property of the scoring function that the paper did not notice and a later paper named, the training setup on 140 labels, and the results with their margins measured in points.

Scoring one neighbor

A layer maps the input features $\vec h_i \in \mathbb{R}^F$ of every node to output features $\vec h'_i \in \mathbb{R}^{F'}$. The first step is a linear map $\mathbf W \in \mathbb{R}^{F' \times F}$ applied to every node with the same weights. On Cora's first layer $F = 1433$ and $F' = 8$, so $\mathbf W$ is 8 × 1,433, 11,464 weights shared by all 2,708 nodes, and $\mathbf W \vec h_j$ is an 8-vector per node. The score of neighbor $j$ for node $i$ is a function of the two projected vectors:

The paper reads $e_{ij}$ as the importance of node $j$'s features to node $i$. The function $a$ is a single-layer network: concatenate the two 8-vectors into a 16-vector, take its dot product with a learned vector $\vec{\mathbf a} \in \mathbb{R}^{2F'}$, and pass the result through a LeakyReLU with negative slope 0.2 (the identity for positive inputs; negative inputs are multiplied by 0.2). With the softmax of the next section folded in, the paper's fully expanded form is:

Here $\|$ is concatenation and $\mathcal N_i$ is the neighborhood of $i$. The dot product with a concatenation splits into two dot products. Write $\vec{\mathbf a} = [\vec{\mathbf a}_1; \vec{\mathbf a}_2]$ with each half in $\mathbb{R}^{F'}$; then

The official code computes it this way: two 1 × 1 convolutions over the projected features produce $s$ and $t$ as one number per node, and adding $s$ as a column to $t$ as a row gives every $s_i + t_j$ at once. So the per-node work is two dot products of length 8, and the per-edge work is one addition and one LeakyReLU. A node's own $s_i$ is the same for all its neighbors; what separates the neighbors is $t_j$, the projection of each neighbor's features onto $\vec{\mathbf a}_2$.

For a worked row, take a node $i$ with four neighbors, so five members of the neighborhood once $i$ itself is included (the next section says why it is). Suppose $s_i = 0.40$ and the five $t$ values are 0.25 for $i$ itself and 0.90, −0.30, 0.10, −1.20 for the neighbors. The sums are 0.65, 1.30, 0.10, 0.50, −0.80, and the LeakyReLU leaves the positive ones alone and multiplies −0.80 by 0.2, giving scores of 0.65, 1.30, 0.10, 0.50, −0.16. A plain ReLU would have sent −0.80 to exactly 0 and given that neighbor a zero gradient into $\vec{\mathbf a}$; the paper does not state its reason for the leaky variant, and this is the natural one. Since the LeakyReLU is monotone it never reorders the neighbors; it compresses the negative side of the scale, so the softmax takes less weight away from low-scoring neighbors than it would from the raw sums.

In the figure below the plane is the projected feature space, each amber point is one neighbor's $\mathbf W \vec h_j$, and the amber arrow is $\vec{\mathbf a}_2$. The dashed lines are lines of equal $t_j$, perpendicular to the arrow. Drag a neighbor along the arrow and its score and weight rise; drag it along a dashed line and nothing changes. The figure's second slider (the first rotates $\vec{\mathbf a}_2$) scales the whole vector $\vec{\mathbf a}$; at zero every score is equal and every weight is 1/6, which is the constant-attention baseline the paper uses as a control in the PPI experiment.

Masking and the softmax

Equation (1) as written could score every pair of nodes, and the paper says so: in its most general form every node attends to every other node and the graph structure is dropped entirely. The structure enters by computing $e_{ij}$ only for $j \in \mathcal N_i$, the first-order neighbors of $i$, with $i$ itself included. The paper calls this masked attention. The scores are then normalized within the neighborhood so they are comparable across nodes:

For the worked row, the exponentials of 0.65, 1.30, 0.10, 0.50, −0.16 are 1.916, 3.669, 1.105, 1.649, 0.852, their sum is 9.191, and dividing gives weights of 0.208 for the node itself and 0.399, 0.120, 0.179, 0.093 for the neighbors. Every weight is positive and the five sum to one. The neighbor with $t_j = 0.90$ takes almost 40% of the row; the neighbor at −1.20 keeps 9%, since the leaky slope kept its score close to the others.

The code does the masking with a dense matrix. It computes all $N \times N$ scores, adds a bias matrix that is 0 on every edge and self-loop and −10⁹ everywhere else, and takes the softmax of the sum; the exponential of a score near −10⁹ underflows to exactly 0, so non-neighbors receive weight 0 and each row still sums to 1 over the true neighborhood. The self-loops come from adding the identity to the adjacency matrix before the mask is built. On Cora that matrix has 2,708² = 7,333,264 entries, of which about 13,600 (two per edge plus one per node) are inside the mask, and at 4 bytes per entry it costs 29.3 MB per head, 235 MB for the eight heads of the first layer. Pubmed has 19,717 nodes, so its dense score matrix has 388.8 million entries and costs 1.56 GB per head. That is why the authors wrote a sparse variant of the layer. Their framework, as the paper says, only supports sparse multiplication of rank-2 tensors, which limits that variant to one graph at a time.

Two properties of the result are used below. The weights are asymmetric: $e_{ij} = \mathrm{LeakyReLU}(s_i + t_j)$ while $e_{ji} = \mathrm{LeakyReLU}(s_j + t_i)$, and the two rows are normalized separately, so $\alpha_{ij}$ and $\alpha_{ji}$ generally differ. And the graph need not be undirected: for a directed graph the code leaves $\alpha_{ij}$ out whenever the edge $j \to i$ is absent, and nothing else changes. The figure shows the three matrices in the code's order: dense scores, the mask, and the softmax of their sum. Click an off-diagonal cell to add or remove an edge and watch the two affected rows renormalize, and hover a cell to compare $\alpha_{ij}$ with $\alpha_{ji}$.

The paper gives the cost of one head as $O(|V| F F' + |E| F')$: the projection of every node, plus the weighted sum over every edge. On Cora the projection is 2,708 × 1,433 × 8, about 31.0 million multiply-adds, and the aggregation over the 13,600 mask entries is about 109,000, so the projection dominates and the cost matches a GCN layer of the same width. There is no eigendecomposition and no matrix inverse anywhere; spectral methods need the eigenbasis of the graph Laplacian, which is what ties them to one graph.

The layer output

With the weights in hand, node $i$'s new features are the weighted sum of its neighbors' projected features, followed by a nonlinearity $\sigma$:

For the worked row that is

an 8-vector, with an ELU (exponential linear unit: the identity for positive inputs, e^x − 1 below zero) as the hidden layer's nonlinearity. The code adds a bias vector before the nonlinearity, which equation (4) omits. The same $\mathbf W$ that produced the scores produces the values being summed, so one matrix does the work that a Transformer splits across query, key and value projections.

Set this next to the GCN rule over the same closed neighborhood,

The shapes match line for line; the only change is the coefficient in front of each neighbor, a fixed function of two degrees in the GCN and a learned function of two feature vectors here. The paper's constant-attention control, $a(x, y) = 1$, sits between the two: every score is equal, so the softmax gives each member of the closed neighborhood $1/|\mathcal N_i|$, a plain mean.

The whole head, in the shapes the code uses:

# one attention head: N nodes, F features in, F' features out
Wh    = h @ W.T                      # [N, F']  shared projection (eq. 1)
s     = Wh @ a1                      # [N]      query half of a^T[Wh_i || Wh_j]
t     = Wh @ a2                      # [N]      neighbor half
e     = leaky_relu(s[:, None] + t[None, :], 0.2)   # [N, N]  every pair (eq. 3)
e     = e + mask                     # 0 on edges and self-loops, -1e9 elsewhere
alpha = softmax(e, axis=1)           # [N, N]  each row sums to 1 over N_i (eq. 2)
alpha = dropout(alpha, p=0.6)        # training only: a random sub-neighborhood
h_out = elu(alpha @ Wh + b)          # [N, F']  eq. 4 (the code adds a bias b)

Gradients reach $\mathbf W$ through two routes. The loss is a cross-entropy on the 140 labeled nodes. Each labeled node's logits depend on $\mathbf W$ through the summed values $\mathbf W \vec h_j$ and, separately, through the scores that set $\alpha_{ij}$; the attention vector $\vec{\mathbf a}$ gets gradient only through the softmax. An unlabeled neighbor contributes its features to a labeled node's output, so its projection is in the computation graph and the shared $\mathbf W$ learns from it, as in a GCN. The weight count of a head is $F' F + 2F'$, 11,480 on Cora's first layer before the code's bias terms, with no term in $N$ or in the edge count, which is the whole reason the same weights can be applied to a different graph.

Several heads, concatenated or averaged

One head computes one set of weights per neighborhood. The paper runs $K$ heads in parallel, each with its own $\mathbf W^k$ and $\vec{\mathbf a}^k$, and in a hidden layer concatenates their outputs:

The output has $K F'$ features. Cora's first layer uses $K = 8$ heads of $F' = 8$, so each node leaves the layer with 64 features, and the layer holds 8 × 11,480 = 91,840 parameters. Each head weights the same neighborhood differently, since each has its own $\vec{\mathbf a}^k_2$ to project onto. The paper's stated reason for several heads is that they stabilize training, following the Transformer; it reports no ablation over $K$.

In the output layer concatenation would leave $K C$ logits for $C$ classes, so the paper averages the heads instead and applies the final nonlinearity (a softmax over classes, or a sigmoid per label) after the average:

Cora's output layer is a single head with $C = 7$ outputs: $\mathbf W$ is 7 × 64 and $\vec{\mathbf a}$ has 14 entries, 462 parameters. Pubmed uses 8 output heads averaged; the PPI model uses 6 output heads of 121 outputs each. The figure draws one edge fan per head and the resulting output vector under each rule. At $K = 1$ the two rules coincide.

Every node ranks its neighbors the same way

The paper says its attention closely follows Bahdanau, Cho and Bengio (2015). Their alignment score is $\vec v_a^{T}\tanh(\mathbf W_a \vec s + \mathbf U_a \vec h)$: a nonlinearity applied to the sum of two projections, then a dot product with a learned vector. Equation (3) has the two steps in the other order, a dot product first and the LeakyReLU after it.

Fix a node $i$. Its score for neighbor $j$ is $\mathrm{LeakyReLU}(s_i + t_j)$, and $s_i$ is the same number for every neighbor. LeakyReLU is monotone increasing, so if $t_j > t_k$ then $e_{ij} > e_{ik}$, and the softmax preserves order, so $\alpha_{ij} > \alpha_{ik}$. The comparison never involved $i$. Every node in the graph ranks the nodes it can see by the same single number, $t_j = \vec{\mathbf a}_2^{T}\mathbf W \vec h_j$. What can vary between nodes is how peaked the weights are: $s_i$ shifts every sum toward or away from the kink at zero, which stretches or compresses the gaps between scores. Brody, Alon and Yahav (2022) named this static attention, proved that no choice of $\mathbf W$ and $\vec{\mathbf a}$ escapes it, and showed that restoring Bahdanau's order, $\vec{\mathbf a}^{T}\mathrm{LeakyReLU}(\mathbf W[\vec h_i \,\|\, \vec h_j])$, lets the ranking depend on the query at the same cost. They call the result GATv2.

The figure has two query nodes that share five neighbors. Under the GAT tab, drag either query anywhere and the rank numbers over the two bar charts stay identical; only the bar heights change. Under the GATv2 tab the orders come apart as the queries move.

Training on 140 labels

The transductive experiments (the test nodes and the whole graph are visible during training; only their labels are hidden) follow the Planetoid split of Yang et al. (2016): on Cora, 140 labeled nodes for training, 500 for validation, 1,000 for testing, with every node's features and the full graph visible throughout. The model is two layers, the first with 8 heads of 8 features and an ELU, the second a single head with 7 outputs and a softmax, 92,302 weights in total. Training uses Adam at learning rate 0.005, Glorot initialization (weights drawn with variance 2 / (fan-in + fan-out)), an L2 penalty of 0.0005 on the weights, and early stopping with a patience of 100 epochs on validation loss and accuracy together. Citeseer reuses the Cora settings. Pubmed, with 60 labeled nodes, needed 8 averaged output heads, an L2 penalty of 0.001, and learning rate 0.01.

# the Cora model: 1433 -> 8 heads x 8 -> 7 classes
h1 = concat([head_k(x, adj) for k in range(8)])     # [2708, 64], ELU inside
logits = head_out(h1, adj)                          # [2708, 7], no nonlinearity
loss = cross_entropy(softmax(logits)[train_idx], y[train_idx])   # 140 nodes
loss += 0.0005 * sum(w**2 for w in params)          # L2, all parameters
# backprop reaches W^k, a^k, b^k of both layers through alpha and through Wh

Dropout at $p = 0.6$ is applied to both layers' inputs and also to the normalized attention coefficients themselves. Each training step keeps each $\alpha_{ij}$ with probability 0.4 and scales the survivors by $1/(1 - p) = 2.5$, so a node aggregates a random subset of its neighborhood and the surviving weights no longer sum to one, though their expectation does. The paper describes this as exposing each node to a stochastically sampled neighborhood at every iteration. The code also applies the input dropout a second time to the projected features $\mathbf W \vec h$ before they are summed. At test time nothing is dropped and the weights of (2) are used as computed. Slide $p$ in the figure to the paper's 0.6 and resample a few times; then push it to 0.9, where most steps drop nearly the whole neighborhood.

The inductive experiment (the model is scored on graphs it never saw in training) uses the protein-protein interaction dataset: 24 graphs of human tissues, 56,944 nodes and 818,716 edges in all, 50 features per node, and 121 binary labels per node from gene ontology, so a node can carry several labels. Twenty graphs train, two validate, two test, and the test graphs are never seen during training. The model is three layers: two layers of 4 heads × 256 features with ELU, a skip connection across the middle layer in the style of a residual network, and an output layer of 6 heads × 121 averaged and passed through a sigmoid. With 44,906 training nodes the authors used no dropout and no L2 penalty, and trained on batches of two graphs. The control is the same network with $a(x, y) = 1$, the constant-attention model.

Results and their margins

On the citation graphs the paper reports mean test accuracy over 100 runs. Cora: 83.0 ± 0.7 against 81.5 for the published GCN and 81.4 ± 0.5 for GCN-64*, the authors' own 64-feature GCN with the better of ReLU and ELU. Citeseer: 72.5 ± 0.7 against 70.3 and 70.9 ± 0.5. Pubmed: 79.0 ± 0.3 against 79.0 and 79.0 ± 0.3, a tie. The paper states the margins over GCNs as 1.5% on Cora and 1.6% on Citeseer. Both are percentage points, and each is measured against the best GCN variant in its column: 83.0 − 81.5 on Cora, and 72.5 − 70.9 on Citeseer, where the margin over the published GCN would be 2.2 points. A 1.5-point margin with a run-to-run standard deviation of 0.7 is about two standard deviations.

On PPI, GAT reaches a micro-F1 of 0.973 ± 0.002. The same architecture with constant attention, Const-GAT, reaches 0.934 ± 0.006, so the learned weights account for 3.9 points. The best GraphSAGE the authors could build, a three-layer LSTM aggregator with 512, 512 and 726 features, reaches 0.768, and the published GraphSAGE-LSTM 0.612. Of the 20.5-point gap to GraphSAGE*, 16.6 points are already present in Const-GAT, which shares GAT's 1,024-wide layers, skip connection, and view of the full neighborhood; the paper attributes the gap to observing the entire neighborhood, since GraphSAGE samples a fixed-size subset of it.

The paper also plots a t-SNE (a two-dimensional embedding made for visualization) of the first layer's 64-dimensional features on Cora, colored by class, with edge thickness set to the attention weight averaged over the eight heads. The classes separate into visible clusters. On what the weights mean, the paper says that interpreting them will require domain knowledge about the dataset and leaves it to future work.

Costs and limits

The dense implementation stores an $N \times N$ matrix per head, which is what the memory arithmetic of the masking section priced at 1.56 GB per head on Pubmed. The sparse implementation removes that but, in the authors' framework, handles one graph per batch. The receptive field is bounded by depth, one hop per layer, the same limit a GCN has, and the paper points to skip connections as the way to add layers. Parallelizing across all edges repeats the projection of a node once per edge it touches unless the projection is cached per node, which the code does and a distributed implementation would have to arrange. The layer uses no edge features, and the paper evaluates node classification only; graph classification, edge features, and larger batches are the listed extensions.

The paper's list omits the static ranking shown earlier, in the section on how every node ranks its neighbors. Because the LeakyReLU sits outside the dot product, a trained GAT assigns every node the same neighbor ranking, and any task that needs node $i$ to prefer neighbor $j$ while node $k$ prefers $l$, with $j$ and $l$ both visible to each, is outside what the scoring function can represent. GATv2 changes one line of the layer, applies the nonlinearity before the dot product with $\vec{\mathbf a}$, and keeps the cost at $O(|V| F F' + |E| F')$.