VerifiedarXiv:1704.0121229 min
Architecture · Graph learning

MPNN: Neural Message Passing for Quantum Chemistry

Each atom updates its vector from learned messages sent by the other atoms, and a readout of the final vectors predicts 11 of 13 molecular properties computed by density functional theory to within chemical accuracy.

The paper writes eight earlier graph networks as one recipe of three learned functions (a message, an update and a readout) and searches that recipe for the best model of small organic molecules. The model it finds turns each interatomic distance into a message matrix with a small neural network.

Explaining the paperNeural Message Passing for Quantum ChemistryGilmer, Schoenholz, Riley, Vinyals, Dahl · ICML 2017 · arXiv:1704.01212 ↗

A density functional theory calculation takes about an hour for a molecule with 9 heavy atoms. Trained on 110,462 such calculations, the network is more accurate than every earlier method in the paper's comparison on all 13 properties.

Predicting what a quantum chemistry calculation would output

Quantum mechanics determines a molecule's energy, the shape of its electron cloud and the frequencies it vibrates at, but the equations cannot be solved exactly for anything bigger than a few electrons. Chemists use approximations, and the workhorse is density functional theory (DFT). DFT is still slow: its cost grows as the cube of the number of electrons, and one QM9-sized molecule (up to 9 heavy atoms, meaning atoms other than hydrogen) takes about an hour on one core of a 2.2 GHz Xeon. A molecule with 17 heavy atoms takes up to 8 hours.

The paper treats DFT as a function to be learned. QM9 (Ramakrishnan et al., 2014) is a public dataset of about 134,000 small organic molecules made of hydrogen, carbon, nitrogen, oxygen and fluorine, each with a DFT-optimized 3D geometry and 13 computed properties. The paper uses 130,462 of them, holds out 10,000 for validation and 10,000 for testing, and trains on the remaining 110,462. The 13 properties fall into four groups: four atomization energies (U0U_0, UU, HH, GG: how much energy it takes to pull the molecule apart into atoms, under different temperature and pressure conditions), two vibrational quantities (the highest vibrational frequency ω1\omega_1 and the zero-point vibrational energy, ZPVE), three electron energies (the highest occupied and lowest unoccupied orbital energies, HOMO and LUMO, and the gap between them) and three measures of the electron distribution (the dipole moment μ\mu, the polarizability α\alpha and the electronic spatial extent R2R^2). The thirteenth, the heat capacity CvC_v, appears in every table but is not described in the text. Each property is a separate regression problem.

Errors are reported as an error ratio: the mean absolute error (MAE) on the test set divided by a per-property threshold called chemical accuracy, taken from Faber et al. (2017). For the energies the threshold is 0.043 eV, which is 1 kcal/mol, the precision chemists conventionally need for a computed energy to be useful. An error ratio below 1 means the model predicts the DFT value within chemical accuracy. The best model in the paper has a ratio of 0.45 on U0U_0, which is an average error of 0.45 × 0.043 = 0.019 eV, or 0.45 kcal/mol. The errors are measured against the DFT values, not experiment, so a ratio below 1 says the network reproduces the DFT calculation, errors included.

The paper makes two contributions. It defines a framework, the message passing neural network (MPNN), that covers eight models published before it, and it searches that framework for the model that does best on QM9. The rest of this page follows that order: the input, the framework, the earlier models inside it, the four design changes that produced the best model, its training, and its results.

A molecule as a graph

An MPNN reads a molecule as a graph. Each atom is a node vv with a feature vector xvx_v: its element as a one-hot vector over H, C, N, O and F, its atomic number, whether it accepts or donates electrons, whether it is part of an aromatic ring (a ring whose bonding electrons are shared around the whole ring, as in benzene), its hybridization (sp, sp2 or sp3 one-hot, the shape of its bonding orbitals: linear, planar or tetrahedral), and its number of attached hydrogens, 13 numbers in all (Table 1 of the paper). Each pair of atoms that interacts gets an edge with a feature vector evwe_{vw}. Section 6 compares three choices for the edges:

Hydrogens can be left implicit, as a count in the feature vector of the atom they are attached to, or made into nodes of their own. With explicit hydrogens a QM9 molecule has up to 29 nodes instead of 9, and the paper reports that training slows by a factor of roughly 10. Figure 1 shows all six combinations on pyridine, a benzene ring with one carbon replaced by nitrogen.

Figure 1 · pyridine under the three edge representations
Ring bonds are aromatic (teal), C–H bonds single (gray), and dashed lines are unbonded pairs, colored by distance bin. Click an atom to make it v and another to make it w; the panel prints the atom's 13 features and the edge vector between the two. Geometry: a regular hexagon with 1.39 Å sides and 1.08 Å C–H bonds, within a few hundredths of an angstrom of the real molecule.

The header line of Figure 1 counts directed edges, which is also the number of messages computed in each step of the model. Pyridine with implicit hydrogens has 6 bonds, so 12 messages per step on the chemical graph and 30 with an edge between every ordered pair. With explicit hydrogens it has 11 nodes and 110 ordered pairs. For the largest QM9 molecules the dense representation means 29 × 28 = 812 messages per step against 9 × 8 = 72 with implicit hydrogens, a factor of 11.3, consistent with the roughly tenfold slowdown the paper reports.

A graph has no natural order for its nodes. The same pyridine molecule can be stored with the nitrogen as atom 0 or as atom 4, and a model that gave different predictions for the two listings would be predicting a property of the file format. Earlier work had handled this in the input. The Coulomb matrix of Rupp et al. (2012), a matrix of pairwise nuclear repulsions, changes when the atoms are renumbered, so a model built on it has to learn the invariance from augmented data. The symmetry functions of Behler and Parrinello (2007) are invariant by construction but are hand-designed, and the paper notes they struggle with more than three elements. An MPNN gets the invariance from its structure, as the next section shows.

How a message passing neural network works

Every atom carries a hidden state hvth_v^t, a vector of dd numbers (the released code defaults to d=50d = 50; the towers experiments use 200). The initial state hv0h_v^0 is the feature vector xvx_v, padded with zeros up to length dd. The forward pass then runs TT steps of message passing. In step tt each atom collects a message from each of its neighbors N(v)N(v) and sums them, then updates its state from the old state and the summed message:

mvt+1=∑w∈N(v)Mt(hvt, hwt, evw)hvt+1=Ut(hvt, mvt+1)m_v^{t+1} = \sum_{w \in N(v)} M_t\big(h_v^t,\, h_w^t,\, e_{vw}\big) \qquad h_v^{t+1} = U_t\big(h_v^t,\, m_v^{t+1}\big)mvt+1=∑w∈N(v)Mt(hvt, hwt, evw)hvt+1=Ut(hvt, mvt+1)\begin{gathered} m_v^{t+1} = \sum_{w \in N(v)} M_t\big(h_v^t,\, h_w^t,\, e_{vw}\big) \\ h_v^{t+1} = U_t\big(h_v^t,\, m_v^{t+1}\big) \end{gathered}
(1), (2)

MtM_t is the message function and UtU_t the update function, both learned. After TT steps a readout function RR, also learned, maps the set of final states to the prediction:

y^=R({hvT∣v∈G})\hat y = R\big(\{h_v^T \mid v \in G\}\big)
(3)

Here is one step with d=2d = 2 and made-up numbers. The pyridine nitrogen has two neighbors on the chemical graph, both joined by aromatic bonds, with states hC1=(0.5, 1.0)h_{C_1} = (0.5,\ 1.0) and hC5=(0.2, 0.4)h_{C_5} = (0.2,\ 0.4). Take the message function of the gated graph neural network discussed below, which multiplies the neighbor's state by a learned matrix chosen by the bond type, and suppose the aromatic matrix is

Aarom=(0.500.51)A_{\text{arom}} = \begin{pmatrix} 0.5 & 0 \\ 0.5 & 1 \end{pmatrix}

Then the two messages are (0.25, 1.25)(0.25,\ 1.25) and (0.10, 0.50)(0.10,\ 0.50), and their sum is mN=(0.35, 1.75)m_N = (0.35,\ 1.75). Swap which carbon is called C1C_1 and which C5C_5, and the sum does not change. The update in the section on the GRU turns this mNm_N and the old state hN=(1,0)h_N = (1, 0) into a new state.

Three properties follow from the form of (1) to (3). First, renumbering the atoms permutes the states without changing any of them, because each message depends only on the two atoms and their edge, and the sum in (1) ignores order; if RR also ignores order, the prediction is invariant to atom numbering, which the paper requires. Second, the same MtM_t, UtU_t and RR apply to a molecule of any size, so one set of weights covers all of QM9. Third, after tt steps the state of atom uu can depend on the input of atom ss only if uu is within tt edges of ss: each step moves information one edge further. Figure 2 shows the third property on 2-phenylethanol, a QM9-sized molecule with a benzene ring and a two-carbon chain ending in OH.

Figure 2 · how far information travels in t steps
t = 0
Heavy atoms of 2-phenylethanol on the chemical graph. Atoms whose state depends on the source atom's input light up teal; the small numbers count the steps needed. Press Play or drag the slider through t = 0 to 8, click an atom to make it the source, and switch the graph to add virtual edges or a master node M.

With the oxygen as source and bonds only, the atom at the far side of the ring is 6 bonds away, so a model with T=3T = 3 cannot let the oxygen's features affect that atom's state. The readout can still combine information from both ends, because it takes every final state as input. The two other graph options in Figure 2 are the paper's fixes for long distances, covered in their own section below: virtual edges make every atom one step from every other at the price of 72 messages per step instead of 18, and a master node makes every atom at most two steps from every other for 36.

Eight earlier models written as message passing

Section 2 of the paper rewrites eight published models by naming their MtM_t, UtU_t and RR. The exercise makes the models comparable piece by piece. Five of them are described below. The other three are the molecular graph convolutions of Kearnes et al. (2016), the only one of the eight that keeps a hidden state on each edge and updates it alongside the node states, and the spectral graph convolutions of Bruna et al. (2013) and Defferrard et al. (2016), whose messages are matrices built from eigenvectors of the graph Laplacian (Kipf and Welling's model, below, is their simplified special case).

Molecular fingerprints (Duvenaud et al., 2015). The message is the concatenation (hw,evw)(h_w, e_{vw}), and the update is σ(Htdeg⁡(v)mvt+1)\sigma(H_t^{\deg(v)} m_v^{t+1}), a sigmoid of a matrix chosen by the atom's degree. Summing concatenations gives (∑whw, ∑wevw)\big(\sum_w h_w,\ \sum_w e_{vw}\big), the sum of neighbor states next to the sum of edge features. The summed message keeps no record of which neighbor came with which bond: a carbon bonded to an oxygen by a double bond and to a nitrogen by a single bond receives the same message as one bonded to the oxygen by a single bond and to the nitrogen by a double bond. The paper names this as the reason the model cannot capture correlations between edges and nodes.

Gated graph neural networks (GG-NN, Li et al., 2016). The message is AevwhwtA_{e_{vw}} h_w^t, a learned matrix per discrete edge type times the neighbor's state. The update is a gated recurrent unit (GRU), the same at every step. The readout is

R=∑v∈Vσ(i(hvT,hv0))⊙j(hvT)R = \sum_{v \in V} \sigma\big(i(h_v^{T}, h_v^{0})\big) \odot j\big(h_v^{T}\big)
(4)

where ii and jj are neural networks, σ\sigma is the logistic sigmoid and ⊙\odot is elementwise multiplication. The sigmoid term acts as a per-atom, per-channel gate that sets how much of each atom's contribution jj enters the sum. GG-NN is the paper's starting point, and every model in its experiments keeps the GRU update.

Interaction networks (Battaglia et al., 2016). The message is a neural network applied to the concatenation (hv,hw,evw)(h_v, h_w, e_{vw}), so it depends on both endpoints; the update is a neural network on (hv,xv,mv)(h_v, x_v, m_v); the readout is a network applied to ∑vhvT\sum_v h_v^T. The original work used a single step, T=1T = 1.

Deep tensor neural networks (DTNN, Schütt et al., 2017). The message multiplies a projection of the neighbor's state elementwise by a projection of the edge (an expansion of the interatomic distance), and the update adds it: hvt+1=hvt+mvt+1h_v^{t+1} = h_v^t + m_v^{t+1}. The readout passes each atom through a one-hidden-layer network and sums the outputs, so the molecule's energy is a sum of atomic contributions.

Graph convolutional networks (Kipf and Welling, 2016). The message is cvwhwtc_{vw} h_w^t with a scalar weight cvw=(deg⁡(v)deg⁡(w))−1/2Avwc_{vw} = (\deg(v)\deg(w))^{-1/2} A_{vw}, and the update is ReLU(Wtmvt+1)\mathrm{ReLU}(W^t m_v^{t+1}). Appendix 10.1.1 derives this from the layer σ(D~−1/2A~D~−1/2HW)\sigma(\tilde D^{-1/2} \tilde A \tilde D^{-1/2} H W), where A~\tilde A is the adjacency matrix with self-loops added, so the degrees there count the self-loop. Every edge gets one scalar, so the message is a weighted sum of neighbor states and cannot use edge features. The graph convolutional networks explainer covers that layer.

The edge network message function

GG-NN's message, AevwhwA_{e_{vw}} h_w, needs a finite set of edge types, one learned d×dd \times d matrix each. To feed it distances the paper uses the 14-symbol alphabet of Section 6, so the model learns 14 matrices, and every unbonded pair whose distance falls in the same 0.5 Å bin gets the same matrix. Why not just enlarge the alphabet? Finer bins mean more matrices, each trained on fewer pairs, and the model still cannot use the fact that 2.49 Å and 2.51 Å are almost the same distance. The paper's answer, the edge network, makes the matrix a function of the edge vector:

M(hv,hw,evw)=A(evw) hwM\big(h_v, h_w, e_{vw}\big) = A(e_{vw})\, h_w

where AA is a neural network that takes the 5-dimensional raw edge vector (distance and bond one-hot) and outputs d2d^2 numbers, reshaped into a d×dd \times d matrix. In the released code it has four hidden layers of 50 ReLU units; at d=50d = 50 its output layer has 2,500 units and the whole network 135,450 weights. Figure 3 compares the two message functions for a single unbonded pair as the distance between the atoms changes, with d=2d = 2 so the matrix has four entries.

Figure 3 · a message matrix per distance bin, or a function of distance
2.40 Å
The four entries of a toy 2 × 2 message matrix for two unbonded atoms, against their distance; the gray vertical lines are the paper's bin edges. Drag the distance from 0.8 to 7.5 Å and switch between the two message functions; the selected one is drawn solid and the other faint. The functions are hand-written stand-ins for a trained edge network; the binned matrices are their average over each bin.

With distance bins, moving from 2.51 Å to 2.99 Å leaves the matrix and the message unchanged, and crossing 3.00 Å changes them in one jump. With the edge network the message changes continuously, and two distances that share a bin still get different messages. In the paper's Table 10, switching from the chemical graph with matrix multiplication to raw distances with the edge network lowers the average error ratio from 2.57 to 0.98 (part of that gap is an input difference, flagged in the Provenance panel), and every one of the 13 targets improves.

The code builds the edge network's matrices once per molecule and reuses them at every step, because the edges do not change between steps. The cost is in the products: with an edge between every pair, one step multiplies n(n−1)n(n-1) different d×dd \times d matrices by vectors, which is the O(n2d2)O(n^2 d^2) per step the paper cites as the main scalability problem. The undirected molecule is also treated as a directed graph with an incoming and an outgoing copy of every edge, each with its own edge network, and the two summed messages are concatenated, so mvm_v has 2d2d entries. The direction carries no chemical meaning here; the paper uses it only for parameter tying, and it doubles the parameters.

The paper also tried a pair message, a neural network of (hw,hv,evw)(h_w, h_v, e_{vw}) as in interaction networks, on the reasoning that a message which also depends on the receiving atom might use the message channel better. It did worse: trained jointly on all 13 targets, its average error ratio was 3.98 against 1.53 for the edge network, which won on 11 of the 13 targets. It was also harder to train, and the authors did not pursue it.

The GRU update and weight tying

Every model in the experiments updates states with the GRU of GG-NN, a gated recurrent unit (introduced by Cho et al., 2014, and covered in the RNN encoder-decoder explainer) whose input is the summed message and whose hidden state is the atom's state. In the convention of GG-NN and the released code,

z=σ(Wzmv+Uzhv),r=σ(Wrmv+Urhv)h~=tanh⁡(Wmv+U(r⊙hv))hv′=(1−z)⊙hv+z⊙h~\begin{aligned} z &= \sigma(W_z m_v + U_z h_v), \qquad r = \sigma(W_r m_v + U_r h_v) \\ \tilde h &= \tanh\big(W m_v + U (r \odot h_v)\big) \\ h_v' &= (1 - z) \odot h_v + z \odot \tilde h \end{aligned}z=σ(Wzmv+Uzhv),r=σ(Wrmv+Urhv)h~=tanh⁡(Wmv+U(r⊙hv))hv′=(1−z)⊙hv+z⊙h~\begin{aligned} z &= \sigma(W_z m_v + U_z h_v), \\ r &= \sigma(W_r m_v + U_r h_v) \\ \tilde h &= \tanh\big(W m_v + U (r \odot h_v)\big) \\ h_v' &= (1 - z) \odot h_v + z \odot \tilde h \end{aligned}

The update gate zz sets, coordinate by coordinate, how much of the old state to replace with the candidate h~\tilde h, and the reset gate rr sets how much of the old state the candidate is computed from. With zz near 0 an atom keeps its state, so the depth of the computation does not force every step to overwrite what earlier steps found. PyTorch's nn.GRUCell uses the opposite convention for zz (its zz weights the old state), which matters only when porting weights.

Continuing the worked example with the simplest weights (Wz=W=IW_z = W = I and every UU zero, so rr plays no role), the gate and the candidate are

z=σ(0.35, 1.75)=(0.587, 0.852),h~=tanh⁡(0.35, 1.75)=(0.336, 0.941)z = \sigma(0.35,\ 1.75) = (0.587,\ 0.852), \qquad \tilde h = \tanh(0.35,\ 1.75) = (0.336,\ 0.941)z=σ(0.35, 1.75)=(0.587, 0.852),h~=tanh⁡(0.35, 1.75)=(0.336, 0.941)\begin{gathered} z = \sigma(0.35,\ 1.75) = (0.587,\ 0.852), \\ \tilde h = \tanh(0.35,\ 1.75) = (0.336,\ 0.941) \end{gathered}

and the new state of the nitrogen is

hN′=(0.413⋅1+0.587⋅0.336,  0.148⋅0+0.852⋅0.941)=(0.610, 0.802)h_N' = (0.413 \cdot 1 + 0.587 \cdot 0.336,\ \ 0.148 \cdot 0 + 0.852 \cdot 0.941) = (0.610,\ 0.802)hN′=(0.413⋅1+0.587⋅0.336,  0.148⋅0+0.852⋅0.941)=(0.610, 0.802)\begin{aligned} h_N' &= (0.413 \cdot 1 + 0.587 \cdot 0.336,\ \ 0.148 \cdot 0 + 0.852 \cdot 0.941) \\ &= (0.610,\ 0.802) \end{aligned}

The first coordinate, where the message was small, kept 41% of its old value; the second, where the message was large, was almost entirely replaced.

All experiments tie the weights across steps: one MM and one UU serve every tt, so the model is a recurrent network unrolled TT times over the graph. The authors tried separate weights per step in preliminary runs and found that tying the weights and spending the parameters on a larger dd worked better.

Long-range information: virtual edges, a master node and set2set

On the chemical graph, without distances, an atom's state depends only on atoms within TT bonds, and properties such as the dipole moment depend on where charge sits across the whole molecule. The paper tests three ways to let information travel further, each added to a GG-NN trained on the chemical graph.

Virtual edges. Add an edge of a new type between every pair of atoms that is not bonded. This is a preprocessing step and makes every pair one step apart, as the "+ virtual edges" option of Figure 2 shows.

A master node. Add one extra node, connected to every atom by an edge of its own type, with its own state size dmasterd_{\text{master}} and its own GRU weights. Each step every atom writes to it and reads from it, so any two atoms are at most two steps apart. The paper gives the cost as O(∣E∣d2+n dmaster2)O(|E|d^2 + n\, d_{\text{master}}^2) and argues the master node can be made large without much extra compute.

The set2set readout. Replace the gated sum of equation (4) with set2set (Vinyals et al., 2016), a readout designed for sets. Each final state is concatenated with the atom's input features and linearly projected, giving a set of memories mim_i. An LSTM with no input then runs MM steps; at each step it emits a query, attends over the memories and appends what it read to its state:

qt=LSTM(qt−1∗),ei,t=f(mi,qt)ai,t=exp⁡(ei,t)∑jexp⁡(ej,t),rt=∑iai,t mi,qt∗=[qt, rt]\begin{aligned} q_t &= \mathrm{LSTM}(q^{*}_{t-1}), \qquad e_{i,t} = f(m_i, q_t) \\ a_{i,t} &= \frac{\exp(e_{i,t})}{\sum_j \exp(e_{j,t})}, \qquad r_t = \sum_i a_{i,t}\, m_i, \qquad q^{*}_t = [q_t,\ r_t] \end{aligned}qt=LSTM(qt−1∗),ei,t=f(mi,qt)ai,t=exp⁡(ei,t)∑jexp⁡(ej,t),rt=∑iai,t mi,qt∗=[qt, rt]\begin{aligned} q_t &= \mathrm{LSTM}(q^{*}_{t-1}), \\ e_{i,t} &= f(m_i, q_t) \\ a_{i,t} &= \frac{\exp(e_{i,t})}{\sum_j \exp(e_{j,t})}, \\ r_t &= \sum_i a_{i,t}\, m_i, \qquad q^{*}_t = [q_t,\ r_t] \end{aligned}

The score ff is a scalar function of a memory and the query (in the released code a small additive attention, v⊤tanh⁡(q+m)v^\top \tanh(q + m)). The read rtr_t is an attention-weighted sum, so it ignores the order of the atoms, and the final qM∗q^*_M goes through a neural network to the prediction. Where the gated sum adds up per-atom contributions in one pass, set2set reads the set MM times, and each query can depend on what the earlier reads returned. The search chose MM between 1 and 12. The attention is the same content-based weighting as in Bahdanau attention, applied to atoms instead of source words.

Table 3 of the paper averages the error ratio over the 13 targets for each modification: 3.47 for plain GG-NN, 2.90 with virtual edges, 2.62 with a master node and 2.57 with set2set. The set2set model reaches chemical accuracy on 5 targets, the four atomization energies and ω1\omega_1, using the chemical graph alone. The paper says all three modifications help on all 13 targets; its own per-target Table 7 shows two exceptions (virtual edges raise the R2R^2 error from 14.74 to 17.11, and the master node raises μ\mu from 3.94 to 4.02). These runs, and the towers runs below, used a partial-charge input feature that is itself a DFT output and would not be available in practice; the headline results of Table 2 do not use it.

Towers: wider states for less compute

A step with an edge between every pair costs O(n2d2)O(n^2 d^2), and the paper wants larger dd for larger molecules. The towers variant splits each atom's dd-dimensional state into kk pieces of d/kd/k each, runs a separate message and update function on each piece, and then mixes the pieces with a network gg shared by all atoms:

(hvt,1,hvt,2,…,hvt,k)=g(h~vt,1,h~vt,2,…,h~vt,k)\big(h_v^{t,1}, h_v^{t,2}, \ldots, h_v^{t,k}\big) = g\big(\tilde h_v^{t,1}, \tilde h_v^{t,2}, \ldots, \tilde h_v^{t,k}\big)(hvt,1,hvt,2,…,hvt,k)=g(h~vt,1,h~vt,2,…,h~vt,k)\begin{aligned} &\big(h_v^{t,1}, h_v^{t,2}, \ldots, h_v^{t,k}\big) \\ &\quad = g\big(\tilde h_v^{t,1}, \tilde h_v^{t,2}, \ldots, \tilde h_v^{t,k}\big) \end{aligned}
(5)

Each tower costs O(n2(d/k)2)O(n^2 (d/k)^2), so the kk towers together cost O(n2d2/k)O(n^2 d^2 / k) plus the mixing network, which is per atom and does not grow with n2n^2. At d=200d = 200 and k=8k = 8 each tower has 25 dimensions, and the message products take 8 times fewer multiplications than one tower of 200. The measured inference speedup on 9-atom molecules was a factor of 2, since the mixing network and everything outside message passing still cost the same. This is the same block-diagonal idea as grouped convolutions, with gg restoring communication between groups.

The towers also generalized better. With distance bins and no explicit hydrogens, the 8-tower model beat a tuned single-tower GG-NN with set2set on 12 of 13 targets both when one model was trained per target (average 1.37 against 1.53) and when one model was trained on all targets (1.75 against 1.92). The authors suggest towers behave somewhat like an ensemble. Combining towers with the edge network did not improve results in their attempts, so the best model does not use towers.

Inputs and training

The best model, called enn-s2s, combines the edge network message, the GRU update with tied weights, the set2set readout, raw distances on every pair of atoms, and explicit hydrogens. Table 10 compares inputs with set2set and the GRU fixed (the model without distances uses matrix multiplication, the other two the edge network): the average error ratio is 2.57 on the chemical graph, 0.98 with distances, and 0.68 with distances and explicit hydrogens. The hydrogens matter most for μ\mu (0.95 to 0.30) and R2R^2 (0.57 to 0.15), both measures of how charge spreads through space.

Every combination of model and target was tuned with 50 random hyperparameter trials: the number of steps TT between 3 and 8, the set2set steps MM between 1 and 12, an initial learning rate between 1e-5 and 5e-4, and a linear decay that starts between 10% and 90% of the way through training and ends at a factor between 0.01 and 1 of the initial rate. Training uses Adam with batches of 20 molecules for 3 million steps, which is 3,000,000 × 20 / 110,462 ≈ 543 passes over the training set. Each target is normalized to mean 0 and variance 1; the loss is mean squared error, the reported metric mean absolute error; the validation set picks the stopping point and the best trial, and scores are reported on the test set. One model per target consistently beat one model trained on all 13 targets, by up to 40% on some targets.

One forward pass in code

The listing traces enn-s2s on one molecule in array notation, with the default sizes of the released model code (the paper's numbers come from per-target searches over these sizes). The edge networks output a full 50 × 50 matrix for each of the 19 × 19 atom pairs, and the einsum sums A(evw)hwA(e_{vw}) h_w over the neighbors ww.

# enn-s2s forward pass on one molecule, default sizes of brain-research/mpnn
# 2-phenylethanol with explicit hydrogens: n = 19 atoms, d = 50, T = 6
x = atom_features(mol)              # [19, 13]     Table 1 features
e = edge_vectors(mol)               # [19, 19, 5]  (distance, bond one-hot)
h = pad(x, 50)                      # [19, 50]     h^0: x padded with zeros

# the edge networks run once per molecule; their output is reused at every step
A_in  = edge_net_in(e).reshape(19, 19, 50, 50)    # 4 hidden layers of 50 units
A_out = edge_net_out(e).reshape(19, 19, 50, 50)

for t in range(6):                  # the same weights at every step
    m_in  = einsum("vwij,wj->vi", A_in, h)        # sum over w of A(e_vw) h_w
    m_out = einsum("vwij,wj->vi", A_out, h)
    m = concat([m_in, m_out], -1) + b             # [19, 100]   eq (1)
    h = gru(h, m)                                 # [19, 50]    eq (2)

q = set2set(proj(concat([h, x], -1)), steps=12)   # [126], any atom order
y_hat = mlp(q)                                    # one normalized target, e.g. U0
loss = (y_hat - (y - mean_y) / std_y) ** 2        # train on MSE, report MAE
# gradients reach both edge networks, b, the GRU, proj, the set2set LSTM and
# the MLP; no weight belongs to one atom, so the same model runs on any molecule

The released repository, brain-research/mpnn, contains only these model classes and their tests, not the QM9 reader or the training loop. Its model classes expect a batch of molecules padded to the size of the largest, with a mask that zeroes the padded atoms after each GRU update and excludes them from the readout. Two details in it differ from the paper's equations and are listed in the Provenance panel: the GG-NN readout feeds the atom features xvx_v to both networks of equation (4) and leaves out the tanh on jj, and the 19 × 19 pairs include each atom with itself.

Results on QM9

Table 2 compares enn-s2s with five hand-engineered molecular representations evaluated by Faber et al. (2017) (the Coulomb matrix CM, bag of bonds BOB, BAML, the ECFP4 fingerprint and HDAD) and two MPNN baselines (the graph convolutions of Kearnes et al. and GG-NN with distance bins). Figure 4 plots the error ratios on a log axis.

Figure 4 · QM9 error ratios (Tables 2, 6, 7 and 10)
U0
Error ratio is test MAE divided by the chemical-accuracy threshold; the shaded band left of 1 is within chemical accuracy. Switch views with the tabs, and click a row (or use the slider) to print each value with its MAE in physical units. "Best earlier" is the lowest of the seven baselines for that target. The averages of the training-size view are ours.

The enn-s2s model has the lowest error of all methods on every target, with an average error ratio of 0.68 against 1.35 for HDAD and 1.36 for GG-NN, and it is within chemical accuracy on 11 of 13. The two misses are the HOMO-LUMO gap (1.60, an MAE of 0.069 eV) and ZPVE (1.27). HOMO, at 0.99, is inside the threshold by 1%. Averaging the five models with the lowest validation error lowers the average to 0.52, and lowers every target, but the gap (1.23) and ZPVE (1.10) still miss.

The largest gains are on targets that depend on the spatial arrangement of electrons: R2R^2 falls from 0.80 (bag of bonds) to 0.15, and μ\mu from 0.70 (Kearnes et al.) to 0.30. The ECFP4 fingerprint, which encodes substructures but no geometry, has error ratios between 30 and 242 on 8 of the 13 targets. Table 6 of the paper trains enn-s2s on smaller subsets: with 11,000 molecules, a tenth of the training set, it already matches or beats the best earlier method trained on all 110,000 for R2R^2 (0.28 against 0.80) and comes within 0.01 of it for ω1\omega_1 (0.28 against 0.27); the paper calls both equal or better. The errors mostly fall as the training set grows, with a few exceptions such as ω1\omega_1, which is 0.15 at 82,000 molecules and 0.19 at 110,000.

DTNN appears in Table 2 with a single number, 0.84, taken from its own paper and trained on a different split of 100,000 molecules; Table 2 prints it in the GG row, but Schütt et al. report it for the total energy, the quantity in the U0U_0 row, where enn-s2s scores 0.45.

Limits and later work

The benchmark is narrow by design. QM9 molecules have at most 9 heavy atoms, the labels are DFT values, and the test molecules come from the same distribution as the training molecules. Table 5 of the paper puts the DFT error against experiment above the chemical-accuracy threshold for 11 of the 12 targets that have an estimate (dipole moment is the exception, at 0.1 against 0.1), so the network now matches DFT more closely than DFT matches nature. The abstract concludes that future work should move to larger molecules or more accurate labels.

The best model needs a geometry. Its edge vectors are distances from the DFT-optimized structure in QM9, so applying it to a new molecule requires a 3D structure from some other method first. Without distances the best model in the paper is within chemical accuracy on 5 targets, not 11.

Cost grows with the square of the molecule size. With distance edges every atom receives a message from every other atom, and each message is a d×dd \times d matrix product. The conclusion names generalization to larger molecules as the main open problem, for two reasons: the distribution of distances changes with size, and the number of summed messages grows with the number of atoms. It proposes attention over incoming messages, which graph attention networks introduced later the same year for citation and protein graphs. GraphSAGE, published two months after this paper, reduced the cost by sampling a fixed number of neighbors per node.

Message, aggregate and update became the standard way to describe graph neural networks, and the edge network's idea of generating interaction weights from distances continued in SchNet (Schütt et al., 2017), whose convolution filters are a learned function of interatomic distance. Battaglia et al. (2018) extended the framework with explicit edge states and a global state, the roles that Kearnes et al.'s edge updates and the master node play here. Xu et al. (2019) proved that a message passing network cannot tell apart two graphs that the Weisfeiler-Lehman graph isomorphism test cannot tell apart, which bounds what any model inside this framework can distinguish on a bond graph without distances.

Provenance Verified against primary literatureHow we verify
Gilmer et al. (2017), v2Equations (1) to (3) and the eight models of Section 2 (equation (4) is GG-NN's readout); Section 5: hidden size d, separate incoming and outgoing message channels (2d), h^0 = x padded to d, weight tying, GRU update; 5.1 matrix multiplication, edge network and pair message; 5.2 virtual edges and the master node with cost O(|E|d^2 + n d_master^2); 5.3 set2set; 5.4 towers, equation (5), O(n^2 d^2 / k) and the measured 2x at k = 8, n = 9, d = 200; Section 6 and Table 1 (atom features, 10 distance bins, an alphabet of 14, 5-dimensional raw edge vectors, explicit hydrogens up to 29 nodes and about 10x slower training); Section 7 (50 random trials per model and target, 3 <= T <= 8, 1 <= M <= 12, Adam, batch 20, 3 million steps, learning rate 1e-5 to 5e-4 with linear decay by a factor in [0.01, 1], 10,000 validation and 10,000 test molecules out of 130,462, targets normalized, MSE trained and MAE reported); Section 8 and Tables 2 to 4, including the partial-charge note; Appendix 10.1.1 (Kipf and Welling as an MPNN), Table 5 (chemical accuracy and DFT error), Tables 6 to 10.
mpnn code: mpnn.pybrain-research/mpnn, mpnn.py. message_pass, lines 97 to 129: the incoming and outgoing products are concatenated into a 2d message and a bias is added. EdgeNetwork, 254 to 394: the edge vector is a one-hot edge class concatenated with the distance; feed_forward_nn maps it to node_dim^2 numbers reshaped to a d x d matrix, with separate networks for incoming and outgoing edges, built once per molecule and reused at every step. GRUUpdate, 400 to 481: h_t = (1 - z) * h + z * h_tilde, with the 2d message as input. GraphLevelOutput, 538 to 600: i and j both take [h^T, x], j has no tanh ("For regression tasks identity is VERY important here"), summed over atoms. Set2VecOutput, 644 to 687: a 1x1 projection, set2vec, then a feed-forward network. default_hparams, 725 to 751: node_dim 50, 6 propagation steps, 12 set2set steps, edge network of 4 hidden layers of 50, output hidden size 200, batch 20, learning rate 1.3e-4, gradient-norm clip 4. MPNN._fprop, 826 to 874: x padded to node_dim, readout on concat(h^T, x). The edge network and message_pass compute a matrix for every ordered pair of the n x n grid, including each atom with itself.
mpnn code: set2vec, READMEbrain-research/mpnn, set2vec.py 166 to 205: an LSTM with no input emits a query each step; the default attention score is v^T tanh(query + memory), the "dot" option a dot product; the read is the attention-weighted sum, concatenated to the LSTM output. README: the repository holds only the model classes and tests; the paper's numbers came from a separate 50-trial hyperparameter search per target.
Li et al. (2016)Li, Tarlow, Brockschmidt & Zemel, Gated Graph Sequence Neural Networks, Section 3: h^(1) = [x, 0] (eq 1), messages through per-edge-type matrices in both directions (eq 2), the GRU-like update of eqs (3) to (6) with h = (1 - z) h_old + z h_tilde, and the graph readout of eq (7), tanh(sum_v sigma(i(h^T, x)) * tanh(j(h^T, x))), with the note that both tanh can be replaced by the identity.
Vinyals et al. (2016)Vinyals, Bengio & Kudlur, Order Matters: Sequence to Sequence for Sets, equations (3) to (7): q_t = LSTM(q*_{t-1}), e_{i,t} = f(m_i, q_t), a_{i,t} = softmax, r_t = sum_i a_{i,t} m_i, q*_t = [q_t, r_t]; the final state is invariant to the order of the memories.
Schütt et al. (2017)Schütt, Arbabzadah, Chmiela, Müller & Tkatchenko, Deep Tensor Neural Networks, Table II: GDB-9 with 100k training molecules and T = 3 interaction passes, mean absolute error 0.84 kcal/mol on the molecular total energy.
Our recomputationTraining set 130,462 - 20,000 = 110,462 molecules; 3,000,000 steps x 20 / 110,462 = 543 epochs (the paper says about 540). Chemical accuracy counts from Table 2: enn-s2s is below 1 on 11 targets and above it on gap (1.60) and ZPVE (1.27), and beats the best earlier method on all 13. Table 7: set2set is below GG-NN on all 13, virtual edges are above it on R2 and the master node on mu. Table 8: towers win 12 of 13 in each regime (losing R2 jointly, mu individually). Table 9: edge network wins 11 of 13. MAE values in Figure 4 are error ratio times the Table 5 threshold. Pair counts: 9 x 8 = 72 and 29 x 28 = 812 ordered pairs, a ratio of 11.3. Edge network weights at the code defaults with a 5-dimensional input: 300 + 3 x 2,550 + 127,500 = 135,450 per direction. Table 6 averages in Figure 4 are ours (the paper prints none).
correctionFour slips in the paper, each checked against its own tables or the cited work. (a) Section 4 defines the electron energy gap as ε_HOMO − ε_LUMO, which is negative for every molecule; the gap is ε_LUMO − ε_HOMO, as Appendix 10.2 says. (b) Section 8 says the virtual edge, master node and set2set modifications "help on all 13 targets"; in Table 7 virtual edges make R2 worse (17.11 against 14.74) and the master node makes mu worse (4.02 against 3.94); only set2set improves all 13. (c) Table 2 prints DTNN's 0.84 in the G row; Schütt et al. report 0.84 kcal/mol for the molecular total energy with no thermal correction, the quantity of the U0 row, on a 100k-molecule training split. (d) Table 10's no-distance column repeats Table 7's set2set column, which Section 8 says was run with a DFT-computed partial-charge input, while the other two columns match Table 2, run without it, so part of the 2.57 to 0.98 step compares different inputs. Separately, the released model code differs from equation (4): its j network reads the concatenation of h^T and x, not h^T alone, and has no tanh on its output.

Questions you might still have

?

Is an MPNN the same thing as a graph neural network?
Most graph neural networks in use fit the MPNN form: a per-edge message, a sum or other order-independent combination at each node, a per-node update, and for graph-level tasks an order-independent readout. The paper did not invent the form. It named it, showed that eight published models are special cases, and searched inside it. Graph attention networks and GraphSAGE, both from later in 2017, also fit it; the explainers on this site for each walk through their message and aggregation choices.

?

Why does equation (1) sum the messages instead of averaging them?
The paper does not discuss the choice. A sum keeps the number of neighbors in the result: a carbon with four neighbors receives four messages, a carbon with two receives two, and an average would erase that difference. Any order-independent combination keeps the model independent of atom numbering; the sum is the simplest one that also preserves counts. Xu and colleagues (2019) later proved that a sum can map different multisets of neighbor features to different results where a mean or a max cannot: a mean gives the same answer for one neighbor of a kind as for two of the same kind.

?

Does the best model need a 3D geometry for every molecule it predicts?
Yes. The edge network reads interatomic distances, and in QM9 those come from the same DFT optimization that produced the labels. Without distances the best average error ratio is 2.57 instead of 0.68 (Table 10). In an application the geometry would have to come from a cheaper method, which adds its own error that the paper does not measure. The graph-only models reach chemical accuracy on 5 of 13 targets, all five of them energies or the highest vibrational frequency.

?

Why train a separate model for each of the 13 targets?
Because it measured better: one model per target beat a single model trained on all 13, by up to 40% on some targets. In Table 4 the joint GG-NN baseline averages 1.92 against 1.53 when trained per target, and the towers model 1.75 against 1.37. Each model and target pair also got its own 50-trial hyperparameter search, so the headline numbers come from 13 separately tuned networks.

?

How many message-passing steps does a molecule need?
The search chose T between 3 and 8, and the paper reports that any T of at least 3 works. On the bond graph a QM9 molecule with 9 heavy atoms is at most 8 bonds across, so 3 steps do not connect every pair. The best models use distance edges between every pair of atoms, where one step already connects every atom to every other, and extra steps let information combine over several hops through intermediate atoms.

?

What does "300,000 times faster than DFT" compare?
A DFT calculation for one 9-heavy-atom molecule takes about an hour on one core of a 2.2 GHz Xeon E5-2660 with Gaussian 09, and up to 8 hours at 17 heavy atoms. The footnote says the networks discussed are 300,000 times faster at inference, which works out to about 12 milliseconds per molecule against the one-hour figure. The paper does not say what hardware ran the network or whether molecules were batched.

?

Does the model work on molecules larger than the 9 heavy atoms of QM9?
The paper does not test it and lists it as the main open direction. It names two problems: the distribution of pairwise distances depends on the number of atoms, and with distance edges every atom receives one message from every other atom, so the size of the summed message grows with the molecule. It suggests attention over the incoming messages as a fix, the mechanism that graph attention networks introduced a few months later.

?

Is chemical accuracy against DFT the same as chemical accuracy against experiment?
No. The labels are DFT outputs, and DFT itself has errors relative to nature that are larger than the chemical-accuracy threshold for most targets: Table 5 lists an estimated DFT error of 0.1 eV for the atomization energies against a threshold of 0.043 eV, and 2.0 eV for the HOMO energy. A model that matches DFT within 0.043 eV can still be 0.1 eV away from an experiment. The paper says this, and argues the next benchmark should have more accurate labels or larger molecules.

Footnotes & further reading

  1. The paper: Gilmer, Schoenholz, Riley, Vinyals and Dahl, Neural Message Passing for Quantum Chemistry (ICML 2017; this page follows arXiv v2). Model code: brain-research/mpnn (archived; model classes only, line numbers in the Provenance panel refer to its master branch).
  2. The dataset: Ramakrishnan, Dral, Rupp and von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules (Scientific Data, 2014). The thresholds and baselines: Faber et al., Fast machine learning models of electronic and energetic properties consistently reach approximation errors better than DFT accuracy (2017).
  3. The starting model: Li, Tarlow, Brockschmidt and Zemel, Gated Graph Sequence Neural Networks (ICLR 2016). Its update: Cho et al., On the Properties of Neural Machine Translation: Encoder-Decoder Approaches (2014); the GRU is explained on this site at RNN encoder-decoder.
  4. The readout: Vinyals, Bengio and Kudlur, Order Matters: Sequence to Sequence for Sets (ICLR 2016), Section 4.2.
  5. Earlier models in the framework: Duvenaud et al., Convolutional Networks on Graphs for Learning Molecular Fingerprints (NeurIPS 2015); Battaglia et al., Interaction Networks for Learning about Objects, Relations and Physics (NeurIPS 2016); Kearnes et al., Molecular Graph Convolutions: Moving Beyond Fingerprints (2016); Schütt et al., Quantum-chemical insights from deep tensor neural networks (Nature Communications, 2017); Kipf and Welling, Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017), explained at Graph Convolutional Networks.
  6. Later work: Schütt et al., SchNet: A continuous-filter convolutional neural network for modeling quantum interactions (NeurIPS 2017); Battaglia et al., Relational inductive biases, deep learning, and graph networks (2018); Xu, Hu, Leskovec and Jegelka, How Powerful are Graph Neural Networks? (ICLR 2019); Veličković et al., Graph Attention Networks (ICLR 2018), explained at Graph Attention Networks.