MPNN: Neural Message Passing for Quantum Chemistry
Each atom updates its vector from learned messages sent by the other atoms, and a readout of the final vectors predicts 11 of 13 molecular properties computed by density functional theory to within chemical accuracy.
The paper writes eight earlier graph networks as one recipe of three learned functions (a message, an update and a readout) and searches that recipe for the best model of small organic molecules. The model it finds turns each interatomic distance into a message matrix with a small neural network.
Explaining the paperNeural Message Passing for Quantum ChemistryA density functional theory calculation takes about an hour for a molecule with 9 heavy atoms. Trained on 110,462 such calculations, the network is more accurate than every earlier method in the paper's comparison on all 13 properties.
Predicting what a quantum chemistry calculation would output
Quantum mechanics determines a molecule's energy, the shape of its electron cloud and the frequencies it vibrates at, but the equations cannot be solved exactly for anything bigger than a few electrons. Chemists use approximations, and the workhorse is density functional theory (DFT). DFT is still slow: its cost grows as the cube of the number of electrons, and one QM9-sized molecule (up to 9 heavy atoms, meaning atoms other than hydrogen) takes about an hour on one core of a 2.2 GHz Xeon. A molecule with 17 heavy atoms takes up to 8 hours.
The paper treats DFT as a function to be learned. QM9 (Ramakrishnan et al., 2014) is a public dataset of about 134,000 small organic molecules made of hydrogen, carbon, nitrogen, oxygen and fluorine, each with a DFT-optimized 3D geometry and 13 computed properties. The paper uses 130,462 of them, holds out 10,000 for validation and 10,000 for testing, and trains on the remaining 110,462. The 13 properties fall into four groups: four atomization energies (, , , : how much energy it takes to pull the molecule apart into atoms, under different temperature and pressure conditions), two vibrational quantities (the highest vibrational frequency and the zero-point vibrational energy, ZPVE), three electron energies (the highest occupied and lowest unoccupied orbital energies, HOMO and LUMO, and the gap between them) and three measures of the electron distribution (the dipole moment , the polarizability and the electronic spatial extent ). The thirteenth, the heat capacity , appears in every table but is not described in the text. Each property is a separate regression problem.
Errors are reported as an error ratio: the mean absolute error (MAE) on the test set divided by a per-property threshold called chemical accuracy, taken from Faber et al. (2017). For the energies the threshold is 0.043 eV, which is 1 kcal/mol, the precision chemists conventionally need for a computed energy to be useful. An error ratio below 1 means the model predicts the DFT value within chemical accuracy. The best model in the paper has a ratio of 0.45 on , which is an average error of 0.45 × 0.043 = 0.019 eV, or 0.45 kcal/mol. The errors are measured against the DFT values, not experiment, so a ratio below 1 says the network reproduces the DFT calculation, errors included.
The paper makes two contributions. It defines a framework, the message passing neural network (MPNN), that covers eight models published before it, and it searches that framework for the model that does best on QM9. The rest of this page follows that order: the input, the framework, the earlier models inside it, the four design changes that produced the best model, its training, and its results.
A molecule as a graph
An MPNN reads a molecule as a graph. Each atom is a node with a feature vector : its element as a one-hot vector over H, C, N, O and F, its atomic number, whether it accepts or donates electrons, whether it is part of an aromatic ring (a ring whose bonding electrons are shared around the whole ring, as in benzene), its hybridization (sp, sp2 or sp3 one-hot, the shape of its bonding orbitals: linear, planar or tetrahedral), and its number of attached hydrogens, 13 numbers in all (Table 1 of the paper). Each pair of atoms that interacts gets an edge with a feature vector . Section 6 compares three choices for the edges:
- Chemical graph. Edges only between bonded atoms, labeled with one of four bond types: single, double, triple or aromatic.
- Distance bins. An edge between every pair of atoms. A bonded pair keeps its bond type; any other pair gets one of 10 distance bins: [0, 2) Å, eight bins of 0.5 Å covering [2, 6) Å, and [6, ∞). Each edge therefore carries one of 4 + 10 = 14 symbols. The authors found the distance between bonded atoms almost completely determined by the bond type, so the binned representation loses little for bonded pairs.
- Raw distance. An edge between every pair, with a 5-dimensional vector: the Euclidean distance, then a one-hot encoding of the bond type (all zeros for an unbonded pair).
Hydrogens can be left implicit, as a count in the feature vector of the atom they are attached to, or made into nodes of their own. With explicit hydrogens a QM9 molecule has up to 29 nodes instead of 9, and the paper reports that training slows by a factor of roughly 10. Figure 1 shows all six combinations on pyridine, a benzene ring with one carbon replaced by nitrogen.
The header line of Figure 1 counts directed edges, which is also the number of messages computed in each step of the model. Pyridine with implicit hydrogens has 6 bonds, so 12 messages per step on the chemical graph and 30 with an edge between every ordered pair. With explicit hydrogens it has 11 nodes and 110 ordered pairs. For the largest QM9 molecules the dense representation means 29 × 28 = 812 messages per step against 9 × 8 = 72 with implicit hydrogens, a factor of 11.3, consistent with the roughly tenfold slowdown the paper reports.
A graph has no natural order for its nodes. The same pyridine molecule can be stored with the nitrogen as atom 0 or as atom 4, and a model that gave different predictions for the two listings would be predicting a property of the file format. Earlier work had handled this in the input. The Coulomb matrix of Rupp et al. (2012), a matrix of pairwise nuclear repulsions, changes when the atoms are renumbered, so a model built on it has to learn the invariance from augmented data. The symmetry functions of Behler and Parrinello (2007) are invariant by construction but are hand-designed, and the paper notes they struggle with more than three elements. An MPNN gets the invariance from its structure, as the next section shows.
How a message passing neural network works
Every atom carries a hidden state , a vector of numbers (the released code defaults to ; the towers experiments use 200). The initial state is the feature vector , padded with zeros up to length . The forward pass then runs steps of message passing. In step each atom collects a message from each of its neighbors and sums them, then updates its state from the old state and the summed message:
is the message function and the update function, both learned. After steps a readout function , also learned, maps the set of final states to the prediction:
Here is one step with and made-up numbers. The pyridine nitrogen has two neighbors on the chemical graph, both joined by aromatic bonds, with states and . Take the message function of the gated graph neural network discussed below, which multiplies the neighbor's state by a learned matrix chosen by the bond type, and suppose the aromatic matrix is
Then the two messages are and , and their sum is . Swap which carbon is called and which , and the sum does not change. The update in the section on the GRU turns this and the old state into a new state.
Three properties follow from the form of (1) to (3). First, renumbering the atoms permutes the states without changing any of them, because each message depends only on the two atoms and their edge, and the sum in (1) ignores order; if also ignores order, the prediction is invariant to atom numbering, which the paper requires. Second, the same , and apply to a molecule of any size, so one set of weights covers all of QM9. Third, after steps the state of atom can depend on the input of atom only if is within edges of : each step moves information one edge further. Figure 2 shows the third property on 2-phenylethanol, a QM9-sized molecule with a benzene ring and a two-carbon chain ending in OH.
With the oxygen as source and bonds only, the atom at the far side of the ring is 6 bonds away, so a model with cannot let the oxygen's features affect that atom's state. The readout can still combine information from both ends, because it takes every final state as input. The two other graph options in Figure 2 are the paper's fixes for long distances, covered in their own section below: virtual edges make every atom one step from every other at the price of 72 messages per step instead of 18, and a master node makes every atom at most two steps from every other for 36.
Eight earlier models written as message passing
Section 2 of the paper rewrites eight published models by naming their , and . The exercise makes the models comparable piece by piece. Five of them are described below. The other three are the molecular graph convolutions of Kearnes et al. (2016), the only one of the eight that keeps a hidden state on each edge and updates it alongside the node states, and the spectral graph convolutions of Bruna et al. (2013) and Defferrard et al. (2016), whose messages are matrices built from eigenvectors of the graph Laplacian (Kipf and Welling's model, below, is their simplified special case).
Molecular fingerprints (Duvenaud et al., 2015). The message is the concatenation , and the update is , a sigmoid of a matrix chosen by the atom's degree. Summing concatenations gives , the sum of neighbor states next to the sum of edge features. The summed message keeps no record of which neighbor came with which bond: a carbon bonded to an oxygen by a double bond and to a nitrogen by a single bond receives the same message as one bonded to the oxygen by a single bond and to the nitrogen by a double bond. The paper names this as the reason the model cannot capture correlations between edges and nodes.
Gated graph neural networks (GG-NN, Li et al., 2016). The message is , a learned matrix per discrete edge type times the neighbor's state. The update is a gated recurrent unit (GRU), the same at every step. The readout is
where and are neural networks, is the logistic sigmoid and is elementwise multiplication. The sigmoid term acts as a per-atom, per-channel gate that sets how much of each atom's contribution enters the sum. GG-NN is the paper's starting point, and every model in its experiments keeps the GRU update.
Interaction networks (Battaglia et al., 2016). The message is a neural network applied to the concatenation , so it depends on both endpoints; the update is a neural network on ; the readout is a network applied to . The original work used a single step, .
Deep tensor neural networks (DTNN, Schütt et al., 2017). The message multiplies a projection of the neighbor's state elementwise by a projection of the edge (an expansion of the interatomic distance), and the update adds it: . The readout passes each atom through a one-hidden-layer network and sums the outputs, so the molecule's energy is a sum of atomic contributions.
Graph convolutional networks (Kipf and Welling, 2016). The message is with a scalar weight , and the update is . Appendix 10.1.1 derives this from the layer , where is the adjacency matrix with self-loops added, so the degrees there count the self-loop. Every edge gets one scalar, so the message is a weighted sum of neighbor states and cannot use edge features. The graph convolutional networks explainer covers that layer.
The edge network message function
GG-NN's message, , needs a finite set of edge types, one learned matrix each. To feed it distances the paper uses the 14-symbol alphabet of Section 6, so the model learns 14 matrices, and every unbonded pair whose distance falls in the same 0.5 Å bin gets the same matrix. Why not just enlarge the alphabet? Finer bins mean more matrices, each trained on fewer pairs, and the model still cannot use the fact that 2.49 Å and 2.51 Å are almost the same distance. The paper's answer, the edge network, makes the matrix a function of the edge vector:
where is a neural network that takes the 5-dimensional raw edge vector (distance and bond one-hot) and outputs numbers, reshaped into a matrix. In the released code it has four hidden layers of 50 ReLU units; at its output layer has 2,500 units and the whole network 135,450 weights. Figure 3 compares the two message functions for a single unbonded pair as the distance between the atoms changes, with so the matrix has four entries.
With distance bins, moving from 2.51 Å to 2.99 Å leaves the matrix and the message unchanged, and crossing 3.00 Å changes them in one jump. With the edge network the message changes continuously, and two distances that share a bin still get different messages. In the paper's Table 10, switching from the chemical graph with matrix multiplication to raw distances with the edge network lowers the average error ratio from 2.57 to 0.98 (part of that gap is an input difference, flagged in the Provenance panel), and every one of the 13 targets improves.
The code builds the edge network's matrices once per molecule and reuses them at every step, because the edges do not change between steps. The cost is in the products: with an edge between every pair, one step multiplies different matrices by vectors, which is the per step the paper cites as the main scalability problem. The undirected molecule is also treated as a directed graph with an incoming and an outgoing copy of every edge, each with its own edge network, and the two summed messages are concatenated, so has entries. The direction carries no chemical meaning here; the paper uses it only for parameter tying, and it doubles the parameters.
The paper also tried a pair message, a neural network of as in interaction networks, on the reasoning that a message which also depends on the receiving atom might use the message channel better. It did worse: trained jointly on all 13 targets, its average error ratio was 3.98 against 1.53 for the edge network, which won on 11 of the 13 targets. It was also harder to train, and the authors did not pursue it.
The GRU update and weight tying
Every model in the experiments updates states with the GRU of GG-NN, a gated recurrent unit (introduced by Cho et al., 2014, and covered in the RNN encoder-decoder explainer) whose input is the summed message and whose hidden state is the atom's state. In the convention of GG-NN and the released code,
The update gate sets, coordinate by coordinate, how much of the old state to replace with the candidate , and the reset gate sets how much of the old state the candidate is computed from. With near 0 an atom keeps its state, so the depth of the computation does not force every step to overwrite what earlier steps found. PyTorch's nn.GRUCell uses the opposite convention for (its weights the old state), which matters only when porting weights.
Continuing the worked example with the simplest weights ( and every zero, so plays no role), the gate and the candidate are
and the new state of the nitrogen is
The first coordinate, where the message was small, kept 41% of its old value; the second, where the message was large, was almost entirely replaced.
All experiments tie the weights across steps: one and one serve every , so the model is a recurrent network unrolled times over the graph. The authors tried separate weights per step in preliminary runs and found that tying the weights and spending the parameters on a larger worked better.
Long-range information: virtual edges, a master node and set2set
On the chemical graph, without distances, an atom's state depends only on atoms within bonds, and properties such as the dipole moment depend on where charge sits across the whole molecule. The paper tests three ways to let information travel further, each added to a GG-NN trained on the chemical graph.
Virtual edges. Add an edge of a new type between every pair of atoms that is not bonded. This is a preprocessing step and makes every pair one step apart, as the "+ virtual edges" option of Figure 2 shows.
A master node. Add one extra node, connected to every atom by an edge of its own type, with its own state size and its own GRU weights. Each step every atom writes to it and reads from it, so any two atoms are at most two steps apart. The paper gives the cost as and argues the master node can be made large without much extra compute.
The set2set readout. Replace the gated sum of equation (4) with set2set (Vinyals et al., 2016), a readout designed for sets. Each final state is concatenated with the atom's input features and linearly projected, giving a set of memories . An LSTM with no input then runs steps; at each step it emits a query, attends over the memories and appends what it read to its state:
The score is a scalar function of a memory and the query (in the released code a small additive attention, ). The read is an attention-weighted sum, so it ignores the order of the atoms, and the final goes through a neural network to the prediction. Where the gated sum adds up per-atom contributions in one pass, set2set reads the set times, and each query can depend on what the earlier reads returned. The search chose between 1 and 12. The attention is the same content-based weighting as in Bahdanau attention, applied to atoms instead of source words.
Table 3 of the paper averages the error ratio over the 13 targets for each modification: 3.47 for plain GG-NN, 2.90 with virtual edges, 2.62 with a master node and 2.57 with set2set. The set2set model reaches chemical accuracy on 5 targets, the four atomization energies and , using the chemical graph alone. The paper says all three modifications help on all 13 targets; its own per-target Table 7 shows two exceptions (virtual edges raise the error from 14.74 to 17.11, and the master node raises from 3.94 to 4.02). These runs, and the towers runs below, used a partial-charge input feature that is itself a DFT output and would not be available in practice; the headline results of Table 2 do not use it.
Towers: wider states for less compute
A step with an edge between every pair costs , and the paper wants larger for larger molecules. The towers variant splits each atom's -dimensional state into pieces of each, runs a separate message and update function on each piece, and then mixes the pieces with a network shared by all atoms:
Each tower costs , so the towers together cost plus the mixing network, which is per atom and does not grow with . At and each tower has 25 dimensions, and the message products take 8 times fewer multiplications than one tower of 200. The measured inference speedup on 9-atom molecules was a factor of 2, since the mixing network and everything outside message passing still cost the same. This is the same block-diagonal idea as grouped convolutions, with restoring communication between groups.
The towers also generalized better. With distance bins and no explicit hydrogens, the 8-tower model beat a tuned single-tower GG-NN with set2set on 12 of 13 targets both when one model was trained per target (average 1.37 against 1.53) and when one model was trained on all targets (1.75 against 1.92). The authors suggest towers behave somewhat like an ensemble. Combining towers with the edge network did not improve results in their attempts, so the best model does not use towers.
Inputs and training
The best model, called enn-s2s, combines the edge network message, the GRU update with tied weights, the set2set readout, raw distances on every pair of atoms, and explicit hydrogens. Table 10 compares inputs with set2set and the GRU fixed (the model without distances uses matrix multiplication, the other two the edge network): the average error ratio is 2.57 on the chemical graph, 0.98 with distances, and 0.68 with distances and explicit hydrogens. The hydrogens matter most for (0.95 to 0.30) and (0.57 to 0.15), both measures of how charge spreads through space.
Every combination of model and target was tuned with 50 random hyperparameter trials: the number of steps between 3 and 8, the set2set steps between 1 and 12, an initial learning rate between 1e-5 and 5e-4, and a linear decay that starts between 10% and 90% of the way through training and ends at a factor between 0.01 and 1 of the initial rate. Training uses Adam with batches of 20 molecules for 3 million steps, which is 3,000,000 × 20 / 110,462 ≈ 543 passes over the training set. Each target is normalized to mean 0 and variance 1; the loss is mean squared error, the reported metric mean absolute error; the validation set picks the stopping point and the best trial, and scores are reported on the test set. One model per target consistently beat one model trained on all 13 targets, by up to 40% on some targets.
One forward pass in code
The listing traces enn-s2s on one molecule in array notation, with the default sizes of the released model code (the paper's numbers come from per-target searches over these sizes). The edge networks output a full 50 × 50 matrix for each of the 19 × 19 atom pairs, and the einsum sums over the neighbors .
# enn-s2s forward pass on one molecule, default sizes of brain-research/mpnn
# 2-phenylethanol with explicit hydrogens: n = 19 atoms, d = 50, T = 6
x = atom_features(mol) # [19, 13] Table 1 features
e = edge_vectors(mol) # [19, 19, 5] (distance, bond one-hot)
h = pad(x, 50) # [19, 50] h^0: x padded with zeros
# the edge networks run once per molecule; their output is reused at every step
A_in = edge_net_in(e).reshape(19, 19, 50, 50) # 4 hidden layers of 50 units
A_out = edge_net_out(e).reshape(19, 19, 50, 50)
for t in range(6): # the same weights at every step
m_in = einsum("vwij,wj->vi", A_in, h) # sum over w of A(e_vw) h_w
m_out = einsum("vwij,wj->vi", A_out, h)
m = concat([m_in, m_out], -1) + b # [19, 100] eq (1)
h = gru(h, m) # [19, 50] eq (2)
q = set2set(proj(concat([h, x], -1)), steps=12) # [126], any atom order
y_hat = mlp(q) # one normalized target, e.g. U0
loss = (y_hat - (y - mean_y) / std_y) ** 2 # train on MSE, report MAE
# gradients reach both edge networks, b, the GRU, proj, the set2set LSTM and
# the MLP; no weight belongs to one atom, so the same model runs on any moleculeThe released repository, brain-research/mpnn, contains only these model classes and their tests, not the QM9 reader or the training loop. Its model classes expect a batch of molecules padded to the size of the largest, with a mask that zeroes the padded atoms after each GRU update and excludes them from the readout. Two details in it differ from the paper's equations and are listed in the Provenance panel: the GG-NN readout feeds the atom features to both networks of equation (4) and leaves out the tanh on , and the 19 × 19 pairs include each atom with itself.
Results on QM9
Table 2 compares enn-s2s with five hand-engineered molecular representations evaluated by Faber et al. (2017) (the Coulomb matrix CM, bag of bonds BOB, BAML, the ECFP4 fingerprint and HDAD) and two MPNN baselines (the graph convolutions of Kearnes et al. and GG-NN with distance bins). Figure 4 plots the error ratios on a log axis.
The enn-s2s model has the lowest error of all methods on every target, with an average error ratio of 0.68 against 1.35 for HDAD and 1.36 for GG-NN, and it is within chemical accuracy on 11 of 13. The two misses are the HOMO-LUMO gap (1.60, an MAE of 0.069 eV) and ZPVE (1.27). HOMO, at 0.99, is inside the threshold by 1%. Averaging the five models with the lowest validation error lowers the average to 0.52, and lowers every target, but the gap (1.23) and ZPVE (1.10) still miss.
The largest gains are on targets that depend on the spatial arrangement of electrons: falls from 0.80 (bag of bonds) to 0.15, and from 0.70 (Kearnes et al.) to 0.30. The ECFP4 fingerprint, which encodes substructures but no geometry, has error ratios between 30 and 242 on 8 of the 13 targets. Table 6 of the paper trains enn-s2s on smaller subsets: with 11,000 molecules, a tenth of the training set, it already matches or beats the best earlier method trained on all 110,000 for (0.28 against 0.80) and comes within 0.01 of it for (0.28 against 0.27); the paper calls both equal or better. The errors mostly fall as the training set grows, with a few exceptions such as , which is 0.15 at 82,000 molecules and 0.19 at 110,000.
DTNN appears in Table 2 with a single number, 0.84, taken from its own paper and trained on a different split of 100,000 molecules; Table 2 prints it in the row, but Schütt et al. report it for the total energy, the quantity in the row, where enn-s2s scores 0.45.
Limits and later work
The benchmark is narrow by design. QM9 molecules have at most 9 heavy atoms, the labels are DFT values, and the test molecules come from the same distribution as the training molecules. Table 5 of the paper puts the DFT error against experiment above the chemical-accuracy threshold for 11 of the 12 targets that have an estimate (dipole moment is the exception, at 0.1 against 0.1), so the network now matches DFT more closely than DFT matches nature. The abstract concludes that future work should move to larger molecules or more accurate labels.
The best model needs a geometry. Its edge vectors are distances from the DFT-optimized structure in QM9, so applying it to a new molecule requires a 3D structure from some other method first. Without distances the best model in the paper is within chemical accuracy on 5 targets, not 11.
Cost grows with the square of the molecule size. With distance edges every atom receives a message from every other atom, and each message is a matrix product. The conclusion names generalization to larger molecules as the main open problem, for two reasons: the distribution of distances changes with size, and the number of summed messages grows with the number of atoms. It proposes attention over incoming messages, which graph attention networks introduced later the same year for citation and protein graphs. GraphSAGE, published two months after this paper, reduced the cost by sampling a fixed number of neighbors per node.
Message, aggregate and update became the standard way to describe graph neural networks, and the edge network's idea of generating interaction weights from distances continued in SchNet (Schütt et al., 2017), whose convolution filters are a learned function of interatomic distance. Battaglia et al. (2018) extended the framework with explicit edge states and a global state, the roles that Kearnes et al.'s edge updates and the master node play here. Xu et al. (2019) proved that a message passing network cannot tell apart two graphs that the Weisfeiler-Lehman graph isomorphism test cannot tell apart, which bounds what any model inside this framework can distinguish on a bond graph without distances.
Questions you might still have
Is an MPNN the same thing as a graph neural network?
Most graph neural networks in use fit the MPNN form: a per-edge message, a sum or other order-independent combination at each node, a per-node update, and for graph-level tasks an order-independent readout. The paper did not invent the form. It named it, showed that eight published models are special cases, and searched inside it. Graph attention networks and GraphSAGE, both from later in 2017, also fit it; the explainers on this site for each walk through their message and aggregation choices.
Why does equation (1) sum the messages instead of averaging them?
The paper does not discuss the choice. A sum keeps the number of neighbors in the result: a carbon with four neighbors receives four messages, a carbon with two receives two, and an average would erase that difference. Any order-independent combination keeps the model independent of atom numbering; the sum is the simplest one that also preserves counts. Xu and colleagues (2019) later proved that a sum can map different multisets of neighbor features to different results where a mean or a max cannot: a mean gives the same answer for one neighbor of a kind as for two of the same kind.
Does the best model need a 3D geometry for every molecule it predicts?
Yes. The edge network reads interatomic distances, and in QM9 those come from the same DFT optimization that produced the labels. Without distances the best average error ratio is 2.57 instead of 0.68 (Table 10). In an application the geometry would have to come from a cheaper method, which adds its own error that the paper does not measure. The graph-only models reach chemical accuracy on 5 of 13 targets, all five of them energies or the highest vibrational frequency.
Why train a separate model for each of the 13 targets?
Because it measured better: one model per target beat a single model trained on all 13, by up to 40% on some targets. In Table 4 the joint GG-NN baseline averages 1.92 against 1.53 when trained per target, and the towers model 1.75 against 1.37. Each model and target pair also got its own 50-trial hyperparameter search, so the headline numbers come from 13 separately tuned networks.
How many message-passing steps does a molecule need?
The search chose T between 3 and 8, and the paper reports that any T of at least 3 works. On the bond graph a QM9 molecule with 9 heavy atoms is at most 8 bonds across, so 3 steps do not connect every pair. The best models use distance edges between every pair of atoms, where one step already connects every atom to every other, and extra steps let information combine over several hops through intermediate atoms.
What does "300,000 times faster than DFT" compare?
A DFT calculation for one 9-heavy-atom molecule takes about an hour on one core of a 2.2 GHz Xeon E5-2660 with Gaussian 09, and up to 8 hours at 17 heavy atoms. The footnote says the networks discussed are 300,000 times faster at inference, which works out to about 12 milliseconds per molecule against the one-hour figure. The paper does not say what hardware ran the network or whether molecules were batched.
Does the model work on molecules larger than the 9 heavy atoms of QM9?
The paper does not test it and lists it as the main open direction. It names two problems: the distribution of pairwise distances depends on the number of atoms, and with distance edges every atom receives one message from every other atom, so the size of the summed message grows with the molecule. It suggests attention over the incoming messages as a fix, the mechanism that graph attention networks introduced a few months later.
Is chemical accuracy against DFT the same as chemical accuracy against experiment?
No. The labels are DFT outputs, and DFT itself has errors relative to nature that are larger than the chemical-accuracy threshold for most targets: Table 5 lists an estimated DFT error of 0.1 eV for the atomization energies against a threshold of 0.043 eV, and 2.0 eV for the HOMO energy. A model that matches DFT within 0.043 eV can still be 0.1 eV away from an experiment. The paper says this, and argues the next benchmark should have more accurate labels or larger molecules.
Footnotes & further reading
- The paper: Gilmer, Schoenholz, Riley, Vinyals and Dahl, Neural Message Passing for Quantum Chemistry (ICML 2017; this page follows arXiv v2). Model code: brain-research/mpnn (archived; model classes only, line numbers in the Provenance panel refer to its master branch).
- The dataset: Ramakrishnan, Dral, Rupp and von Lilienfeld, Quantum chemistry structures and properties of 134 kilo molecules (Scientific Data, 2014). The thresholds and baselines: Faber et al., Fast machine learning models of electronic and energetic properties consistently reach approximation errors better than DFT accuracy (2017).
- The starting model: Li, Tarlow, Brockschmidt and Zemel, Gated Graph Sequence Neural Networks (ICLR 2016). Its update: Cho et al., On the Properties of Neural Machine Translation: Encoder-Decoder Approaches (2014); the GRU is explained on this site at RNN encoder-decoder.
- The readout: Vinyals, Bengio and Kudlur, Order Matters: Sequence to Sequence for Sets (ICLR 2016), Section 4.2.
- Earlier models in the framework: Duvenaud et al., Convolutional Networks on Graphs for Learning Molecular Fingerprints (NeurIPS 2015); Battaglia et al., Interaction Networks for Learning about Objects, Relations and Physics (NeurIPS 2016); Kearnes et al., Molecular Graph Convolutions: Moving Beyond Fingerprints (2016); Schütt et al., Quantum-chemical insights from deep tensor neural networks (Nature Communications, 2017); Kipf and Welling, Semi-Supervised Classification with Graph Convolutional Networks (ICLR 2017), explained at Graph Convolutional Networks.
- Later work: Schütt et al., SchNet: A continuous-filter convolutional neural network for modeling quantum interactions (NeurIPS 2017); Battaglia et al., Relational inductive biases, deep learning, and graph networks (2018); Xu, Hu, Leskovec and Jegelka, How Powerful are Graph Neural Networks? (ICLR 2019); Veličković et al., Graph Attention Networks (ICLR 2018), explained at Graph Attention Networks.
How could this explainer be improved? Found an error, or something unclear? I read every message.