Quantum mechanics and human dynamics

Purpose of this article

The full story of the development of quantum mechanics has been told many times, sometimes vividly. ¹ The story is awkward, starting with an unexpected twist, and inevitably entangled with human nature and human dynamics. Repeating this story here would not do justice to its full complexity and sheer length. Instead, this article will highlight a few lesser-known facts and do so in an undisguised and candid manner. Subsequently, a short but poignant digression explains where things can go wrong when quantum mechanics treads beyond its familiar territory, as it does in cosmology and high-energy physics. We then move on to the safer grounds of quantum chemistry in order to concisely set the scene for three sustained contributions made by the author’s group: FFLUX, AIBL and REG. All three will be explained in a balancing act of brevity and enticement, deliberately leaving out most details such that the overall message is preserved.

The mission of the International Year of Quantum Science and Technology (IYQ) instructs participants to be honest. The mission statement recognises that “many things have been and will be said about quantum science and technology that are untrue or misleading. All people proceed from learning simple, sometimes incorrect stories to deeper, more nuanced ones.” The author strove to keep all this in mind while writing this article. He stayed away from the red-carpet language we read and hear too much. Too much excellent and important work has been done, over many decades if not centuries, by largely silent or even self-effacing contributors who often pioneered. This phenomenon is a product of human dynamics. But what is meant by human dynamics? In this article an answer will be provided, as the author discovered it himself, both from personal experience and by having read on other scientists.

Human dynamics expresses itself in a number of phenomena appearing in how science is done. The first example of human dynamics is that just mentioned above, which produces forgotten pioneers amongst others. There is a large number of scientists who do not end up with the credit they are due for various reasons: their modesty, lack of interest in public recognition, lack of confidence, low self-esteem, not fitting in the red-carpet world, self-isolation by focusing on scientific topics that are (not yet) popular, or just out to have fun solving problems whoever understands or appreciates their solutions. But human dynamics can also literally mean travelling to a far destination such as the United States of America, for example in 1925, at a crucial time in the development of quantum mechanics taking place not there but in Europe. This unfortunate travel decision by Born, in an internetless world with ocean steamers, weakened the position of Göttingen (where he was then based) in the swift development of quantum mechanics at that time. Human dynamics can also refer to groupthink, the excesses of which can lead to the abysmal situation in string theory where no decision is ever taken to abandon it altogether in spite of its massive, stubborn and widely acknowledged flaws. The structure of careers and funding, another product of human dynamics, is partially to blame for this sad state of affairs. Human dynamics is also at work when strong personalities dominate the direction in which a research field progresses, or the way to think about it. If sustained, it brings about power structures that decide who publishes in fancy journals and who not. Next, the naming of an invention, discovery, equation, law or concept is also subject to human dynamics. Stigler’s law of eponymy ² claims that no scientific discovery is named after its original discoverer. With this law is associated a long list of examples such as Pythagoras’s theorem, which was known to Babylonian mathematicians. Ironically, even Stigler’s law was actually not attributable to him. Human dynamics also simply refers to tensions and even rivalries between individual scientists or groups thereof. Finally, human dynamics also involves intellectual laziness, causing scientists to be unaware of developments of which they do not (yet?) realise the importance. Such inertia has afflicted science (and mathematics) on a number of occasions.

Finally, one should not forget what a clear-headed European thinker, such as Penrose, thinks about quantum mechanics. He believes that saying that quantum mechanics is incomplete is polite; according to him, the truth is that quantum mechanics is wrong. ³ Physics Nobel prize winner ‘t Hooft also claims that quantum mechanics is wrong in its current form. Of course, as an intellectual creation it is formidable and can boast many successes, both in solid state physics and by underpinning the “whole of chemistry”, hinting at Dirac’s adage ⁴ of 1929. This practical success cannot be taken away by the remaining conceptual confusion surrounding quantum mechanics, in particular in connection with the measurement problem (the collapse of the wave function) where Penrose’s main gripe lies.

In summary, the full story of quantum mechanics is not linear nor one-dimensional; this article seeks to convey a flavour of this characterisation.

The origin of quantum mechanics

It is about a hundred years ago that modern quantum mechanics was born. In fact, it was a twin birth because modern quantum mechanics came in two versions. Both were proposed in rapid succession (by several months) compared to the slower development of the old quantum theory, which had ruled between 1900 and 1925. As a bunch of heuristic corrections to classical mechanics, the old quantum theory was ready to be replaced by something consistent and more powerful, while working for the right reasons.

The first version is called matrix mechanics, the germs of which were proposed by Heisenberg in his mid twenties. However, it was the technically versed but less famous Kramers who had laid ⁵ the groundwork for it. The second, slightly younger, version is named wave mechanics and was introduced by Schrödinger while in his late thirties. Although this version was more of a single-handed product it was not developed in total isolation, as can be deduced from two facts. First, before Schrödinger thought of his priceless equation, he had been spurred to look for such an equation by Debije, also known as Debye, while living outside of his native Netherlands. Indeed, back then, there was no predictive mathematical framework inspired by, and corresponding to, the wave-like nature of particulate matter that de Broglie had introduced. Secondly, immediately after Schrödinger had obtained his equation, he sought feedback from the mathematician Weyl. Fascinatingly, Schrödinger initially feared that his equation would not work, fatally so, a fact that is little known. The reason for this (misplaced) fatalism was that he could not imagine how proper vibration frequencies can appear without boundary conditions, unlike in the case of the partial differential equation governing pressure in the classical mechanics of an elastic body.

The beginnings of quantum mechanics were humble and actually rooted in solving a practical problem of commercial value. The manufacturing industry wondered how the efficiency of its light bulbs could be improved, which led to the set-up of the well-known black-body radiation experiment. Many a student may have wondered where this quite bizarre experiment actually comes from, as undergraduate textbooks are typically silent on the motivation behind it all. This conundrum is not solved by artificial intelligence (AI) (readily used with a Google search engine), which may wrongly inform those same inquisitive students that “black-body radiation was studied because it led to the development of quantum mechanics”. Not so: no one had an inkling that the whole new science of quantum mechanics would emerge from researching such a mundane problem. A properly informed human knows, unlike AI, that quantum mechanics was stumbled upon during the rather boring fitting of curves measured in connection with black-body radiation, a problem studied for practical and even commercial reasons.

Indeed, Planck, who unwittingly originated quantum mechanics, was one of several researchers hunting for the mathematical expression that best fits the spectral energy density of the black-body as a function of its temperature and emitted frequency. It took the otherwise conservative Planck a leap of imagination to divide energy into discrete increments in order to discover the best fit. More precisely, Planck assumed that radiation energy is emitted not continuously but in discrete packets called quanta. Although he saw this hypothesis as a mere mathematical device, it became soon clear that, right here, nature revealed its fundamentally different character at very small scales; the radically new concept was quantisation.

Ironically, Planck was advised not to study physics because of the then prevailing opinion that physics was complete. This misleading mind set was surely inspired by the spectacular success of classical mechanics and thermodynamics. Alas, misplaced hubris failed to recognise that 19th century physics explained only part of the world. Nature forced completely new thinking upon the scientific enterprise, and with this new paradigm shift heralded an era of spectacular success.

The doctrine of quantum mechanics

Quantum mechanics as we know it today, and as used in solid-state physics, materials science and computational chemistry, was consolidated by the ever so forceful Bohr who was also portrayed as a fuzzy thinker, as expressed both in his writing and speaking. As a dominant figure, he persuaded the Carlsberg brewery to fund a new building originally called the Institute for Theoretical Physics of the University of Copenhagen. Actually, there were three geographical hotspots of quantum mechanics: Göttingen, Munich and Copenhagen. The latter won out, due to human dynamics, one can argue, including visiting the United States of America at the wrong time, as Göttingen-based academic Born later found out. Secondly, it is again human dynamics that should illuminate why Munich ended up as an underrated hotspot acting as a theatre backstage of “mere” mentoring. How is that Sommerfeld, while in Munich, supervised 7 researchers, both doctorally and postdoctorally, who all went on to win a Nobel prize while he himself failed to win one, in spite of being nominated 84 times? There also existed researchers such as Einstein, Schrödinger and Dirac, who remained largely unaffiliated with these hotspots, yet they managed to contribute immensely. Of course being in the right place at the right time, in a different way, helped.

It was the so-called Copenhagen interpretation of quantum mechanics that eventually gave the impression that quantum mechanics was a closed and completely understood subject. This impression tuned out to be false, as one could expect from the unfortunate combination of Bohr’s fuzziness and forcefulness. After all, is he not one who signed it off? The community ignored any healthy and ultimately helpful criticism, such as that ⁶ of Hermann. She exposed a fundamental misunderstanding about von Neumann’s proof on the impossibility of embedding quantum mechanics in a hidden variable theory. She showed that the theorem was only true under the specific assumptions made by von Neumann. Her work ⁷ was overlooked or dismissed for decades until brave and lucid minds such as Bell discovered the flaw and independently of Hermann published ⁸ the same criticism about 3 decades later, in 1966. One can only imagine how quantum mechanics would look today had its pioneers Hermann’s work seriously when published in 1935. Bell’s work led to Bell’s inequality, a falsifiable cornerstone of quantum mechanics with far reaching implications, even philosophical ones. That this inequality is violated was experimentally verified, ultimately by closing any interpretational loopholes. This pivotal result not only vindicated quantum mechanics but also sharpened the contours of its bizarre, counterintuitive character. Curiously, as one of the (last) bastions of major and healthy advances in physics, it does not appear to feature in cosmology. A reader of popular science would intuitively expect Bell’s theorem to heavily shape the regrettably wild speculations that routinely feature in this research field. Finally, we should mention that another such bastion is the experimental confirmation ⁹ of the Brout-Englert-Higgs boson, also simply known as the Higgs boson. Here too, human dynamics must surely be at work but one can only speculate which type: last incoming scientist has last word? Networking and international appeal? The community’s laziness?

The Schrödinger equation revisited

The triumph of the Schrödinger equation lies in the fact that it contains only fundamental physical constants (e.g. the speed of light, the mass of the electron, Planck’s constant) while nevertheless being able to predict the whole of chemistry. It does so without invoking any chemical parameters but may need an inordinate amount of computer time. In fact, the challenge of predicting chemical phenomena is quite similar to that of predicting the weather from the Navier–Stokes equations, which again do not contain any “weather parameters”. After all, both these equations and the Schrödinger equation are information generators; they necessitate the art of finding the most accurate prediction within the least amount of computation time. These compact but powerful equations stimulate the third leg of science: computation and simulation. Whereas the first two legs, theory and experiment, delivered insight into a large array of natural phenomena, it has been clear for a while that precious key equations are not enough to make actual predictions. It is quite an achievement to extract Newton’s equations of motion from a large set of observations but it is another thing to then use them to predict configurations of our solar system. Even with only three celestial bodies analytical techniques start faltering, and it is computer simulation that comes to the rescue.

The main challenge for this leg of science is to make the mapping between simulation results and experimental observation exact and perfect. Unfortunately for the field of molecular simulation, we are still a long way off this ambitious goal. As we shall see below, we still cannot predict more than one property of liquid water, all even within say 50 % from their respective experimental values. Moreover, we have to remain vigilant and critically examine if any agreement is fortuitous. It is important to be convinced that agreement between experiment and simulation has been obtained for the right reasons. Only then can computational science become a source of reliable information, and can we allow it to subsume older sciences such as chemistry, physics, geology or biology, as special applications.

Quite soon after the Schrödinger equation had proved that it successfully predicted and fully explained the spectrum of a single hydrogen atom, it had to prove its prowess in doing the same for something as simple, yet surprisingly complicated, as helium. This was looked into ¹⁰ by Heisenberg who was initially agitated by the Schrödinger equation as a competitor to his own version of quantum mechanics. However, he managed to embrace this equation as a computationally superior alternative, almost as soon as it came out of the oven in 1926. Later work ¹¹ by Hylleraas was very much motivated by what he called an “extremely important question, if the numerical calculation according to wave mechanics also leads to exact results for many-electron problems”. The outcome of his pioneering work was promising towards this grand goal. Such tests have continued to be carried out, ¹² even very recently, involving increasingly involved calculations but with the aspiration of constructing ever more compact wavefunctions. Moving away for a moment, from theoretical and computational chemistry towards quantum electrodynamics, even more extreme agreement between experiment and quantum mechanics was reached for the anomalous magnetic moment of the electron. However, the path ¹³ of this spectacular agreement (of currently 12 digits) was scandalously tortuous, with manipulation as well as genuine error upon error.

In its full form, the Schrödinger equation describes the nuclei that appear in molecules also in terms of a wavefunction. This description is counterintuitive to more than the proverbial 99.99 % of chemists who think of nuclei as points in space, without extension or probabilistic character. The Born-Oppenheimer approximation gives them a license to think like this: molecules are a framework of nuclear points surrounded by a swarm of wavefunction-like electrons. Almost half a century ago Woolley published a paper ¹⁴ entitled “Must a molecule have shape?” where he argues that molecular structure is not an intrinsic property. He states that molecular-beam gas-phase experiments probe molecular stationary states that cannot be understood in terms of molecular structures.

Quantum mechanics going astray

The present article sits firmly within computational chemistry as will soon become clear. However, as it has been commissioned in the widest context of quantum mechanics’ role and influence, it is appropriate to slip in a word about how science can go astray, or perhaps even disappear altogether. This sad phenomenon occurs when any experimental verification (i) is absent, or (ii) fails the pivotal attribute of repeatability. The first situation is the case in the field of high-energy physics and the second in cosmology, two fields where quantum mechanics also plays a role.

First, in cosmology we cannot re-run the experiment that is the creation of the Universe, or the scenario of whatever happened at its beginning, if there even is one. Similarly, the grand challenge of the Origin of Life also suffers from the impossibility of experimental repetition. Instead, one has to work like a detective who is actually in the same position: he (yes, the detective is male this time) cannot re-run the murder. He cannot repeat the murder in different conditions each time, as if it were a typical scientific experiment. Yet, scientific experiments are designed in that way; they exaggerate and control a very small piece of repeatable reality, and then draw cause-effect relationships from the observations. Yet, a detective has to work with the observable consequences of a single, poorly understood event from the past. The type of reasoning needed to proceed here is called abduction (or abductive inference). ¹⁵ It is the last fortress of human thought that current AI has failed to conquer. Only, so-called General AI can start to master abduction. Any attempts at this human-like capability have hitherto nose-dived.

Recently, it is often said that cosmology is in a crisis, a situation exacerbated by an increasing number of incoming experimental observations, one firmer than the previous; the upshot is that they deviate from predictions made by standard theory, louder and clearer as time progresses. It is crucial to take these deviations seriously and resolve them by a minimal and substantiated theory. It is futile and dangerous to ignore these deviations and to fantasise about the character of the Universe when it was supposedly very small, an ever varying number of billions of years ago. A fine example of a researcher who focuses hard, with his graduate students, on proposing a new theory that endeavours to explain these deviations is Turok. Over and above working in this correct way, and thereby maintaining a proven scientific tradition, he has a strong sense of minimality. Turok constructs ¹⁶ his explanation and theory in a grounded way while using the smallest number of assumptions. This is at variance with the “style” of Hawking whose ever-changing and unfalsifiable speculations ¹⁷ appear far-fetched and non-minimal. This is the same Hawking who stated that physics would be more interesting if the Brout-Englert-Higgs boson had not been found.

Secondly, in theoretical high-energy physics, the state-of-the-art is also dire, that is, it also finds itself in a crisis. The imposing goal of this piece of physics is to unify general relativity^[1] and quantum mechanics into a so-called “theory of everything”. The latter name is highly regrettable and even arrogant: it reveals a stunning ignorance about the Universe, and all that is in it, including how it works. Can such a theory of everything explain why the vast majority of humans has ten fingers, for example? Or predict when and how the Russo-Ukrainian war will finish?

Let us never forget that the ultimate arbiter of a theory or a model is experiment. No matter how minimal or elegant this theory, if its predictions do not “make proper contact with experiment”, it is all wrong. By “proper contact” is meant that the theory generates at least one falsifiable prediction but ideally a number of them, which collectively lead to excellent (at least promising) agreement between theory and experiment. The hard lesson that such contact is often lacking has yet to be learnt in fundamental physics, for example, where string theory is increasingly seen as failing to be that theory of everything. Note that this failure is far beyond the ultimate veracity of experimental values. In other words, even when better experimental technology offers theoreticians a superior experimental value string theory would still be wrong. In fact, at the moment it does not even make a single prediction. String theory, which was presented as the only game in town for too many years has not lived up to its repeated promises. This demise was clear already two decades ago. ¹⁸ This disconcerting conclusion must have been reached, perhaps even earlier, by perspicacious researchers who are honest with themselves. If free from human dynamics, they can express the view that even updated versions of string theory do not tackle old, well-known problems with it, but actually introduce new problems. Disturbingly, string theory has not been verified by experiment. However, string theorists will remind everyone that the word “yet” should be added here. Worse, string theory has not made even a single experimental prediction. This is why it is apt to designate it as “not even wrong” and not even science. This is what happens if a theory (if it is one in the first place, as some argue) is allowed to develop in an experimental vacuum as it were. The theory expands in an unbridled way, without any direction from, or judgement by, experiment. No wonder that you-tuber Hossenfelder is becoming increasingly frantic and increasingly spews venom. This is where quantum mechanics, and the broader science that it connects to, go wrong, in the author’s opinion.

Quantum chemistry

An area where quantum mechanics did not go astray this badly is in the study of matter at atomic scale, i.e. in quantum chemistry and solid-state physics. However, here too, one should stay watchful about whether a calculation or prediction works for the right reasons, as illustrated in the next section. Very soon after the introduction of the Schrödinger equation, its immediate application to molecular hydrogen again survived this stringent test. Suddenly, the mystery, up to that point in 1927, of the stability and formation of (nonpolar) chemical bonds disappeared. This article is not the place to review the history of quantum chemistry but, in continuation of this article’s undisguised style, it will just mention a quick “the-emperor-is-naked truth” about density functional theory (DFT).

The original dream behind DFT was for it to offer a much faster, and a more intuitive, computational framework to predict properties of matter at roughly ambient conditions. On one hand, DFT’s superior speed would be guaranteed by the three-dimensional nature of the electron density, as opposed to the horribly high dimensionality of wavefunction space. On the other hand, DFT’s superior intuition would be guaranteed by the more tangible, real-space character of the electron density. Moreover, the latter is measurable and observable. The same cannot be said of a wavefunction, whether describing one electron or many electrons. Yet, ironically, it is those wavefunctions that came to the rescue of DFT, thrice actually. The first time was to finally make the kinetic energy accurate enough, compared to a true functional of the electron density, such as some power of the electron density or of the magnitude of the electron density’s gradient. This development is often described as the birth of DFT, most likely because at last it made DFT sufficiently accurate. However, in reality, this decision caused the partial death of DFT because it moved DFT away from its original wavefunctionless tenet. The second time a wavefunction contamination improved DFT’s accuracy, and thereby further killed off DFT’s main tenet, is the introduction of hybrid functionals. Here, wavefunctions came to the aid of calculating the exchange energy more accurately, as achieved by something as basic as Hartree–Fock theory. The third time was when parameters crept into the functionals, and experimental data of a set of guiding molecules were fitted to them. This decision also undermined the original, first-principles character of DFT.

In summary, quantum mechanics has its firm origins in embracing an experimental anomaly. From this work emerged a novel paradigm that has withstood extensive testing, each time vindicated as correct. Yet, quantum mechanics’ deep nature is not understood. Quite a few scientists are content with just using it. However, there are also no-nonsense thinkers, such as Penrose, who rightly call the “observer problem” a red herring. In the end it is dangerous to surrender to something that is not understood and leave it at that. The history of science offers several examples where allowing mysteries to be unresolved ultimately leads to a dead end, classical mechanics being a famous example. Quantum mechanics cannot be an exception. Thus, it is best to indeed bother with what quantum mechanics actually means. Whereas ionic liquids, high throughput screening, systems biology, genomics, buckminsterfullerene, graphene, room temperature superconductivity and biofuels have come and gone (at least in terms of their hype, while a valuable rump remains), quantum mechanics has stayed, and will surely do so over the next 100 years but hopefully in a more approachable form.

A critical look at quantum chemistry in action: examples of remaining challenges

On the background of Quantum Chemical Topology (QCT)

The atom is a key concept in chemistry. Hence, one needs a trustworthy way of defining it, and indeed calculating its properties, as it exists within a molecule, or more broadly, in a system of atoms (excluding extreme conditions). The co-workers of Bader, and Bader himself, have provided the community with a solution for this problem, one that has gone from strength to strength since its inception ²⁸ more than half a century ago. This approach is referred to as a Quantum Theory of Atoms in Molecules (QTAIM), ^{29 , 30 , 31 , 32} which has been explained ^{31 , 32 , 33 , 34} and reviewed ^{35 , 36 , 37} several times before. The essence of this approach is to use the language of dynamical systems (attractor, separatrix, critical point, gradient path, …) in order to understand molecules, their aggregates and condensed matter.^[4]

Originally, this beautiful, powerful and universal piece of mathematics was only applied to the electron density and to its Laplacian, two quantum mechanical functions of great import, both in real 3D space. Over time, more functions were analysed in this distinct way such that the author found it useful, in response to human dynamics, to bundle them under the name Quantum Chemical Topology (QCT).^[5] For example, the electron localisation function ³⁹ served as a wealth of information after its first quantum topological analysis ⁴⁰ leading to decades of fruitful follow-up research. The keyword of QCT united the factions that had contributed to what is best considered as a single QCT approach, which is then logically distinct from alternative, non-QCT approaches. This distinction applies to population analyses, energy decomposition analyses, and any other tool of interpretational quantum chemistry. In 2001 we showed ⁴¹ how our six-dimensional volume integration over two quantum topological atoms freed QTAIM from the constraint of the virial theorem because this achievement enabled potential energy to be computed independently of this theorem. Hence, a QCT-based energy partitioning ⁴² could henceforth operate on a molecular geometry that is not a stationary point. Shortly thereafter, Blanco et al. ⁴³ proposed a version of topological energy partitioning called Interacting Quantum Atoms (IQA), which we extended ⁴⁴ to be used in conjunction with Density Functional Theory.

Perhaps even more time is needed before “the theoretical and computational chemistry establishment” (another construction of human dynamics) is au fait with the QCT perspective, for it to hopefully appreciate it too, at that long-awaited point. More details on QCT can be found elsewhere ^{38 , 45 , 46} including the one that there is a sweet spot ⁴⁷ for the selection of quadrature grids for integration over atomic volumes.

Contribution 1: FFLUX or designing a force field that sees the electrons

FFLUX (formerly known as QCTFF) is the longest running project in the group. So far, it has involved more than 20 group members, and even more MChem, MSc and summer students. Several other, very different projects have run simultaneously. However, only FFLUX became the flagship project, which continuously progressed over the best part of 25 years, if one starts with its early days of investigating the electrostatic interaction. ^{48 , 49 , 50} The motivation of FFLUX was clear from the onset: replace classical force fields such as AMBER or CHARMM by something more reliable. Our strategy was always to start from scratch and to overhaul the architecture of such classical force fields. Critical and honest reading of the classical force field literature reveals that they are unable to make proper contact with experiment.

This alarming state-of-affairs has been spelled out already in 2003 where versions of AMBER, CHARMM, GROMOS and OPLS supply a cacophony ⁵¹ of molecular dynamics results that disagree between themselves, More alarmingly, their predictions deviate, even qualitatively, from NMR, Raman and infrared experiments, for something as simple as trialanine in aqueous solution. The status quo is equally disconcerting ⁵² for structural ensembles of IDPs. This is why the FFLUX research program started ab ovo, and proposed a force field that is much closer to the underlying quantum mechanics. The key to do so is to directly link the parametric values of the force field to atomic properties that can also directly and transparently be linked to (reduced) density matrices (of which the electron density itself is a special case) and the energies that correspond to these density matrices, including the kinetic energy ⁵³ and the electron correlation energy. ⁵⁴ In doing so, it abandons design principles that have proven to stifle progress over decades. Indeed, instead of observing a gradient of irreversible progress, one witnesses an unresolved going forwards and backwards: energy terms are added and then taken away again a few year later, and endless reparameterisations makes one gain some agreement with experiment here while losing some there.

Reviewing the more than 80 papers behind FFLUX is best done by means of book such as an issue in the Springer series “Lecture Notes in Chemistry”, which is planned over the next few years as FFLUX nears completion. Such a platform will be more instructive than a review because the former offers a much needed didactic dimension. It is high time this research effort reaches a wider audience, which would benefit from a breath of fresh air and a future-proof alternative. In this article, it is sounder to highlight FFLUX’s characteristics and design decisions, which would otherwise drown in details and milestones that are better understood from the original publications.

The following points are siphoned off and numbered in order to clarify FFLUX to the now large machine learning community in computational chemistry, with the anticipation that FFLUX will start resonating with it:

1. All atomic properties (charge, energy, dipole moment, …) come from one single formula, which is an integration over atomic volume(s) of a quantum density. This decision is crucial to guarantee consistency in the processing of atomic information. Put differently: if the force field, which is based on a haystack of decisions, makes a prediction that does not agree well with experiment then the culprit needle will not be hiding in this part of the haystack.

2. FFLUX started out with a thorough, complete and systematic treatment of electrostatics. A proper understanding of the convergence behaviour ⁵⁵ of the multipolar expansion, combined with multipolar Ewald summation, ⁵⁶ enable perfect electrostatics, or at least clear monitoring of its performance. Getting electrostatics right is a rock to build the force field on; in order to make future-proof progress, at least one part has to be robust and trustworthy.

3. Polarisation is handled without polarisabilities. ⁵⁷ FFLUX is interested in the result of the polarisation process, not the process itself. This result is expressed in the values that atomic charges or multipole moments take on in the presence of a particular arrangement of surrounding atoms.

4. Perturbation theory is never used. At long-range it defines polarisabilities and dispersion coefficients but neither appear in FFLUX. Dispersion energy is subsumed in the dynamic electron correlation energy, which FFLUX captures. Fundamentally, FFLUX does so, not only between atoms that are in different molecules, but also within the same molecule. Moreover, FFLUX has access to this type of energy even within an atom. As a result, FFLUX does not depend on any external ad hoc, add-on dispersion corrections, versions of which became popular from the mid-2000s. The full energy balance is always accounted for by all the types of atomic energies ^{42 , 43} (kinetic energy as subsumed in steric energy, ⁵⁸ electrostatic, exchange and correlation energies) offered by the quantum topological partitioning. Finally, at short-range, perturbation theory becomes ill-defined and using it is a poor strategy from a computational point of view. It is better to proceed with the supermolecular approach, which handles Pauli’s antisymmetrisation impeccably. Moreover, this approach automatically delivers electron densities that are compatible with the idea of a self-consistent field. Perturbation theory reaches this mandatory property only after iterating towards it, which is time-consuming in a molecular dynamics simulation. Never mind the polarisation catastrophe, the physical breakdown of a model that is suppressed by a mathematical patch-up called damping functions.

5. Machine learning steps in to acquire and then predict how an atomic property changes as its environment changes. Originally, and long before it became the hype that it is today, we invoked ⁵⁹ machine learning to simplify the treatment of polarisation while using distributed polarisabilities. ^{57 , 60} Although starting off with neural networks we soon discovered that the completely unrelated method of Gaussian Process Regression ⁶¹ is more accurate than neural nets. The price one pays for this performance is a higher computational cost but this cost is automatically mitigated by Moore’s Law. We pioneered ⁶² the use of Gaussian Process Regression in the area of atomic potential design.

6. It is useful to be aware of four key aspects of our particular use of machine learning. They are, not in any particular order:

   1. Our work is well embedded in the field of intermolecular potentials, and more broadly, in that of quantum chemistry. This situation should be contrasted with more cavalier uses of machine learning.

   2. FFLUX is unique in learning already partitioned atomic information. In other words, alternative machine-learnt potentials use machine learning itself to partition a quantum system. In contrast, we let the machine learning operate on already obtained, physically meaningful atomic properties. Those alternative methods typically return amazing prediction errors for the whole molecule trained on. However, a serious problem is that “rogue atoms” (e.g. a hydrogen with the energy of an oxygen) may emerge because the machine learning has no clue how to create meaningful atoms.

   3. We are creative in the use and modification of machine learning methods. For example, we were the first to use the grey-wolf algorithm ⁶³ (and our improved version of it) in the search for optimal hyperparameters. Moreover, for the first time we proposed a solution ⁶⁴ for the breakdown of the principle of the maximisation of the likelihood function as a guide to improve model performance.

   4. We made a breakthrough in an old problem: how to make molecular flexibility compatible with atomic multipole moments. Before our introduction of machine learning, atomic multipole moments were only used for rigid molecules while the first and second derivatives ⁶⁵ of their intermolecular potentials required Euler angles. This restriction also hampered the field of polymorphism prediction of molecular crystals. FFLUX is able to contribute to this field ⁶⁶ for any type of molecule.

7. FFLUX treats bonded and non-bonded atoms in exactly the same way. This is a consequence of the way QCT sees condensed matter: “atoms are atoms are atoms” wherever they are. There is no special treatment for solutes as opposed to solvents; all solvation is explicit with each atom interacting with any other. Another consequence of this principle is that intramolecular phenomena or effects (such as hydrogen bonding) are treated in the same way as intermolecular ones.

8. The architecture of FFLUX is highly modular. This characteristic is an advantage because each module can be independently improved, which is not the case with classical force fields where the improvement of one energy term can spoil the overall performance. This is impossible unless there is error compensation: in classical force fields the energy terms do not represent what it says on the tin; they hold up each other like a house of cards.

9. Our in-house software (for example ^{67 , 68} ) is being streamlined, automated and documented in preparation as a one-click package, as an ongoing process of development. Most packages that generate machine-learnt potentials stay in the lab of origin, aware that a public release offers a whole new level of scrutiny and vulnerability.

10. Chemical insight accompanies the geometry optimisations or molecular dynamics trajectories executed by FFLUX. This element is precious because classical force fields fail to provide such a narrative. The honest reason is that their energy terms are not necessarily what they say they are. In contrast, FFLUX works with IQA energies, which are well-connected to chemical insight, as extracted from modern wavefunctions. The next section shows how to obtain this chemical insight rigorously, by computation and with few assumptions, if any.

Contribution 2: the Relative Energy Gradient (REG) method or how to compute chemical insight

Again, it is better to discuss only the bigger picture of the REG method, ⁶⁹ which was proposed in 2017. By not reporting the technical details here, there is a chance that REG’s universality and power may be clearer and shine on all who can benefit from it.

When chemists explain a phenomenon, such as the stability of a molecular complex or a rotation barrier, they often do the same thing, which boils down to pointing out which atoms are responsible for this phenomenon and how they are responsible for it. The “how” part of this comprehensive question typically involves one (or more) energy types; they can be written down precisely, in terms of quantum mechanical density matrices. There are four of these energy types and they are: (i) steric, (ii) electrostatic (attractive or repulsive), (iii) pure exchange (a degree of covalent bonding), or (iv) (dynamic electron) correlation (“dispersion”, often sloppily called “van der Waals”) (which, for clarity, excludes the exchange energy with which it is often lumped together).

Is such an analysis safe? Is it unique? How does it connect to the underlying quantum reality, if at all? This type of analysis is carried out all the time but the outcome is often contentious by contradicting other analyses. For example, even a simple phenomenon, such as the origin of the rotation barrier in ethane, is already controversial. Is the barrier due to steric repulsion between two hydrogens? Or is it due to hyperconjugation stabilising the staggered conformation most such that the eclipsed conformation becomes unfavourable and thereby effectively creates an energy barrier?

The first thing one needs, inevitably, is an energy partitioning method. There is a plethora of these methods, going back all the way to the 1970s. An unusually critical review ⁷⁰ reveals that they are all problematic at some point, even after having been modified to remedy a particular issue. Although published in 2015, this review does not mention IQA. This is a missed opportunity because IQA does not suffer from the typical problem that non-IQA energy partitioning suffer from. The vast majority of REG studies operate on IQA energies. However, REG is not tied to IQA, as shown by a recent study, ⁷¹ which used Natural Bonding Orbitals (besides IQA) to determine the factors accounting for reaction selectivity. The IQA energy decomposition scheme is additive, which means that the sum of all intra-atomic and interatomic energy contributions sum up to the total, original energy of the partitioned system (except for an acceptable “recovery error” due to the numerical nature of the atomic volume integration). Now, hoes does REG work? This is best explained with a metaphor, the full account ⁷² of which has been published elsewhere.

We bring in the metaphor of a football game and its analysis, in particular, that of a goal. This goal corresponds to the chemical phenomenon we are aiming to understand. Each player corresponds to an atom, such that all 22 players constitute the molecule (or system that has been partitioned). It is a human habit or even desire, both in football and in chemistry, to ascribe a phenomenon (i.e. goal or rotation barrier, for example) in terms of a few entities (i.e. atoms or players). We are aware that all entities contribute all the time, at least in principle. However, it makes sense to tease out which ones contribute the most. This is exactly what REG does.

Metaphorically speaking, REG cannot do this by looking at a single snapshot of the match but rather needs a video clip. Returning to chemistry, REG requires the availability of a relevant sequence of molecular geometries, centred around the phenomenon under study, and each geometry endowed with all atomic energies. This sequence embodies a change in the total system, and is governed by a control coordinate. An example of a control coordinate is a dihedral angle in the case of ethane’s rotation barrier, or the O⋯O distance in the water dimer. But how can the REG method point out which atom is largely responsible for the phenomenon at hand, which occurs in the whole system?

How REG works, essentially, can be described right here, in this short paragraph. First, REG calculates the (energy) gradient for the whole system as it varies as a function of a change in the control coordinate. Then REG calculates the gradient for each atom’s own energy contribution. Then REG calculates the ratio of each atomic gradient over the system’s gradient. The larger the (absolute value) of this ratio the more important the role of the corresponding atom in explaining the overall chemical phenomenon.

Several important details have been skipped in the previous paragraph, motivated by a desire to convey the wood for the trees. REG delivers a ranking of the importance of atoms in terms of how well their energetic behaviour resembles that of the total system. For example, if an atom’s energy increases, dramatically speaking, say five times more than that of the total system then surely this atom contributes much to the total system’s behaviour. Returning to the football metaphor, this means that the key action of one (or a few) player(s) can be linked to the scoring of that goal. It is he (or they) that can be associated with the overall game’s “phenomenon” of goal scoring.

The REG method has been applied in many case studies, each revisiting well-known chemical phenomena or questions, often debated. Amongst these are: the gauche effect, ⁷³ the anomeric effect, ⁷⁴ the torsional barrier in biphenyl, ⁷² the mechanism of an enzymatic reaction, ⁷⁵ halogen-alkane nucleophilic substitution (S_N ²) reactions, ⁷⁶ an advanced re-examination of the secondary interaction hypothesis in DNA base pairs, ⁷⁷ and finally halogen bonding, ⁷⁸ also in the presence of electron correlation, ⁷⁹ to name a few. REG quantified the degree of covalency in a hydrogen bond, solved the old controversy of whether biphenyl’s planar energy barrier is due to steric repulsion between its ortho-hydrogens, demystified so-called “through-space” interactions in enzymatic active sites, explained how a molecular crystal is held together in the curious absence of hydrogen bonds, and computed a pharmacophore from scratch in a way that is fully compatible with the underlying quantum mechanics.

Contribution 3: Ab Initio Bond Length (AIBL) or how to correct a pK_a experiment

Here again we will avoid technical details; we will be brief, even more so than in the previous section, in an attempt to reach the punchlines faster. However, the next paragraph makes a small detour in order to set the scene better. Some light background reading on the AIBL method can be found in an article ⁸⁰ in the popular science magazine Research Outreach.

The roots of the AIBL method lie in the construction of a Quantitative Structure Activity Relationship (QSAR) using the idea of the so-called Bond Critical Point (BCP) space. ⁸¹ A critical point is one where the gradient of the electron density vanishes. For our current purpose, a BCP is a point in space at which quantum mechanical properties (e.g. the electron density, its Laplacian, the ellipticity) are evaluated. From a practical and heuristic point of view BCP properties can characterise a bond, at least to some extent. Each BCP can then be plotted in a hyperdimensional space called BCP space, where each point compactly characterises a bond. The installation of a straightforward Euclidean distance in this space enabled the fast calculation of molecular similarity. In 2001, we showed ⁸² that this approach was more efficient and effective than that of the then familiar and popular Carbo similarity index and its variants. However, the most important discovery ⁸¹ was that this distance in BCP space made contact with experiment. In particular, the author found a very strong correlation between the experimentally determined pK_a and the distance in BCP space for a set of para-substituted benzoic acids. Actually, this major result was achieved through the sigma-constant that appears in Hammett’s equation. With BCP space now firmly anchored in physical organic chemistry, the idea of BCP space proved to be successful in many other case studies including that ⁸³ of base-promoted hydrolysis rate constants for a set of 40 esters. This method soon deserved ⁸² a name, which settled on Quantum Topological Molecular Similarity (QTMS), especially because it was able to point out the active centre, that is, that part of the molecule that is responsible for the QSAR.

Eventually it was asserted that for a set of molecules that have structural or chemical commonality there exists a novel type of linear free energy relationship (LFER). This LFER connects a single “active” bond length (i.e. in an ab initio equilibrium geometry in the gas phase) with an experimental pK_a value in aqueous solution. This powerful observation led to the acronym AIBLHiCoS (Ab Initio Bond Length High Correlation Subsets) to coin this pK_a prediction method, ⁸⁴ which was later just called AIBL. One just needs to geometry-optimise, in the gas phase, a set of molecules while getting away with modest levels of theory. The proposed method can isolate erroneous experiments, operate in non-aqueous solutions and at different temperatures. Moreover, the existence of active fragments is demonstrated in a variety of sizeable biomolecules for which the pK_a is successfully predicted.

In subsequent work ⁸⁵ we presented solutions to some of the more complex challenges in pK_a prediction, namely that of (i) multiprotic compounds, (ii) macroscopic values for compounds that tautomerise, and (iii) compounds with more than 50 atoms. More spectacularly, we re-measured ⁸⁵ a pK_a that AIBL identified as wrong. Lo and behold: the new experimental value agreed much better with the computational prediction than before. This is a theoretical or computational chemist’s dream. In particular, we identified that the literature values for drug compounds celecoxib, glimepiride and glipizide were inaccurate. Our newly measured experimental values matched our initial predictions to within 0.26 pK_a units, whereas previous values were found to deviate by up to 1.68 pK_a units.

The accurate prediction of aqueous pK_a values for tautomerisable compounds is a formidable task, even for the most established in silico tools. Empirical approaches often fall short due to a lack of pre-existing knowledge of dominant tautomeric forms. In fairly recent work ⁸⁶ AIBL predicted pK_a values for herbicide/therapeutic derivatives of 1,3-cyclohexanedione and 1,3-cyclopentanedione to within just 0.24 units. Our AIBL model uses a single ab initio bond length from one protonation state. Moreover, AIBL is as accurate as other more complex regression approaches using more input features, and even outperforms the popular computer program Marvin.

There is other elegant work ⁸⁷ that predicts pK_a within the QCT context based on electron localisation-delocalisation matrices but currently the landscape of pK_a predictors is dominated by very fast but very crude methods. This status quo sets a paradigm that industry has accepted and maintains with disappointing inertia, even if better results can be achieved with computer power already existing in that same industry. Gas-phase ab initio optimisations are routine and cheap in the sense of a CPU-minute scale. Over the years AIBL has shown an incredible robustness in terms of theory and experiment always converging. If ever there was a disagreement between theory and experiment it was because AIBL focused on the wrong conformation, or identified a new LFER with a tighter molecular skeleton, or experiment was simply wrong. In all case studies, things always fell into place. Finally, it is useful to think of AIBL as in the context of a pick-up sticks game such as Mikado. Each stick corresponds to a high-correlation subset, that is, a congeneric set of molecules housing a bond whose (gas-phase) length magically correlates with pK_a in solution. Chemical space then falls apart into a large bundle of pick-up sticks, each pointing in an appropriate direction in the plotting space of pK_a versus bond length. It is fair to say that AIBL is strong and reliable because it makes predictions and most likely works for the right reason as it is based on a minimal number of descriptors and without exception.