Quantum chemistry and large systems – a personal perspective

Introduction

I want to begin this article with a disclaimer: I was asked to provide a perspective article on the quantum chemistry of large molecules. Hence, I kindly ask the reader to treat this manuscript as an opinion piece and not as a scientific article. The statements made in this article merely reflect the author’s opinion on the subject and the reader is invited to disagree with them. Given the nature of this article, no attempt will be made to provide a comprehensive bibliography or to justify every statement made with extensive recourse to the literature.

Quantum chemistry might be broadly characterized as the application of the laws of quantum mechanics to the phenomena of chemistry. This implies a large overlap of our discipline with the field of theoretical physics. In fact, one of my lifelong friends, a deep fundamental physicist, once stated during an intense debate that “all you are doing are trivial applications of quantum electrodynamics” (paraphrased). I could not really disagree with him at the time. However, even if the point of departure of quantum chemistry as a discipline may, at first glance, look like an engineering exercise, the challenges that quantum chemists face run far deeper. In his now famous quote, Dirac stated ¹

The fundamental laws necessary for the mathematical treatment of a large part of physics and the whole of chemistry are thus completely known, and the difficulty is only that the exact application of these laws leads to equations much too complicated to be soluble

It is less well-known that he followed this truly monumental statement by saying:

It therefore becomes desirable that approximate practical methods of applying quantum mechanics should be developed, which can lead to an explanation of the main features of complex atomic systems without too much computation.

Despite the fact that Dirac’s statement (referred to as Dirac’s prophecy below) was incredibly bold, the historical development of our discipline has largely proven it to be correct. It is also truly remarkable that Dirac used the word “computation” long before digital computers arrived on the scene, let alone ones that can perform billions of operations in a second.

However, Dirac’s statement (like the one from my friend) is only emphasizing the engineering aspect of quantum chemistry that might be summarized by asking the question “what does it take to generate a sufficiently accurate solution to the quantum mechanical equations for systems relevant to chemistry”. This implies that once these solutions are available to sufficient numerical precision, all scientifically relevant questions of chemistry are solved. It is the aim of this article to speculate on how close we are today to fulfil Dirac’s prophecy in practical terms and also to ask whether all we should be longing for as quantum chemists are “sufficiently accurate” numbers? In doing so, many sentiments will be echoed that were discussed in far greater depth in a recent series of articles by Hoffmann and Malrieu. ^{2 , 3 , 4}

A brief look into the evolution of quantum chemical methods

Dirac’s profound statement was made amidst the excitement created by the advent of quantum mechanics by Heisenberg ⁵ and Schrödinger ⁶ in 1925 and 1926 respectively and Dirac’s subsequent proposal of his ground breaking equation marrying quantum mechanics and relativity in 1928. ⁷ The formalism of quantum mechanics developed very quickly during this time and stands essentially unchanged today.

Unfortunately, it turned out that closed-form solutions to either the Schrödinger, let alone the Dirac equation, were confined to model systems like the harmonic oscillator, the rigid rotator and the hydrogen atom. Hence, a rigorous application to chemistry on the basis of these equations was impossible and consequently, quantum chemistry started to evolve in terms of conceptual models like the Heitler-London treatment of the chemical bond, ⁸ Pauling’s Valence Bond theory, ⁹ Mulliken’s Molecular Orbital (MO) theory, ^{10 , 11} Bethe’s treatment paving the way to crystal-field theory of coordination compounds, ¹² or Hückel’s theory of aromatic compounds. ¹³

The first numerical calculations started as early as 1927 with Hartree’s calculations on atoms, ¹⁴ Fock’s extension of it to include the exact exchange, ¹⁵ Burrau’s calculations ¹⁶ on H₂ ⁺ and the calculations on H₂ by James and Coolidge in 1933. ¹⁷ The linear-combination of atomic orbitals (LCAO) theory was formulated by Roothaan in 1951 and stands essentially unchanged today. ^{18 , 19} Of course, a major development for quantum chemistry was the proposal of density functional theory (DFT). DFT has been around almost as long as quantum mechanics itself with the early contributions from Thomas and Fermi, ²⁰ Dirac ²¹ and Slater’s exchange (formulated as a simplification of the Hartree–Fock method). ²² More than three decades later, Kohn and co-authors gave it the formal basis in use today (Hohenberg-Kohn, ²³ Kohn-Sham ²⁴ ). However, the large-scale adoption by DFT in chemistry only started with the early molecular DFT program by Baerends, ²⁵ the important applications by Ziegler ²⁶ and the contributions of Pople et al. ²⁷ and Handy et al. ²⁸ who showed how to incorporate the Kohn-Sham formalism into the Hartree–Fock framework by then familiar to quantum chemists.

However, the limited capacity of digital computers and the enormous effort to program them and run the programs at the computer centers at the time made progress in the field somewhat slow. I have vivid memories of Prof. Peyerimhoff telling me how in the 1960s she travelled with suitcases full of punch cards that were necessary to even implement a minimal basis set calculations on small hydrides ²⁹ and how these calculations could be ruined by border control opening the suitcases and messing up the order of the punch cards.

From today’s perspective, one cannot help but admire how much vision and foresight the pioneer’s of quantum chemistry had to assume that all of this could ever lead to something that is now an indispensable research tool in all branches of chemistry and its many neighboring disciplines such as material sciences or biochemistry. It is clear, that conceptual progress was fast while the rigorous implementation of these concepts was comparatively slow. In 1991, Zerner, a theoretician with considerable in- and foresight, stated:

Although it is true that computers are becoming faster, and our knowledge about electronic structure theories and calculation is increasing, it is unlikely that this increase in capability will ever catch the experimentalists’ interest in larger and larger molecular systems

While referring to Hartree–Fock calculations with double-zeta basis sets. ³⁰ Today, DFT calculations with thousands of atoms and more than 100 000 basis functions are possible (Fig. 1) thus demonstrating the limitations that were still present at the time have definitely been overcome.

Fig. 1:

Among the largest DFT calculations possible today on contemporary computers feature more than 5000 atoms (an Octamer of the small protein Crambin) and well over 100 000 basis functions. These calculations were performed as demonstrative examples with the ORCA program, version 6.1.

Given the decade-spanning technological limitations, it is understandable that much of the scientific efforts in the quantum chemistry community were focused on overcoming the limitations posed by the hardware. In fact, the design of clever algorithms that provide accurate approximations to the quantum chemical equations has been a major research focus for decades. To illustrate this point, I can quote a recent computer experiment ³¹ in which we recompiled and linked a 25 year-old version of the ORCA code and compared its performance to the latest iteration of the program. ³² The result was that the algorithmic speedup for the same calculation on a 176 atom molecule was about a factor of 200 while in the same period the single-thread hardware speedup is only about a factor of 7. This may well be taken as a sign that the community’s search for more efficient algorithms has met with significant success.

The success of the algorithm design efforts in the community are probably even more apparent in the field of correlated ab initio calculations. It is a fascinating, yet frustrating aspect of quantum chemistry that our field is forced to reach extremely high accuracy in the solution of the quantum chemical equations. For example, a comparison of the Hartree–Fock energy of the Ne atom at the basis set limit with the sum of the first ten ionization potentials, one finds that the Hartree–Fock energy is 99.8 % accurate and with a scalar relativistic correlation this error of 0.2 % is further reduced. Yet, the remaining error of 0.2 % or less is still very large on a chemical scale and, according to the definition of Löwdin, ³³ constitutes the correlation energy. “Chemical accuracy” is commonly quoted as 1 kcal/mol in relative energies. The correlation energies of larger molecules are on the order of 10 (atomic units, Eh) which corresponds of 6127 kcal/mol. Thus, if one wants to calculate chemically accurate energies without heavily leaning on error cancellation, one is forced to approximate the correlation energy to an accuracy of about 99.9–99.99 % which is a significant challenge.

Unfortunately, the methods designed to calculate correlation energies in molecules (broadly categorized as configuration interaction (CI), nth order many-body perturbation theory (MPn) or coupled cluster theory (CC)) scale with high powers of the molecule (MP2 – O(N⁵), CISD and CCSD – O(N⁶) and ‘gold standard’ CCSD(T) – O(N⁷), where N is a measure of system size and S, D and T refer to single, double and triple excitations relative to a reference Slater determinant). ³⁴ Clearly, these steep scaling laws preclude the application of correlation theories to large molecules with dozens if not hundreds or thousands of atoms. It may be considered surprising that the answer to that challenge has often been to throw more computational resources at the problem in order to extend the applicability of the methods to larger molecules. However, one has to consider that treating a system that is twice as large as another one that one has already treated with, say, CCSD(T), requires 128 times the computational resources. Thus, even scaling up the computer power by a factor of 100 (disregarding the formidable challenges in software design to also efficiently make use of these resources) will only increase the applicability of the methodology by a factor of 2. Furthermore, such very large-scale computing facilities will not be available to the bulk of computational practitioners, at least not for the foreseeable future.

On the other hand, it has been understood since the early days that the correlation effects are of relatively short range. In fact, the leading term in the correlation energy is the London dispersion energy which falls off as R⁻⁶ with the distance between two electrons. Eventually, the intrinsic scaling of the correlation energy with system size is linear. Consequently, the idea to take advantage of the localized nature of electron correlation is as old as correlation theory itself. However, it has been found to be surprisingly difficult to devise successful algorithms that are practically doable and retain the target accuracy of 99.9–99.99 %. Without attempting an even cursory review of the many approaches that have been proposed, Pulay’s early work on local correlation ³⁵ clearly provided a framework for many of the work that followed. ^{36 , 37 , 38 , 39} Pulay defined correlation domains of projected atomic orbitals (PAOs) that span a subspace of the virtual space for each pair of occupied Hartree–Fock orbitals. These domains become asymptotically independent of system size and, with proper algorithm design, lead to linear scaling algorithms. This was first realized in the extensive work of Werner and co-workers ^{36 , 40 , 41 , 42 , 43} on PAO based correlation theories. These methods were recovered about 98–99 % of the correlation energy. ⁴⁴ While this was a very significant accomplishment, the remaining error proved to be still too high for the applications that computational chemists had in mind. This problem was eventually addressed with the development of domain-based pair-natural-orbital (DLPNO) ^{37 , 38 , 39} methods (based on pioneering work of Meyer ^{45 , 46} ) that led to a compaction of the domain sizes and are able to recover 99.9 % of the correlation energy while retaining linear scaling with system size. ⁴⁷

If these linear scaling correlation theories (or their many successful variants ⁴⁸ ) are combined with the linear-scaling Hartree–Fock algorithms that were extensively developed in the 1990s and early 2000s, ^{49 , 50 , 51} it became eventually possible to treat systems with hundreds and even over a thousand atoms with accurate quantum chemical methods. The first calculation on a small protein, Crambin, at the CCSD(T) level in 2013 ⁵² was considered to be a milestone in this direction (Fig. 2). Since then, local coupled cluster methods have found their way into mainstream computational chemistry.

Fig. 2:

Press echo to the first CCSD(T) calculation on an entire protein, Crambin in 2013.

Algorithmic challenges

Not withstanding the enormous algorithmic challenges that still need to be addressed in the field of linear-scaling highly-correlated wavefunction theories, one might be tempted to argue that the development of such methods represent a realization of Dirac’s prophecy by being able to solve the quantum mechanical equations that govern chemistry to sufficiently high accuracy. However, one might well question whether Dirac’s assertion that these equations govern all of chemistry is truly a fair assessment?

At least in this author’s opinion, the answer to this question is the resounding “no”. Rather, there is a multitude of conceptual and practical problems, even if one takes the over-optimistic view that the problem to solve the Schrödinger equation for large molecules has been achieved.

First of all, one has to admit that local correlation calculation on whole proteins are still resource intensive. Hence, while single point calculations on molecules with hundreds of atoms are perfectly doable, one is right to ask the question of what can be learned from such a calculation? In fact, in our calculation on the Crambin molecule with DLPNO-CCSD(T), ^{39 , 52} we only had a single question: are we able to see the end of the calculation without crashing the computer? It was very satisfying to receive a positive answer to this question. However, we did not learn anything about Crambin and neither did we try to. At the time, the calculation could be considered as a “digital weight lifting” exercise. Many other demonstrative calculations of a similar kind on even larger molecules followed performed by a variety of groups world-wide. These calculations can be regarded as impressive demonstrations of algorithmic achievements. They do, however, not immediately translate to revolutionary new opportunities for computational chemistry.

There are at least three or four requirements for accurate chemical predictions beyond being able to do large-scale closed-shell single point calculations at fixed geometry.

First of all, it is important to point out that computational chemistry required a far larger arsenal than just single point, closed shell energies. In order to be really useful for chemistry, the methods need to be able to treat open-shells, excited states, extended systems and solvated systems. The methods need to be able to provide analytic gradients for geometry optimizations and response properties for connection to molecular spectroscopic data. Finally, one might also need to be able to treat multi-reference systems in order to guarantee applicability of the methodology throughout the periodic table. These are not just extensions that more or less follow trivially once the proof of concept has been achieved with a closed-shell single-point code. In fact, these extensions often require years of work and are significantly more complex than the original methods they are based on. A point in case is the analytic gradient of local correlation methods that are much more complex than the corresponding canonical methods because one is forced to incorporate the derivative of each and every approximation one made in the energy calculation as a constrained into the gradient or otherwise, the gradients will be incorrect. ^{53 , 54 , 55} To the best of my knowledge, the DLPNO methods are the only local correlation methods for which this has been at least approximately achieved, even if neither the DLPNO-CCSD(T) analytic gradient or response properties are available at this time.

Secondly, while focusing on overcoming all the algorithmic problems one faces in solving the Schrödinger equations for large molecules, it is easy to lose track of the fact that chemical reaction energies contain other significant contributions besides the electronic energy. The way that solvation energies, vibrational corrections and entropic corrections are typically calculated in quantum chemistry is rather simplistic and relies on ideal gas thermodynamics as well as continuum solvation models, both are of rather limited accuracy in real-life applications. Hence, one may question the wisdom of obsessively nailing the electronic energies to fractions of a kcal/mol while overlooking errors in the entropy or the solvation on the order of tens of kcal/mol. This is exactly what happened to us after the development of the DLPNO-CCSD(T) method. With a lot of excitement and very high expectations, we started to do real-life computational chemistry with these methods, only to find out that our predictions were frequently not that much more accurate than the ones delivered by standard DFT methods. The reasons were quickly determined to be in the solvation and entropic energies that were still calculated with standard methods. Hence, the lesson learned was that for increasing the accuracy of quantum chemical predictions in real-life applications, it is not enough to be a specialist in the calculation of electronic energies, or solvation treatments, or entropy calculations etc. Instead, all sources of error must be considered together – often requiring the collaboration of multiple specialists. In fact, in modelling enzymatic reaction mechanisms, it has been the experience in our department that – more often than not- it is more important to pay attention to the model of the chemical system at hand than to push the electronic structure treatment to the most extreme rigor. Thus, it is more important to get the number of water molecules in the active site, the hydrogen bonds and protonation states right than it is to extend the basis set from triple- to quadruple zeta or to worry about whether triple excitations are enough for the problem at hand.

Third, solving the Schrödinger equation at fixed geometry is also only one of the challenges one faces in quantum chemistry. Another problem of the grand-challenge class is conformational complexity. In fact, the conformational space of molecules increases exponentially with system size. Large molecules typically feature a very large number of close lying minima and, at finite temperatures there will be a dynamical motion where very large conformational changes may be driven along very soft modes. Complex reaction partners may assume many different interaction geometries that one might not be able to successfully guess by inspection. These are all problems that are genuine to large molecule problems that have essentially no counterpart in small molecule quantum chemistry.

There are many different approaches available to deal with conformational complexity. Traditionally, of course, these molecular motions are a domain of molecular dynamics, either in its ab initio form in combination with first-principles electronic structure methods or its classical form on the basis of force fields. Alternatively, in recent years there has been significant progress in finding low-lying conformers of complex molecular systems using approaches like conformer-rotamer-sampling-tool (CREST) ⁵⁶ or the global optimization algorithm (GOAT ⁵⁷ ) and related global optimization algorithms. Such approaches can also be used in relation with molecular docking or explicit solvation approaches.

The principal problem with all of these approaches is that they will require a very large number of energy and gradient evaluations which means that the most accurate quantum chemical approaches cannot be used for that task and one has to rely on lower cost quantum chemical methods such as the very successful XTB method ⁵⁸ or the most efficient DFT variants. However, these methods do not deliver the same kind of accuracy and reliability that the accurate quantum chemical methods can achieve. Hence, one is forced to make the unfortunate choice between running accurate calculations on only a few conformations (which may miss essential aspects of the system’s chemistry) or low accuracy calculations on a representative ensemble (which may introduce unphysical artifacts).

Given the difficulty to combine accuracy and efficiency, it is not surprising that many contemporary research efforts are focused on the development of multi-scale methods in which exploratory calculations are done with low-cost methods followed by a hierarchy of selection and filtering steps that eventually leads to only a few expensive high-level calculations being necessary. One principal difficulty in that endeavor is the heterogeneity of the programs used to carry out the specialized individual calculations since there probably is not a single monolithic program package that can carry out all of the steps in a user-friendly and robust manner.

Very closely related to the problem of conformational flexibility is the important subject of chemical reaction design. Here, one addresses the challenging and important problem of theory guided retro-synthesis with the goal to predict a valid and efficient pathway to the synthesis of a given target structure from starting materials that are as simple and cheap as possible. For centuries, this has been the domain of highly experienced synthetic chemists who are drawing from a lifetime of experience and literature knowledge to select the most likely synthetic route. Clearly, the challenges in predicting an entire synthetic route for a complex natural product is a dauntingly complex task since not only must the algorithm foresee possible side products and their reactivity but also accurately estimate the height of the transition states involved and the influence of solvent effects and temperature. Given this extremely high degree of complexity, recent progress in this field has been impressive. ^{59 , 60}

Please allow me to close this section with the parenthetical remark that the development of successful approximate quantum chemical methods is as much an art as it is science. During algorithm design, there are a large number of decisions to be made and one may meet with a strong temptation to take shortcuts. Over the years, there are two golden rules that have emerged for me personally that I try to respect when working on algorithms that approximate complex quantum chemical equations (Fig. 3). While both rules appear to be trivial (and to some extent are trivial), there are many developments that have failed these tests. The first rule is that the best way to achieve real performance gains is to work on the slowest step in the algorithm. For example, speeding up a step that only takes 1 % of the overall wall clock time by a factor of ten results in an overall performance gain of 0.9 % which is inconsequential for the users of the algorithm. The second rule states that obtaining a bad result fast is useless. The literature appears to contain a significant number of studies that do not state the accuracy of the results when the authors want to impress with speed and vice versa do not state execution time when the goal is to impress with accuracy. In practice, one needs to set oneself an accuracy goal, then tweak the algorithm to meet this accuracy and then try to make it as efficient as possible without sacrificing any additional accuracy. For example, developing a coupled-cluster approximation that makes errors of hundreds of kcal/mol is pointless. With that kind of accuracy, one would have been better off with a minimal basis set Hartree–Fock calculation. At the end of the day, the users of the program (provided the algorithm gets released to public in one way or another), will find out the true strengths and weaknesses of the algorithm anyways.

Fig. 3:

Two golden rules for developing approximate quantum chemical algorithms.

Conceptual challenges

The foregoing discussion has focused on some of the important algorithmic or “engineering” aspects of applying quantum chemistry to large molecular systems. However, there are many more questions to be considered that extend far beyond the numerical accuracy of the calculations. To put this in perspective, we quote Wittel and McGlynn who wrote:

Quantum chemistry is faced with two fundamental problems: How to reduce chemistry, in a practical way, to numerics; and how to avoid doing so and still benefit from theory.

This is an important and balanced statement – it acknowledges the importance of being able to compute “good” numbers but it also contains a warning to reduce our discipline to a branch of what Malrieu once referred to as “numerism”. ⁶¹

In present quantum chemistry benchmarks studies play an important role. In fact, benchmark sets are being devised in large numbers in order to test the accuracy of new emerging theoretical methods for a wide variety of properties and chemical bonding situations. Personally, part of me welcomes these studies with open arms, while another part remains cautious. On one hand a culture of rigorous testing and evaluating new methods is nothing but positive and the culture that has emerged in chemistry is an important development. The extensive benchmark data accumulated by the community are definitely very helpful to choose an appropriate method for a given application. On the other hand, there is a certain danger that benchmarking becomes an end in itself and reduces theoretical chemistry to error statistics. Is it really meaningful if one method is, on average, 0.1 or 0.2 kcal/mol more or less accurate than another method against this or that test set? In my opinion, the answer is “no”, in particular given the fact that the results obtained in benchmark studies do not always (and in some cases even very much) do not translate into real-life applications. For example, a few years ago we calculated a transition state with a method that was praised to be one of the most accurate density functionals available – yet it had an error in excess of 10 kcal/mol relative to the known experimental numbers. ⁶² Had we simply relied on the belief that the benchmark results demonstrate that this particular method is always reliable to the quoted, say, 2.3 kcal/mol, complete wrong predictions would have been obtained.

This example brings us to the overwhelmingly important subject of falsification. Personally, I take the point of view that nothing is a certain way because somebodies calculations said so or because a method does deliver a certain accuracy against a given benchmark set. While in small molecule, gas phase quantum chemistry calculations can often (but not always) be pushed to a point where they are likely as reliable as the best experiments, the same is not true for the chemistry of larger molecules studied in condensed phases. In particular the frequently encountered statement “the reaction mechanism was shown by DFT to be this or that” is completely antithetical to the responsibility of a scientist to self-critically evaluate one’s conclusions by trying to disprove the hypothesis put forward through experiments. I confess to be very strongly influenced by Popper’s philosophy of science. In one of his central works, Popper stated: ⁶³

[…] but I shall certainly admit a system as empirical or scientific only if it is capable of being tested by experience. These considerations suggest that not verifiability but the falsifiability of a system is to be taken as a criterion of demarcation.

Thus, Popper takes the point of view that things that cannot be tested by experiment do not belong to the domain of science. Personally, I do strongly support the concept of falsification, in particular in the framework of computational chemistry. Thus, the fictitious statement above about a calculated reaction mechanism serves very well as a working hypothesis but not as a proof of correctness. Positively speaking, one can get excellent ideas for new experiments or testable hypotheses from theoretical calculations, but it is important to be clear about the status of such calculations.

However, I do not agree with the generality of Popper’s statement. Rather, I regard the un-testable aspects to be an essential part of our disciplines culture. To this end it is very useful to divide the quantities that can be calculated by quantum chemistry into “observables” and “interpretation aids”. Observables can be compared to experimental numbers and objective statements can be made about their accuracy. Hence, it is readily possible to differentiate a good from a less good theoretical result (Fig. 4).

Fig. 4:

Categorization of quantities that can be calculated with quantum chemistry as “observables” and “interpretation aids”.

On the other hand, “interpretation aids” are quantities that cannot be measured and therefore cannot be associated with objective existence. According to Popper, these interpretation aids are not part of science. However, these are quantities that are central to chemical thinking – partial charges, hybridization, resonance, ligand fields, pi-systems – even orbitals themselves (outside 1-electron systems) are interpretation aids. It would be difficult to have a chemical conversation while denying that any of these concepts of chemistry have any status or meaning. A statement often attributed to Max Planck is

Experiments are the only source of knowledge at our disposal. The rest is poetry, imagination

Although the exact location of the quote seems to be difficult to locate (it is quoted in quoted in Adv. Biochem. Psychopharm., 1980, 25, 3). I take the liberty to equate the “poetry and imagination” that Max Planck was speaking about with the “interpretation aids” mentioned above. They are important for scientists to be chemically creative. Hence, it is also meaningful to try to find these concepts in calculations, even if the connection is only approximate. However, in this author’s opinion, it is not worthwhile to fight tooth and nail over which of these concepts is “better” than another concept. At the end of the day, it matters whether the concepts give scientists new ideas, inspire new experiments or lead to new theoretical methods.

Returning the numerical aspects of our discipline, it is probably not possible to overestimate the importance of the question of “what do we even mean by ‘accuracy’”. Any level of accuracy always depends on the choice of reference data. It does matter whether this reference point is an experimental number, a perceived ‘exact’ result or one’s own expectations of what the results should be. In the case of small molecules, this question can often be addressed in a rather rigorous manner when highly reliable experimental data are available that agree with high-level calculations. For large molecules, usually neither are available. The problem of how to tell apart a good theoretical from a bad and misleading theoretical result is much more ambiguous.

This subject deeply touches on the fundamental principles of falsification of theoretical results. Clearly, there are many other properties than the total energy that can be employed to falsify theoretical results. As argued elsewhere, ⁶⁴ among the possible ways to connect theory and experiment spectroscopic properties are particularly powerful and versatile. To me, science was and is always about having what Prigogine called a ‘dialogue with nature’ ⁶⁵ as opposed to having a monologue about it. Since spectroscopy is the experimental way of studying electronic structure, it is uniquely suited to implement this dialogue. Every spectroscopic method is sensitive to a different detail of the system’s geometric and electronic structure and this allows one to ask tailored questions that are far more detailed and fine-grained than the total energy alone – the grand total of everything that happens in the system – allows.

Related but not independent of the subject of falsification is the important question of “how accurate have the results to be in order to be useful for the project at hand”. There appears to be a general feeling in the computational chemistry community that “more accurate = more good”. Thus, theoretical results have higher credibility if they are obtained with a higher-level (and likely more expensive) electronic structure method. However, a theoretical result is not automatically more valuable because a bigger computer has worked on it for a longer time compared to another person’s calculation. In fact, it is worth some reflection of how accurate the results need to be in order to be useful for solving the problem at hand and then focus the attention and resources to those calculations where accuracy really matters. For example, if one computes a highly exothermic reaction energy – does it really matter whether one computes −51 kcal/mol or −47 kcal/mol? Most of the time, probably not. On the other hand, in trying to predict the enantiomeric excess in a stereospecific reaction, a difference of 0.5–1 kcal/mol in the relevant transition state energies has a sizeable impact on the prediction. In computing the spectra of small gas phase species where the experimental absorption spectra are resolved to a linewidth of better than 1 cm⁻¹ one needs to pay a lot more attention to vibrational fine structure compared to a highly heterogeneously broadened condensed phase spectra with a linewidth of hundreds of wavenumbers. It is often more important to get the pattern of strong vs. weak transition correct than it is important to calculate accurate absolute transition energies, to name only a few situations.

A related subject concerns to be mindful about the questions one is seeking to answer in a given study. As a theoretical chemist, one is often confronted with the question “can you calculate this molecule?”. My immediate reaction to this question is to ask back “what exactly do you mean by ‘calculate that molecule’? Calculate what? What is the problem that we are trying to address?”. If the only purpose of the calculation is to fill in some numbers or pictures for a publication that already has the conclusion part written, my personal motivation to engage is very low. Rather, it is important to be clear about the questions one wants to answer in a given theoretical study. These questions will depend on the size of the molecules being studied. For small systems and gas phased studies one can address even the most minute details of their rovibronic spectra or the intricacies of a Rydberg series. On the other hand, for a large molecule, it becomes more and more difficult to ask highly specific, high-resolution questions and the approach one is usually forced to take is more coarse-grained.

Calculation vs. understanding

The difficult relationship between “calculation” and “interpretation” has been a hotly debated subjected throughout the history of our discipline. The first complaints about sacrificing the latter for the former are almost as old as the ability to perform any meaningful numerical calculation in quantum chemistry. Coulson’s historic after dinner speech comes to mind, ⁶⁶ although the famous sentence “Give us insight, not numbers” is neither in the transcript of his speech nor is it apparently found among the writings of Mulliken, who is sometimes credited with that statement too (Prof. Peyerimhoff informed me in a personal communication ca 2019 that she attributes the statement to Robert Mulliken).

Personally, my position always was that it is not necessary to choose between “either-or” but that insight and numbers are not mutually exclusive. Insight does not automatically emerge from numbers, or putting it in Thom’s words: ^{2 , 67}

To predict is not to explain. It is only by accepting the risk of error that we can harvest new discoveries.

Thus, the mere fact that a calculation agrees with a given measurement to a given desirable or desired accuracy does not mean that one understands why and how the calculation came to this result. Neither is one automatically able to infer how future calculations on related systems are likely going to come out. Wigner highlighted this in a somewhat pointed way by saying:

It is nice to know that the computer understands the problem. But I would like to understand it too.

Again, it seems to be difficult to track down the exact source of this quote ⁶⁸ but it certainly echoes the sentiment of those researchers that have a desire to not stop at merely putting numerical results in tables but wish to gain some intuitive understanding of the problem or system studied. The latter will invariably have to involve a scientific model. According to Primas, ⁶⁹ such a model is a deliberate oversimplification that aims at highlighting certain key aspects of reality (or the higher-level calculation that it is based on). Invariably, such models involve adjustable parameters that can be used to numerically parameterize a measurement or a set of measurements. If the model is good, an unambiguous connection between first-principles and the model parameters can be established. One such model in quantum chemistry is the spin-Hamiltonian. In other models, the model involves mathematical expressions that, when taken literally, will produce absurd values for the model parameters. Such models are the Hückel model (where the literal expression would produce a positive resonance integral while the model requires it to be negative to explain experimental observations), or crystal field theory (where the model predicts the correct symmetries of a set of many particle states but leads to values of ligand field splittings that are unrealistic). The great value of models is that they provide insight that leads to an intuitive understanding of the dominant factors that influence a given molecular property across a series of compounds. Michael C. Zerner is credited with the following sentence (paraphrased):

… the best calculations are those that, after the fact, I realize I wouldn‘t have needed

And similar statements are attributed to Dirac with respect to understanding physical equations. As strongly emphasized by Primas, such models always create a language in which scientists can creatively think and communicate. It is, in fact, difficult to imagine how coordination chemistry would have evolved without the language of crystal and ligand field theory or the study of aromatic compounds (or even the concept of aromaticity) without the language provided by Hückel theory. This thinking in models is absolutely essential for quantum chemists to be able to talk chemistry with chemists.

In order to illustrate that point in a less than perfectly serious manner, let us eavesdrop on a conversation that our quantum chemist “TradQC” might have with another colleague “CChem” who is an accomplished coordination chemist, interested in transition metal chemistry.

In this example, “CChem” and “TradQC” were having a productive dialogue because “TradQC” was able to talk to “CCHem” in a language that is familiar to them. Being able to use that language rested on “TradQC” making some compromises in theoretical rigor but gained pictorial insight. No complicated or expensive calculations were necessary to reach this state of discussion. However, there was enough mutual understanding gathered that it is clear what kind of rigorous calculations and with which focus will be necessary in order to help “CChem” with the design of their chemistry.

This kind of intuitive understanding, that might be just “gently guided” by calculations, is clearly not reached by numerical calculations alone. The latter simply provides a single result for a single molecule in a single geometry at a time and any deviation from this situation is an entirely new game. Hence, let us contrast the above conversion with another conversation that colleague “CChem” might have with another colleague “NumQC”, a specialist for accurate multi-reference electronic structure calculations:

Clearly, “CChem” and “NumQC”, despite impeccable intentions from both sides, had a far less productive discussion because they were not able to find common ground. Rather all possible information gain was delegated to large-scale computations which, at this stage, are not helpful to guide “CChem”’s creative process.

The above (exaggerated) example, is not meant to imply that purely numerical results are not useful. Quite to the contrary, in certain situations the numerical result is all that matters and its intuitive or qualitative interpretation may be of secondary or tertiary importance. Situations like this might be met when trying to find the most stable polymorph of a drug molecule or when one works on the spectra assignment of lines in a ultra-high-resolution spectrum.

Thus, purely numerical calculations without any attempt at interpretation definitely have their time and place in the arsenal of quantum chemistry and should not be discarded on the basis of lacking an interpretation. However, gaining an intuitive understanding alongside satisfactory numerical results is, arguably, a more satisfactory research goal. As we have argued in an article published in 2019, this is readily achievable today. ⁷⁰ It will, however, require the willingness of the investigator to engage in an interpretation effort and, to some extent, also the engagement of the method developers to create specific tools that facilitate the interpretation of the numerical results in terms of a model. Examples for such efforts are the ab initio ligand field theory (AILFT) ⁷¹ or the local energy decomposition analysis (LED) ⁷² among very many other approaches that were developed in the community over time.

Physics based approaches vs. machine learning

The article should not be closed without at least a cursory discussion of the more modern development of artificial intelligence (AI) and machine learning (ML). These methods should be contrasted with approaches based on exact or approximate solutions of the Schrödinger equation by referring to the latter as “physics-based approaches” (PBAs). Rather than attempting to solve quantum mechanical equations AI and ML models aim at producing equivalent results by anticipating the results of PBAs using neural networks among other advanced approaches. These methods are currently creating a scientific revolution that is likely to irreversibly change the scientific landscape, in quantum chemistry and beyond.

While some ML models aim at replacing traditional quantum chemistry and PBAs for good, others are focusing on replacing certain, more localized aspects of PBAs with neural network predictions. Clearly, large-scale efforts are underway to, for example, create machine learned density functionals or to use ML potentials in large-scale molecular dynamics calculations. While the best ML models can presently not compete with the accuracy of the best PBA’s, progress is extremely fast and the final outcome of these developments are difficult to predict with a high degree of confidence today. However, it is probably safe to predict that there will be all flavors of AI and ML learning applications in quantum chemistry ranging from ML replacing established PBAs to AI and ML assisted hybrid approaches. At this point in time ML models are still far from being able to deal with the fine details of electronic structure as they arise from multiplet structures, complex spin-coupling, magnetic phenomena or complex multi-reference problems. It is, however, probably a matter of time before these more intricated situations can also be handled by ML and AI. At least for the time being, PBAs will not be irrelevant and will at least serve to train the neural networks in the ML approaches. It remains to be seen how far one will be able to go outside the training set before the ML approaches become unreliable. Engineering a high degree of robustness and error control into these models is certainly one of the more challenging aspects of this research direction.

Coming back to the question about insight-vs-numbers discussed in the previous paragraph it seems clear that ML models are created to have their strength on the side of numbers since it is impossible or at least challenging to dissect the results of ML predictions in terms of physical building blocks or intuitive models. Whether this is considered to be an advantage or disadvantage is certainly a matter of one’s personal research philosophy.

One specific aspect, that I have reflected on in a recent contribution is the problem that the quantum chemistry community has with legacy code. Many of the dominantly used program packages in our community are several decades old and not well-suited to keep up with the rapid development in computer hardware and the emerging massively parallel architectures that will soon by commonplace. The development of sophisticated algorithms and their efficient implementation requires years of effort and many of them (e.g. the implementation of a coupled-cluster geometry gradient) have been achieved many times before and do not constitute valid projects for Ph.D. students or postdocs. In this situation, it is my belief that automatic code generation is a valid answer because many of the complex algorithms can be efficiently implemented using machine generated code. ⁷³ If this is done in a sustainable and “deeply-integrated” manner, machine specific code can be generated without spending extended amounts of human time rewriting legacy code. However, the currently available AI models are rapidly improving to the point where rather sophisticated code can be designed and implemented by them with a relatively small amount of human guidance. If progress keeps up at the present pace, it is entirely conceivable that full featured quantum chemistry program packages of the future may well be designed and written entirely by AI models. How the job of quantum chemical method developers will look under such circumstances remains to be seen.

Summary and conclusions

In 1991 Zerner stated ³⁰

Quantum chemistry has clearly passed through the hands of the theorist and become yet one more tool for the experimentalist to use to interpret and understand his data

And

Quantum chemistry and quantum chemical concepts have had an enormous impact on chemistry, and this impact is accelerating. It is rare now to pick up a chemical journal and read an article in which quantum chemistry or concepts derived from quantum chemistry are not present

Both of these statements are clearly even more relevant today than they were back then. Together with the enormous success that quantum chemistry enjoys and the transformative progress that has been made in the development of efficient algorithms and user-friendly software, there also comes a responsibility on the side of the user to use the available tools in a diligent manner. Here “diligent” refers to a respectful use of quantum chemical software in a self-critical and scientific manner and with an educated awareness of the limitations that still exist.

In this short perspective, I have tried to provide a short overview of the evolution of the field of quantum chemistry from Dirac’s bold prophecy in 1929 to the rapidly evolving landscape we are witnessing today. Particular attention has been given to the question of whether Dirac’s prophecy provides a comprehensive view of the challenges that quantum chemistry is facing, in particular when it comes to applying quantum chemical methods to large molecular systems. Finally, the inevitable question of understanding vs. numbers has been addressed and some speculations about the emerging approaches of artificial intelligence and machine learning have been added. It seems clear that the field of quantum chemistry is vibrant, exciting and rapidly evolving and there is every reason to expect that it will play an ever more important role in chemistry – in it’s own right, alongside experiments or even replacing experiment in selected areas.