2.2 Solution 💡

The important thing idea here is to re-formulate the PINN loss function.

Specifically, we are able to introduce a dynamic weighting scheme to account for various contributions of PDE residual loss evaluated at different temporal locations. Let’s break it down using illustrations.

For simplicity, let’s assume the collocation points are uniformly sampled within the spatial-temporal domain of our simulation, as illustrated within the figure below:

To proceed with one step of gradient descent, we must first calculate the cumulative PDE residual loss across all collocation points. One specific method to do this is by first calculating the losses related to the collocation points sampled at individual time instances, after which performing a “easy sum” to get the entire loss. The next gradient descent step can then be conducted based on the calculated total loss to optimize the PINN weights.

In fact, the precise order of summation over collocation points doesn’t influence the entire loss computation; all methods yield the identical result. Nonetheless, the choice to group loss calculations by temporal order is purposeful, designed to emphasise the element of ‘temporality’. This idea is crucial for understanding the proposed causal training strategy.

On this process, the PDE residual losses evaluated at different temporal locations are treated equally. meaning that each one temporal residual losses are concurrently minimized.

This approach, nevertheless, risks the PINN violating temporal causality, because it doesn’t implement a chronological regularization for minimizing the temporal residual loss at successive time intervals.

So, how can we coax PINN to stick to the temporal precedence during training?

The key is in selectively weighting individual temporal residual losses. As an illustration, suppose that at the present iteration, we would like the PINN to concentrate on approximating the solutions at time instance t₁. Then, we could simply put a better weight on Lᵣ(t₁), which is the temporal residual loss at t₁. This fashion, Lᵣ(t₁) will turn into a dominant component in the ultimate total loss, and in consequence, the optimization algorithm will prioritize minimizing Lᵣ(t₁), which aligns with our goal of approximating solutions at time instance t₁ first.

In the following iteration, we shift our focus to the solutions at time instance t₂. By increasing the load on Lᵣ(t₂), it now becomes the major consider the entire loss calculation. The optimization algorithm is thus directed towards minimizing Lᵣ(t₂), improving the prediction accuracy of the solutions at t₂.

As might be seen from our previous walk-through, various the weights assigned to temporal residual losses at different time instances enables us to direct the PINN to approximate solutions at our chosen time instances.

So, how does this assist in incorporating a causal structure into PINN training? It seems, we are able to design a causal training algorithm (as proposed within the paper), such that the load for the temporal residual loss at time t, i.e., Lᵣ(t), is critical only when the losses before t (Lᵣ(t-1), Lᵣ(t-2), etc.) are small enough. This effectively implies that the neural network begins minimizing Lᵣ(t) only when it has achieved satisfactory approximation accuracy for prior steps.

To find out the load, the paper proposed a straightforward formula: the load ωᵢ is about to be inversely exponentially proportional to the magnitude of the cumulative temporal residual loss from all of the previous time instances. This ensures that the load ωᵢ will only be energetic (i.e., with a sufficiently large value) when the cumulative loss from all previous time instances is small, i.e., PINN can already accurately approximate solutions at previous time steps. That is how temporal causality is reflected within the PINN training.

With all components explained, we are able to piece together the complete causal training algorithm as follows:

Before we conclude this section, there are two remarks value mentioning:

1. The paper suggested using the magnitude of ωᵢ because the stopping criterion for PINN training. Specifically, when all ωᵢ’s are larger than a pre-defined threshold δ, the training could also be deemed accomplished. The advisable value for δ is 0.99.
2. Choosing a correct value for ε is essential. Although this value might be tuned via conventional hyperparameter tuning, the paper advisable an annealing strategy for adjusting ε. Details might be present in the unique paper (section 3).

2.3 Why the answer might work 🛠️

By dynamically weighting temporal residual losses evaluated at different time instances, the proposed algorithm is capable of steer the PINN training to first approximate PDE solutions at earlier times before even attempting to resolve the answer at later times.

This property facilitates the specific incorporation of temporal causality into the PINN training and constitutes the important thing consider potentially more accurate simulations of physical systems.

2.4 Benchmark ⏱️

The paper considered a complete of three different benchmark equations. All problems are forward problems where PINN is used to unravel the PDEs.

• Lorenz system: these equations arise in studies of convection and instability in planetary atmospheric convection. Lorenz system exhibits strong sensitivity to its initial conditions, and it is thought to be difficult for vanilla PINN.

• Kuramoto–Sivashinsky equation: this equation describes the dynamics of assorted wave-like patterns, equivalent to flames, chemical reactions, and surface waves. It is thought to exhibit a wealth of spatiotemporal chaotic behaviors.

• Navier-Stokes equation: this set of partial differential equations describes the motion of fluid substances and constitutes the basic equations in fluid mechanics. The present paper considered a classical two-dimensional decaying turbulence example in a square domain with periodic boundary conditions.

The benchmark studies yielded that:

• The proposed causal training algorithm was capable of achieve 10–100x improvements in accuracy in comparison with the vanilla PINN training scheme.
• Demonstrated that PINNs equipped with causal training algorithm can successfully simulate highly nonlinear, multi-scale, and chaotic systems.

2.5 Strengths and Weaknesses ⚡

Strengths 💪

• Respects the causality principle and makes PINN training more transparent.
• Introduces significant accuracy improvements, allowing it to tackle problems which have remained elusive to PINNs.
• Provides a practical quantitative criterion for assessing the training convergence of PINNs.
• Negligible added computational cost in comparison with the vanilla PINN training strategy. The one added cost is to compute the ωᵢ’s, which is negligible in comparison with auto-diff operations.

Weaknesses 📉

• Introduced a brand new hyperparameter ε, which controls the scheduling of the weights for temporal residual losses. Although the authors proposed an annealing strategy as an alternative choice to avoid the tedious hyper-parameter tuning.
• Complicated the PINN training workflow. Special attention needs to be given to the temporal weights ωᵢ’s, as they are actually functions of the network trainable parameters (e.g., layer weights and bias), and the gradient related to the computation of ωᵢ shouldn’t be back-propagated.

2.6 Alternatives 🔀

There are a pair of other methods which are trying to deal with the identical issue as the present “causal training algorithm”:

• Adaptive time sampling strategy (Wight et al.): as a substitute of weighting the collocation points at different time instances, this strategy modifies the sampling density of collocation points. This has the same effect of shifting the main target of the optimizer on minimizing temporal losses at different time instances.
• “Time-marching”/“Curriculum training” strategy (e.g., Krishnapriyan et al.): the temporal causality is respected via learning the answer sequentially inside separate time windows.

Nonetheless, in comparison with those alternative approaches, the “causal training algorithm” put temporal causality front and center, is more adaptable to quite a lot of problems, and enjoys low added computational cost.

There are several possibilities to further improve the proposed strategy:

• Incorporating more sophisticated data sampling strategies, equivalent to adaptive- and residual-based sampling methods, to further improve the training efficiency and accuracy.

To learn more about how one can optimize the residual points distribution, try this blog within the PINN design pattern series.

• Extend to inverse problem settings. The way to ensure casualty when point sources of data (i.e., observational data) can be found would require an extension of the currently proposed training strategy.

On this blog, we checked out how one can bring causality to PINN training with a reformulation of the training objectives. Listed here are the highlights of the design pattern proposed within the paper:

• [Problem]: The way to make PINNs respect the causality principle underpinning the physical systems?
• [Solution]: Re-formulating the PINN training objective, where a dynamic weighting scheme is introduced to progressively shift the training focus from earlier time steps to later time steps.
• [Potential benefits]: 1. Significantly improved PINNs’ accuracy. 2. Expanded the applicability of PINNs to complex problems.

Here is the PINN design card to summarize the takeaways:

I hope you found this blog useful! To learn more about PINN design patterns, be at liberty to envision out previous posts:

Looking forward to sharing more insights with you within the upcoming blogs!

[1] Wang et al., Respecting causality is all you would like for training physics-informed neural networks, arXiv, 2022.