Setting the foundations right
Within the previous two articles we saw how we are able to implement a basic classifier based on Rosenblatt’s perceptron and the way this classifier might be improved through the use of the adaptive linear neuron algorithm (adaline). These two articles cover the foundations before attempting to implement a man-made neural network with many layers. Moving from adaline to deep learning is a much bigger leap and plenty of machine learning practitioners will opt directly for an open source library like PyTorch. Using such a specialised machine learning library is after all really helpful for developing a model in production, but not necessarily for learning the basic concepts of multilayer neural networks. This text builds a multilayer neural network from scratch. As a substitute of solving a binary classification problem we’ll deal with a multiclass one. We will probably be using the sigmoid activation function after each layer, including the output one. Essentially we train a model that for every input, comprising a vector of features, produces a vector with length equal to the variety of classes to be predicted. Each element of the output vector is within the range [0, 1] and might be understood because the “probability” of every class.
The aim of the article is to turn out to be comfortable with the mathematical notation used for describing mathematically neural networks, understand the role of the assorted matrices with weights and biases, and derive the formulas for updating the weights and biases to minimise the loss function. The implementation allows for any variety of hidden layers with arbitrary dimensions. Most tutorials assume a hard and fast architecture but this text uses a fastidiously chosen mathematical notation that supports generalisation. In this manner we can even run easy numerical experiments to look at the predictive performance as a function of the number and size of the hidden layers.
As in the sooner articles, I used the net LaTeX equation editor to develop the LaTeX code for the equation after which the chrome plugin Maths Equations Anywhere to render the equation into a picture. All LaTex code is provided at the top of the article if it’s good to render it again. Getting the notation right is a component of the journey in machine learning, and essential for understanding neural networks. It’s critical to scrutinise the formulas, and listen to the assorted indices and the foundations for matrix multiplication. Implementation in code becomes trivial once the model is accurately formulated on paper.
All code utilized in the article might be present in the accompanying repository. The article covers the next topics
∘ What’s a multilayer neural network?
∘ Activation
∘ Loss function
∘ Backpropagation
∘ Implementation
∘ Dataset
∘ Training the model
∘ Hyperparameter tuning
∘ Conclusions
∘ LaTeX code of equations utilized in the article
What’s a multilayer neural network?
This section introduces the architecture of a generalised, feedforward, fully-connected multilayer neural network. There are a variety of terms to undergo here as we work our way through Figure 1 below.
For each prediction, the network accepts a vector of features as input
that will also be understood as a matrix with shape (1, n⁰). The network uses L layers and produces a vector as an output
that might be understood as a matrix with shape (1, nᴸ) where nᴸ is the variety of classes within the multiclass classification problem we want to unravel. Every float on this matrix lies within the range [0, 1] and the index of the most important element corresponds to the anticipated class. The (L) notation within the superscript is used to consult with a selected layer, on this case the last one.
But how will we generate this prediction? Let’s deal with the primary element of the primary layer (the input will not be considered a layer)
We first compute the online input that is actually an inner product of the input vector with a set of weights with the addition of a bias term. The second operation is the applying of the activation function σ(z) to which we’ll return later. For now it can be crucial to be mindful that the activation function is actually a scalar operation.
We are able to compute all elements of the primary layer in the identical way
From the above we are able to deduce that we’ve introduced n¹ x n⁰ weights and n¹ bias terms that may must be fitted when the model is trained. These calculations will also be expressed in matrix form
Pay close attention to the form of the matrices. The online output is a results of a matrix multiplication of two matrices with shape (1, n⁰) and (n⁰, n¹) that ends in a matrix with shape (1, n¹), to which we add one other matrix with the bias terms that has the identical (1, n¹) shape. Note that we introduced the transpose of the burden matrix. The activation function applies to each element of this matrix and hence the activated values of layer 1 are also a matrix with shape (1, n¹).
The above might be readily generalised for each layer within the neural network. Layer k accepts as input nᵏ⁻¹ values and produces nᵏ activated values
Layer k introduces nᵏ x nᵏ⁻¹ weights and nᵏ bias terms that may must be fitted when the model is trained. The entire variety of weights and bias terms is
so if we assume an input vector with 784 elements (dimension of a low resolution image in gray scale), a single hidden layer with 50 nodes and 10 classes within the output we want to optimise 785*50+51*10 = 39,760 parameters. The variety of parameters grows further if we increase the variety of hidden layers and the variety of nodes in these layers. Optimising an objective function with so many parameters will not be a trivial undertaking and that is why it took a while from the time adaline was introduced until we discovered the right way to train deep networks within the mid 80s.
This section essentially covers what’s often called the forward pass, i.e. how we apply a series of matrix multiplications, matrix additions and element sensible activations to convert the input vector to an output vector. Should you pay close attention we assumed that the input was a single sample represented as a matrix with shape (1, n⁰). The notation holds even when we we feed into the network a batch of samples represented as a matrix with shape (N, n⁰). There is just small complexity in the case of the bias terms. If we deal with the primary layer we sum a matrix with shape (N, n¹) to a bias matrix with shape (1, n¹). For this to work the bias matrix has its first row replicated as repeatedly because the variety of samples within the batch we use within the forward pass. That is such a natural operation that NumPy does it mechanically in what is named broadcasting. Once we apply forward pass to a batch of inputs it is probably cleaner to make use of capital letters for all vectors that turn out to be matrices, i.e.
Note that I assumed that broadcasting was applied to the bias terms resulting in a matrix with as many rows because the variety of samples within the batch.
Operating with batches is typical with deep neural networks. We are able to see that because the variety of samples N increases we’ll need more memory to store the assorted matrices and perform the matrix multiplications. As well as, using only part of coaching set for updating the weights means we will probably be updating the parameters several times in each pass of the training set (epoch) resulting in faster convergence. There may be a further profit that is probably less obvious. The network uses activation functions that, unlike the activation in adaline, usually are not the identity. In reality they usually are not even linear, which makes the loss function non convex. Using batches introduces noise that’s believed to assist escaping shallow local minima. A suitably chosen learning rate further assists with this.
As a final note before we move on, the term feedforward comes from the proven fact that each layer is using as input the output of the previous layer without using loops that result in the so-called recurrent neural networks.
Activation
Enabling the neural network to unravel complex problem requires introducing some type of nonlinearity. That is achieved through the use of an activation function in each layer. There are a lot of decisions. For this text we will probably be using the sigmoid (logistic) activation function that we are able to visualise with
that produces
The code also includes all imports we’ll need throughout the article.
The activation function maps any float to the range 0 to 1. In point of fact the sigmoid is a more suitable activation of the ultimate layer for binary classification problems. For multiclass problems it could have been more appropriate to make use of softmax to normalize the output of the neural network to a probability distribution over predicted output classes. One strategy to take into consideration that is that softmax enforces that post activation the sum of the entries of the output vector must add as much as 1, that will not be the case with sigmoid. One other strategy to give it some thought is that the sigmoid essentially converts the logits (log odds) to a one-versus-all (OvA) probability. Still, we’ll use the sigmoid activation function to remain as close as possible to adaline since the softmax will not be a component sensible operation and this may introduce some complexities within the back propagation algorithm. I leave this as an exercise for the reader.
Loss function
The loss function used for adaline was the mean square error. In practice a multiclass classification problem would use a multiclass cross-entropy loss. To be able to remain as near adaline as possible, and to facilitate the analytical calculation of the gradients of the loss function with respect to the parameters, we’ll stick on the mean square error loss function. Every sample within the training set, belongs to one in all the nᴸ classes and hence the loss function might be expressed as
where the primary summation is over all samples and the second over classes. The above implies that the known class for sample i has been converted to a one-hot-encoding, i.e. a matrix with shape (1, nᴸ) containing zeros other than the element that corresponds to the sample class that’s one. We adopted yet another notation convention in order that [j] within the superscript is used to consult with sample j. The summation above doesn’t need to make use of all samples within the training set. In practice it would be applied in batches of N’ samples with N’< The loss function is a scalar that is determined by tens or lots of of 1000’s of parameters, comprising weights and bias terms. Typically, these parameters are initialised with random numbers and are updated iteratively in order that the loss function is minimised using the gradient of the loss function with regard to every parameter. Within the case of adaline, the analytical derivation of the gradients was straightforward. For multilayer neural networks the derivation is more involved but stays tractable if we adopt a clever strategy. We enter the world of the back propagation but fear not. Backpropagation essentially boils all the way down to a successive application of the chain differentiation rule from the appropriate to the left. Let’s come back to the loss function. It is determined by the activated values of the last layer, so we are able to first compute the derivatives with regard to those The above might be understood because the (j, i) element of a derivate matrix with shape (N, nᴸ) and might be written in matrix form as where each matrices in the appropriate hand side have shape (N, nᴸ). The activated values of the last layer are computed by applying the sigmoid activation function on each element of the online input matrix of the last layer. Hence, to compute the derivatives of the loss function with regard to every element of this net input matrix of the last layer we simply have to remind ourselves on the right way to compute the derivative of a nested function with the outer one being the sigmoid function: The star multiplication denotes element sensible multiplication. The results of this formula is a matrix with shape (N, nᴸ). If you could have difficulties computing the derivative of the sigmoid function please check here. We are actually able to compute the derivative of the loss function with regard to the weights of the L-1 layer; that is the primary set of weights we encounter once we move from right to left This results in a matrix with the identical shape because the weights of the L-1 layer. We next have to compute the derivative of the online input of the L layer with regard to the weights of the L-1 layer. If we pick one element of the online input matrix of the last layer and one in all these weights we’ve If you could have trouble to know the above, think that for each sample j, the i element of the online input of the L layer only is determined by the weights of the L-1 layer for which the primary index can be i. Hence, we are able to eliminate one in all the summations within the derivative We are able to express all these derivatives in a matrix notation using Essentially the implicit summation within the matrix multiplication absorbs the summation over the samples. Follow together with the shapes of the multiplied matrices and you will notice that the resulting derivative matrix has the identical shape as the burden matrix used to calculate the online input of the L layer. Although the variety of elements within the resulting matrix is restricted to the product of the variety of nodes of the last two layers (the form is (nᴸ, nᴸ⁻¹)), the multiplied matrices are much larger and hence are typically more memory consuming. Hence, the necessity to use batches when training the model. The derivatives of the loss function with respect to the bias terms used for calculating the online input of the last layer might be computed similarly as for the weights to present that results in a matrix with shape (1, nᴸ). We’ve got just computed all derivatives of the loss function with regard to the weights and bias terms used for computing the online input of the last layer. We now turn our attention to the gradients with the regard to the weights and bias terms of the previous layer (these parameters can have the superscript index L-2). Hopefully we are able to start identifying patterns in order that we are able to apply them to compute the derivates with regard to the weights and bias terms for k=0,..,L-2. We could see these patterns emerge if we compute the derivative of the loss function with regard to the activated values of the L-1 layer. These should form a matrix with shape (N, nᴸ⁻¹) that’s computed as Once we’ve the derivatives of the loss with regard to the activated values of layer L-1 we are able to proceed with calculating the derivatives of the loss function with regard to the online input of the layer L-1 after which with regard to the weights and bias terms with index L-2. Let’s recap how we backpropagate by one layer. We assume we’ve computed the derivative of the loss function with regard to the weights and bias terms with index k and we want to compute the derivates of the loss function with regard to the weights and bias terms with index k-1. We want to perform 4 operations: All operations are vectorised. We are able to already start imaging how we could implement these operations in a category. My understanding is that when one uses a specialised library so as to add a completely connected linear layer with an activation function, that is what happens behind the scenes! It is good not to fret in regards to the mathematical notation, but my suggestion could be to undergo these derivations at the very least once. On this section we offer the implementation of a generalised, feedforward, multilayer neural network. The API draws some analogies to the one present in specialised deep learning libraries comparable to PyTorch The code comprises two utility functions: The category The The Reflecting back on my learnings from this text, I felt that the implementation was straightforward. The toughest part was to provide you with a sturdy mathematical notation and work out the gradients on paper. Still, it is straightforward to make mistakes that is probably not easy to detect even when the optimisation seems to converge. This brings me to the special that produces Gradients look so as! We are going to need a dataset for constructing our first model. A famous one often utilized in pattern recognition experiments is the MNIST handwritten digits. We are able to find more details about this dataset within the OpenML dataset repository. All datasets in OpenML are subject to the CC BY 4.0 license that allows copying, redistributing and remodeling the fabric in any medium and for any purpose. The dataset comprises 70,000 digit images and the corresponding labels. Conveniently, the digits have been size-normalized and centered in a fixed-size 28×28 image by computing the middle of mass of the pixels, and translating the image in order to position this point at the middle of the 28×28 field. The dataset might be conveniently retrieved using scikit-learn that prints We are able to see that every image is accessible as a vector with 784 integers between 0 and 255 that were converted to floats within the range [-0.5, 0.5]. This is probably a bit different than the standard feature scaling in scikit-learn where scaling happens per feature somewhat per sample. The category labels were retrieved as strings and converted to integers. The dataset within reason balanced. We next visualise ten images for every digit to acquire a sense on the variations in hand writing that produces We are able to foresee that some digits could also be confused by the model, e.g. the last 9 resembles 8. There might also be hand writing variations that usually are not predicted well, comparable to 7 digits written with a horizontal line in the center, depending on how often such variations are represented within the training set. We now have a neural network implementation and a dataset to make use of it with. In the following section we’ll provide the obligatory code for training the model before we glance into hyperparameter tuning. The primary motion we want to take is to separate the dataset right into a training set, and an external (hold-out) test set. We are able to readily accomplish that using scikit-learn We use stratification in order that the proportion of every class is roughly equal in each the training set and the external (hold-out) dataset. The external (hold-out) test set comprises 10,000 samples and is not going to be used for anything aside from assessing the model performance. On this section we’ll use the 60,000 samples for training set with none hyperparameter tuning. When deriving the gradients of the loss function with regard to the model parameters we show that it’s obligatory to perform several matrix multiplications and a few of these matrices have as many rows because the variety of samples. On condition that typically the variety of samples is kind of large we’ll need a big amount of memory. To alleviate this we will probably be using mini batches in the identical way we used mini batches in the course of the gradient descent optimisation of the adaline model. Typically, each batch can contain 100–500 samples. Reducing the batch size increases the convergence speed because we make more parameter updates throughout the the identical pass of the training set (epoch), but we also increase the noise. We want to strike a balance. First we offer a generator that accepts the training set and the batch size and returns the batches The generator returns batches of equal size that by default contain 100 samples. The entire variety of samples is probably not a multiple of the batch size and hence some samples is not going to be returned in a given go through the training set. Th variety of skipped samples is smaller than the batch size and the set of samples overlooked changes each time the generator is used, assuming we don’t reset the random number generator. Hence, this will not be critical. As we will probably be passing though the training sets multiple times in different epochs we’ll eventually use the training set fully. The rationale for using batches of a relentless size is that we will probably be updating the model parameters after each batch and a small batch can increase the noise and forestall convergence, especially if the samples within the batch occur to be outliers. When the model is initiated we expect a low accuracy that we are able to confirm with that provides an accuracy of roughly 9.5%. That is roughly expected for a fairly balanced dataset as there are 10 classes. We now have the means to observe the loss and the accuracy of every batch passed to the forward pass that we are going to exploit during training. Let’s write the ultimate piece of code to iterate over the epochs and mini batches, update the model parameters and monitor how the loss and accuracy evolves in each the training set and external (hold-out) test set. Using this function training becomes a single line of code that produces We are able to see that after the ten epochs the accuracy for the training set has reached roughly 76%, whilst the accuracy of the external (hold-out) test set is barely higher, indicating that the model has not been overfitted. The lack of the training set keeps decreasing and hence convergence has not been reached yet. The model allows hot starting so we could run one other ten epochs by repeating the only line of code above. As a substitute, we’ll initiate the model again and run it for 100 epochs, increasing the batch size to 200 at the identical time. We offer the entire code for doing so. We first plot the training loss and its rate of change as a function of the epoch number that produces We are able to see the model has converged reasonably well as the speed of the change of the training loss has turn out to be greater than two orders of magnitude smaller in comparison with its value firstly of the training. I’m unsure why we observe a discount in convergence speed at around epoch 10; I can only speculate that the optimiser escaped an area minimum. We can even plot the accuracy of the training set and the test set as a function of the epoch number that produces The accuracy reaches roughly 90% after about 50 epochs for each the training set and external (hold-out) test set, suggesting that there isn’t a/little overfitting. We just trained our first, custom built multilayer neural network with one hidden layer! On this previous section we selected an arbitrary network architecture and fitted the model parameters. On this section we proceed with a basic hyperparameter tuning by various the variety of hidden layers (starting from 1 to three), the variety of nodes within the hidden layers (starting from 10 to 50 in increments of 10) and the training rate (using the values 0.1, 0.2 and 0.3). We kept the batch size constant at 200 samples per batch. Overall, we tried 45 parameter combos. We are going to employ 6-fold cross validation (not nested) which suggests 6 model trainings per parameter combination, which translates to 270 model trainings in total. In each fold we will probably be using 50,000 samples for training and 10,000 samples for measuring the accuracy (called validation within the code). To reinforce the probabilities to realize convergence we’ll perform 250 epochs for every model fitting. The entire execution time was ~12 hours on a single processor (Intel Xeon Gold 3.5GHz). That is roughly what we are able to reasonably run on a CPU. The training speed might be increased using multiprocessing. In reality, the training could be way faster using a specialised deep learning library like PyTorch on GPUs, comparable to the freely available T4 GPUs on Google Colab. This code iterates over all hyperparameter values and folds and stores the loss and accuracy for each the training (50,000 samples) and validation (10,000 samples) in a pandas dataframe. The dataframe is used to search out the optimal hyperparameters that produces We are able to see that there may be little profit in increasing the variety of layers. Perhaps we could have gained barely higher performance using a bigger first hidden layer because the hyperparameter tuning hit the sure of fifty nodes. Some mean cross-validation accuracies are quite low that might be indicative of poor convergence (e.g. when using 3 hidden layers with 10 nodes each). We didn’t investigate further but this could be typically required before concluding on the optimal network geometry. I’d expect that allowing for more epochs would increase accuracy further particular with the larger networks. A final step is to retrain the model with all samples aside from the external (hold-out) set which can be only used for the ultimate evaluation The last 5 epochs are We achieved ~95% accuracy with the external (hold-out) test set. That is magical if we consider that we began with a blank piece of paper! This text demonstrated how we are able to construct a multilayer, feedforward, fully connected neural network from scratch. The network was used for solving a multiclass classification problem. The implementation has been generalised to permit for any variety of hidden layers with any variety of nodes. This facilitates hyperparameter tuning by various the variety of layers and units in them. Nonetheless, we want to be mindful that the loss gradients turn out to be smaller and smaller because the depth of the neural network increases. That is often called the vanishing gradient problem and requires using specialised training algorithms once the depth exceeds a certain threshold, which is out of the scope of this text. Our vanilla implementation of a multilayer neural network has hopefully educational value. Using it in practice would require several improvements though. To begin with, overfitting would must be addressed, by employing some type of drop out. Other improvements, comparable to the addition of skip-connections and the variation of the training rate during training, could also be helpful too. As well as, the network architecture itself might be optimised, e.g. through the use of a convolutional neural network that will be more appropriate for classifying images. Such improvements are best attempted using a specialised library like PyTorch. When developing algorithms from scratch one must be wary of the time it takes and where to attract the road in order that the endeavour stays educational without being extremely time consuming. I hope this text strikes a great balance on this sense. Should you are intrigued I’d recommend this book for further study. The equations utilized in the article might be present in the gist below, in case you desire to to render them again.Backpropagation
Implementation
sigmoid() applies the sigmoid (logistic) activation function to a float (or NumPy array), and int_to_onehot() takes a listing of integers with the category of every sample and returns their one-hot-encoding representation.MultilayerNeuralNetClassifier comprises the neural net implementation. The initialisation constructor assigns random numbers to the weights and bias terms of every layer. For example if we construct a neural network with layers=[784, 50, 10], we will probably be using 784 input features, a hidden layer with 50 nodes and 10 classes as output. This generalised implementation allows changing each the variety of hidden layers and the variety of nodes within the hidden layers. We are going to exploit this once we do hyperparameter tuning in a while. For reproducibility we use a seed for the random number generator to initialise the weights.forward method returns the activated values for every layer as a listing of matrices. The tactic works with a single sample or an array of samples. The last of the returned matrices comprises the model predictions for the category membership of every sample. Once the model is trained only this matrix is used for making predictions. Nonetheless, whilst the model is being trained we want the activated values for all layers as we’ll see below and that is why the forward method returns all of them. Assuming that the network was initialised with layers=[784, 50, 10], the forward method will return a listing of two matrices, the primary one with shape (N, 50) and the second with shape (N, 10), assuming the input x has N samples, i.e. it’s a matrix with shape (N, 784).backward method implements backpropagation, i.e. all of the analytically computed derivatives of the loss function as described within the previous section. The last layer is special because we want to compute the derivatives of the loss function with regard to the model output using the known classes. The primary layer is special because we want to make use of the input as an alternative of the activated values of the previous layer. The center layers are all the identical. We simply iterate over the layers backwards. The code reflects fully the analytically derived formulas. By utilizing NumPy we vectorise all operations that hurries up execution. The tactic returns a tuple of two lists. The primary list comprises the matrices with the derivatives of the loss function with regard to the weights of every layer. Assuming that the network was initialised with layers=[784, 50, 10], the list will contain two matrices with shapes (784, 50) and (50, 10). The second list comprises the vectors with the derivatives of the loss function with regard to the bias terms of every layer. Assuming that the network was initialised with layers=[784, 50, 10], the list will contain two vectors with shapes (50, ) and (10,).backward_numerical method. This method is used for neither training the model nor making predictions. It uses finite (central) differences to estimate the derivatives of the loss function with regard to the weights and bias terms of the chosen layer. The numerical derivatives might be compared with the analytically computed ones returned by the backward function to make sure that the implementation is correct. This method could be too slow for use for training the model because it requires two forward passes for every derivative and in our trivial example with layers=[784, 50, 10] there are 39,760 such derivatives! Nevertheless it is a lifesaver. Personally I’d not have managed to debug the code without it. If you should keep a key message from this text, it could be the usefulness of numerical differentiation for double checking your analytically derived gradients. We are able to check the correctness of the gradients with an untrained modellayer 3: 300 out of 300 weight gradients are numerically equal
layer 3:10 out of 10 bias term gradients are numerically equal
layer 2: 1200 out of 1200 weight gradients are numerically equal
layer 2:30 out of 30 bias term gradients are numerically equal
layer 1: 2000 out of 2000 weight gradients are numerically equal
layer 1:40 out of 40 bias term gradients are numerically equalDataset
original X: X.shape=(70000, 784), X.dtype=dtype('int64'), X.min()=0, X.max()=255
original y: y.shape=(70000,), y.dtype=dtype('O')
processed X: X.shape=(70000, 784), X.dtype=dtype('float64'), X.min()=-1.0, X.max()=1.0
processed y: y.shape=(70000,), y.dtype=dtype('int32')
class counts: 0:6903, 1:7877, 2:6990, 3:7141, 4:6824, 5:6313, 6:6876, 7:7293, 8:6825, 9:6958Training the model
epoch 0: loss_training=0.096 | accuracy_training=0.236 | loss_test=0.088 | accuracy_test=0.285
epoch 1: loss_training=0.086 | accuracy_training=0.333 | loss_test=0.085 | accuracy_test=0.367
epoch 2: loss_training=0.083 | accuracy_training=0.430 | loss_test=0.081 | accuracy_test=0.479
epoch 3: loss_training=0.078 | accuracy_training=0.532 | loss_test=0.075 | accuracy_test=0.568
epoch 4: loss_training=0.072 | accuracy_training=0.609 | loss_test=0.069 | accuracy_test=0.629
epoch 5: loss_training=0.066 | accuracy_training=0.657 | loss_test=0.063 | accuracy_test=0.673
epoch 6: loss_training=0.060 | accuracy_training=0.691 | loss_test=0.057 | accuracy_test=0.701
epoch 7: loss_training=0.055 | accuracy_training=0.717 | loss_test=0.052 | accuracy_test=0.725
epoch 8: loss_training=0.050 | accuracy_training=0.739 | loss_test=0.049 | accuracy_test=0.742
epoch 9: loss_training=0.047 | accuracy_training=0.759 | loss_test=0.045 | accuracy_test=0.765Hyperparameter tuning
optimal parameters: n_hidden_layers=1, n_hidden_nodes=50, learning rate=0.3
best mean cross validation accuracy: 0.944
| n_hidden_layers | 10 | 20 | 30 | 40 | 50 |
|------------------:|---------:|---------:|---------:|---------:|--------:|
| 1 | 0.905217 | 0.927083 | 0.936883 | 0.939067 | 0.9441 |
| 2 | 0.8476 | 0.925567 | 0.933817 | 0.93725 | 0.9415 |
| 3 | 0.112533 | 0.305133 | 0.779133 | 0.912867 | 0.92285 |epoch 245: loss_training=0.008 | accuracy_training=0.958 | loss_test=0.009 | accuracy_test=0.946
epoch 246: loss_training=0.008 | accuracy_training=0.958 | loss_test=0.009 | accuracy_test=0.947
epoch 247: loss_training=0.008 | accuracy_training=0.958 | loss_test=0.009 | accuracy_test=0.947
epoch 248: loss_training=0.008 | accuracy_training=0.958 | loss_test=0.009 | accuracy_test=0.946
epoch 249: loss_training=0.008 | accuracy_training=0.958 | loss_test=0.009 | accuracy_test=0.946Conclusions
LaTeX code of equations utilized in the article