Home Artificial Intelligence How ReLU Enables Neural Networks to Approximate Continuous Nonlinear Functions? ReLU activation Continuous piecewise linear function approximation Continuous curve function approximation Wrap up

How ReLU Enables Neural Networks to Approximate Continuous Nonlinear Functions? ReLU activation Continuous piecewise linear function approximation Continuous curve function approximation Wrap up

0
How ReLU Enables Neural Networks to Approximate Continuous Nonlinear Functions?
ReLU activation
Continuous piecewise linear function approximation
Continuous curve function approximation
Wrap up

Learn the way a neural network with one hidden layer using ReLU activation can represent any continuous nonlinear functions.

Towards Data Science

Activation functions play an integral role in Neural Networks (NNs) since they introduce non-linearity and permit the network to learn more complex features and functions than simply a linear regression. Some of the commonly used activation functions is Rectified Linear Unit (ReLU), which has been theoretically shown to enable NNs to approximate a wide selection of continuous functions, making them powerful function approximators.

On this post, we study specifically the approximation of Continuous NonLinear (CNL) functions, the foremost purpose of using a NN over a straightforward linear regression model. More precisely, we investigate 2 sub-categories of CNL functions: Continuous PieceWise Linear (CPWL), and Continuous Curve (CC) functions. We are going to show how these two function types will be represented using a NN that consists of 1 hidden layer, given enough neurons with ReLU activation.

For illustrative purposes, we consider only single feature inputs yet the concept applies to multiple feature inputs as well.

Figure 1: Rectified Linear Unit (ReLU) function.

ReLU is a piecewise linear function that consists of two linear pieces: one which cuts off negative values where the output is zero, and one that gives a continuous linear mapping for non negative values.

CPWL functions are continuous functions with multiple linear portions. The slope is consistent on each portion, than changes abruptly at transition points by adding recent linear functions.

Figure 2: Example of CPWL function approximation using NN. At each transition point, a brand new ReLU function is added to/subtracted from the input to extend/decrease the slope.

In a NN with one hidden layer using ReLU activation and a linear output layer, the activations are aggregated to form the CPWL goal function. Each unit of the hidden layer is chargeable for a linear piece. At each unit, a brand new ReLU function that corresponds to the changing of slope is added to supply the brand new slope (cf. Fig.2). Since this activation function is all the time positive, the weights of the output layer corresponding to units that increase the slope will likely be positive, and conversely, the weights corresponding to units that decreases the slope will likely be negative (cf. Fig.3). The brand new function is added on the transition point but doesn’t contribute to the resulting function prior to (and sometimes after) that time resulting from the disabling range of the ReLU activation function.

Figure 3: Approximation of the CPWL goal function in Fig.2 using a NN that consists of 1 hidden layer with ReLU activation and a linear output layer.

Example

To make it more concrete, we consider an example of a CPWL function that consists of 4 linear segments defined as below.

Figure 4: Example of a PWL function.

To represent this goal function, we are going to use a NN with 1 hidden layer of 4 units and a linear layer that outputs the weighted sum of the previous layer’s activation outputs. Let’s determine the network’s parameters in order that each unit within the hidden layer represents a segment of the goal. For the sake of this instance, the bias of the output layer (b2_0) is ready to 0.

Figure 5: The network architecture to model the PWL function defined in Fig.4.
Figure 6: The activation output of unit 0 (a1_0).
Figure 7: The activation output of unit 1 (a1_1), which is aggregated to the output (a2_0) to supply the segment (2). The red arrow represents the change in slope.
Figure 8: The output of unit 2 (a1_2), which is aggregated to the output (a2_0) to supply the segment (3). The red arrow represents the change in slope.
Figure 9: The output of unit 3 (a1_3), which is aggregated to the output (a2_0) to supply the segment (4). The red arrow represents the change in slope.

The following sort of continuous nonlinear function that we’ll study is CC function. There shouldn’t be a correct definition for this sub-category, but a casual solution to define CC functions is continuous nonlinear functions that usually are not piecewise linear. Several examples of CC functions are: quadratic function, exponential function, sinus function, etc.

A CC function will be approximated by a series of infinitesimal linear pieces, which is named a piecewise linear approximation of the function. The greater the variety of linear pieces and the smaller the scale of every segment, the higher the approximation is to the goal function. Thus, the identical network architecture as previously with a big enough variety of hidden units can yield good approximation for a curve function.

Nevertheless, in point of fact, the network is trained to suit a given dataset where the input-output mapping function is unknown. An architecture with too many neurons is vulnerable to overfitting, high variance, and requires more time to coach. Due to this fact, an appropriate variety of hidden units must be high enough to properly fit the information and at the identical time, sufficiently small to avoid overfitting. Furthermore, with a limited variety of neurons, a great approximation with low loss has more transition points in restricted domain, quite than equidistant transition points in an uniform sampling way (as shown in Fig.10).

Figure 10: Two piecewise linear approximations for a continuous curve function (in dashed line). The approximation 1 has more transition points in restricted domain and model the goal function higher than the approximation 2.

On this post, we now have studied how ReLU activation function allows multiple units to contribute to the resulting function without interfering, thus enables continuous nonlinear function approximation. As well as, we now have discussed in regards to the selection of network architecture and variety of hidden units in an effort to obtain a great approximation result.

I hope that this post is helpful on your Machine Learning learning process!

Further inquiries to take into consideration:

  1. How does the approximation ability change if the variety of hidden layers with ReLU activation increases?
  2. How ReLU activations are used for a classification problem?

*Unless otherwise noted, all images are by the creator

LEAVE A REPLY

Please enter your comment!
Please enter your name here