Learn the way a neural network with one hidden layer using ReLU activation can represent any continuous nonlinear functions.
Activation functions play an integral role in Neural Networks (NNs) since they introduce non-linearity and permit the network to learn more complex features and functions than simply a linear regression. Some of the commonly used activation functions is Rectified Linear Unit (ReLU), which has been theoretically shown to enable NNs to approximate a wide selection of continuous functions, making them powerful function approximators.
On this post, we study specifically the approximation of Continuous NonLinear (CNL) functions, the foremost purpose of using a NN over a straightforward linear regression model. More precisely, we investigate 2 sub-categories of CNL functions: Continuous PieceWise Linear (CPWL), and Continuous Curve (CC) functions. We are going to show how these two function types will be represented using a NN that consists of 1 hidden layer, given enough neurons with ReLU activation.
For illustrative purposes, we consider only single feature inputs yet the concept applies to multiple feature inputs as well.
ReLU is a piecewise linear function that consists of two linear pieces: one which cuts off negative values where the output is zero, and one that gives a continuous linear mapping for non negative values.
CPWL functions are continuous functions with multiple linear portions. The slope is consistent on each portion, than changes abruptly at transition points by adding recent linear functions.
In a NN with one hidden layer using ReLU activation and a linear output layer, the activations are aggregated to form the CPWL goal function. Each unit of the hidden layer is chargeable for a linear piece. At each unit, a brand new ReLU function that corresponds to the changing of slope is added to supply the brand new slope (cf. Fig.2). Since this activation function is all the time positive, the weights of the output layer corresponding to units that increase the slope will likely be positive, and conversely, the weights corresponding to units that decreases the slope will likely be negative (cf. Fig.3). The brand new function is added on the transition point but doesn’t contribute to the resulting function prior to (and sometimes after) that time resulting from the disabling range of the ReLU activation function.
Example
To make it more concrete, we consider an example of a CPWL function that consists of 4 linear segments defined as below.
To represent this goal function, we are going to use a NN with 1 hidden layer of 4 units and a linear layer that outputs the weighted sum of the previous layer’s activation outputs. Let’s determine the network’s parameters in order that each unit within the hidden layer represents a segment of the goal. For the sake of this instance, the bias of the output layer (b2_0) is ready to 0.
The following sort of continuous nonlinear function that we’ll study is CC function. There shouldn’t be a correct definition for this sub-category, but a casual solution to define CC functions is continuous nonlinear functions that usually are not piecewise linear. Several examples of CC functions are: quadratic function, exponential function, sinus function, etc.
A CC function will be approximated by a series of infinitesimal linear pieces, which is named a piecewise linear approximation of the function. The greater the variety of linear pieces and the smaller the scale of every segment, the higher the approximation is to the goal function. Thus, the identical network architecture as previously with a big enough variety of hidden units can yield good approximation for a curve function.
Nevertheless, in point of fact, the network is trained to suit a given dataset where the input-output mapping function is unknown. An architecture with too many neurons is vulnerable to overfitting, high variance, and requires more time to coach. Due to this fact, an appropriate variety of hidden units must be high enough to properly fit the information and at the identical time, sufficiently small to avoid overfitting. Furthermore, with a limited variety of neurons, a great approximation with low loss has more transition points in restricted domain, quite than equidistant transition points in an uniform sampling way (as shown in Fig.10).
On this post, we now have studied how ReLU activation function allows multiple units to contribute to the resulting function without interfering, thus enables continuous nonlinear function approximation. As well as, we now have discussed in regards to the selection of network architecture and variety of hidden units in an effort to obtain a great approximation result.
I hope that this post is helpful on your Machine Learning learning process!
Further inquiries to take into consideration:
- How does the approximation ability change if the variety of hidden layers with ReLU activation increases?
- How ReLU activations are used for a classification problem?
*Unless otherwise noted, all images are by the creator