Most commonly used Activation Functions
In this article, we're going to look at the top 8 activation functions used in neural networks. These functions are crucial for helping neural networks learn and make sense of complex data. We'll briefly cover what each function does, why it's important, and where it's typically used. If you want to learn even more about neural networks feel free to check out my blog post series.
Activation functions in a neural networks are mathematical functions that determine the output of a node, or 'neuron', in the network. Essentially, they decide whether a neuron should be activated or not, based on the input it receives.
They work by taking the input signal of a neuron and transforming it into an output signal. This process occurs at each neuron in the network. When data is fed into a neural network, each neuron in the hidden layers receives weighted inputs from multiple neurons of the previous layer. These inputs are summed up and then passed through an activation function.
The role of the activation function is to introduce non-linearity into the output of a neuron. This is important because it allows the network to capture complex patterns and relationships in the data. Without non-linearity, no matter how many layers the network has, it would still behave like a single-layer perceptron, which can only handle linear separations.
Depending on the function, the output can be a simple binary yes/no decision (as in the case of step functions), a bounded range (like in sigmoid or tanh functions), or unbounded (like in ReLU). The choice of the activation function affects how the network learns and generalizes from the input data. It determines the neuron's firing rate, i.e., how active the neuron is in response to the given input.
ReLU (Rectified Linear Unit)
The Rectified Linear Unit (ReLU) function, defined as f(x) = max(0, x), is one of the most widely used activation functions in the realm of neural networks, particularly in the hidden layers. Its simplicity and efficiency make it a popular choice; it passes positive values as-is, while negative values are set to zero. This characteristic speeds up the training process significantly, as it introduces non-linearity with less computational cost and without a significant loss of accuracy. A key advantage of ReLU is its ability to address the vanishing gradient problem common in deep networks, enabling models to learn faster and perform better. ReLU's effectiveness is exemplified in its widespread use in Convolutional Neural Networks (CNNs) for image recognition tasks. Its simple yet powerful nature helps in handling complex operations in deep learning models, making it a default choice for many neural network architectures.
def relu(x):
"""
Implements the ReLU (Rectified Linear Unit) function.
Args:
x (float or array-like): A number or an array of numbers.
Returns:
float or array-like: ReLU activation of x.
"""
return max(0, x)ReLU function implemented in Python
Sigmoid
The Sigmoid function, expressed as f(x) = 1 / (1 + exp(-x)), plays a crucial role in machine learning, especially in the context of binary classification problems. It's often employed in the output layer of neural networks. The function maps any input value to a range between 0 and 1, making it particularly useful for models where the output is interpreted as a probability. Its characteristic S-shaped curve provides a smooth gradient and ensures that the output is a smooth, continuous function. This feature is particularly beneficial in reducing the likelihood of abrupt changes in output values. One of the most common applications of the Sigmoid function is in logistic regression models, where it's used to model the probability of a binary outcome. Its ability to handle two-class problems like binary classification, where the outcomes are either 0 or 1, has made it a staple in the machine learning toolkit. The function’s popularity stems from its mathematical properties that make it a natural choice for scenarios where modeling probabilities is essential.
import numpy as np
def sigmoid(x):
"""
Implements the Sigmoid activation function.
Args:
x (float or array-like): A number or an array of numbers.
Returns:
float or array-like: Sigmoid activation of x.
"""
return 1 / (1 + np.exp(-x))Sigmoid function implemented in Python
Tanh (Hyperbolic Tangent)
The Tanh, or Hyperbolic Tangent function, defined by the formula f(x) = (exp(x) - exp(-x)) / (exp(x) + exp(-x)), is a widely used activation function in neural networks, particularly effective in hidden layers. It is similar to the Sigmoid function but has the advantage of being zero-centered. This means that the Tanh function outputs values ranging from -1 to 1, which makes it particularly efficient in scenarios where the data is centered around zero. The zero-centered nature of Tanh helps in modeling inputs that have strongly negative, neutral, and strongly positive values. It is especially popular in Recurrent Neural Networks (RNNs) due to its effectiveness in handling sequential data and its ability to manage the vanishing gradient problem better than the standard Sigmoid function. Tanh's ability to output negative values can be advantageous in scenarios where the model needs to strongly differentiate between negative, neutral, and positive values.
import numpy as np
def tanh(x):
"""
Implements the Tanh (Hyperbolic Tangent) activation function.
Args:
x (float or array-like): A number or an array of numbers.
Returns:
float or array-like: Tanh activation of x.
"""
return np.tanh(x)Tanh function implemented in Python
Softmax
The Softmax function, formulated as f(xi) = exp(xi) / sum(exp(x)) for all i, is an essential activation function in the field of machine learning, particularly in the output layer for multi-class classification tasks. It stands out by converting a vector of values into a probability distribution. Each output of the Softmax function corresponds to the probability of the input belonging to a particular class, with the sum of all outputs adding up to 1. This characteristic makes it highly suitable for scenarios where the model needs to classify inputs into multiple categories, such as in image classification tasks. One of the most notable applications of Softmax is in neural networks where distinguishing among several classes is crucial. Its widespread use in classification problems stems from its ability to handle multiple classes effectively, providing a clear, probabilistic framework for class prediction. The Softmax function's ability to output a probability distribution for multiple classes makes it a cornerstone in numerous machine learning architectures.
import numpy as np
def softmax(x):
"""
Implements the Softmax activation function.
Args:
x (array-like): An array of numbers.
Returns:
array-like: Softmax activation of x.
"""
e_x = np.exp(x - np.max(x)) # Subtracting np.max(x) for numerical stability
return e_x / e_x.sum(axis=0)Softmax function implemented in Python
Leaky ReLU
The Leaky ReLU (Rectified Linear Unit) function, defined as f(x) = x if x > 0, else alpha * x (where alpha is a small constant), is an enhanced variant of the standard ReLU function. It's predominantly used in the hidden layers of neural networks. The key feature of Leaky ReLU is its approach to addressing the 'dying ReLU' problem, where neurons become inactive and only output zero. Unlike ReLU, which outputs zero for all negative input values, Leaky ReLU allows a small, non-zero gradient (multiplied by the alpha factor) when the unit is not active. This small slope for negative values keeps the neurons alive and helps in maintaining a gradient flow through the network, enhancing the learning process. The alpha parameter is typically set to a small value, such as 0.01. Leaky ReLU has been found particularly useful in variants of Convolutional Neural Networks (CNNs), where maintaining active neurons throughout the learning process is crucial. Its ability to prevent neurons from dying out completely makes it an attractive choice in deep learning models, especially in scenarios where ReLU might limit the model's learning capacity.
import numpy as np
def leaky_relu(x, alpha=0.01):
"""
Implements the Leaky ReLU activation function.
Args:
x (float or array-like): A number or an array of numbers.
alpha (float): The slope coefficient for negative values. Default is 0.01.
Returns:
float or array-like: Leaky ReLU activation of x.
"""
return np.maximum(alpha * x, x)Leaky ReLU function implemented in Python
ELU (Exponential Linear Unit)
The Exponential Linear Unit (ELU) function, defined as f(x) = x if x > 0, else alpha * (exp(x) - 1) where alpha is a constant, represents a significant advancement in activation functions used in neural network architectures. Primarily used in the hidden layers, ELU aims to combine the benefits of ReLU and its variants while addressing some of their limitations. For positive values, ELU behaves like ReLU, but for negative values, it outputs values smaller than zero, which helps in reducing the vanishing gradient problem, common in deep neural networks. This negative saturation of ELU for negative inputs allows the network to learn faster and perform better, especially in deeper architectures. The alpha parameter typically has a small value, which controls the value to which an ELU saturates for negative net inputs. ELU's ability to produce negative outputs for negative inputs results in a mean activation closer to zero, which helps in accelerating the learning process. It's particularly effective in deep learning architectures, where maintaining a balance between computational efficiency and learning capability is crucial.
import numpy as np
def elu(x, alpha):
"""
Implements the Exponential Linear Unit (ELU) activation function.
Args:
x (float or array-like): A number or an array of numbers.
alpha (float): The coefficient for negative values. Default is 1.0.
Returns:
float or array-like: ELU activation of x.
"""
return np.where(x > 0, x, alpha * (np.exp(x) - 1))ELU function implemented in Python
Swish
The Swish function, defined as f(x) = x * sigmoid(beta * x), is a relatively newer activation function that has gained attention in the field of deep learning. It's a self-gated function where the output is computed by multiplying the input by the sigmoid of the input. The beta parameter is either a constant or a trainable parameter, which allows the function to adapt during the training process. Swish is designed to be used across different layers in a neural network and is particularly effective in deeper models. It tends to outperform traditional activation functions like ReLU in deeper networks due to its smooth gradient and non-monotonic form. This smoothness helps in mitigating the vanishing gradient problem, a common issue in deep neural networks. Swish's unique characteristic of being bounded below and unbounded above, coupled with its non-monotonic behavior, allows for a more flexible and dynamic range of activation, leading to improved performance in various deep learning architectures. Its versatility and adaptability make it a popular choice for a wide range of neural network applications.
def swish(x, beta):
"""
Implements the Swish activation function.
Args:
x (float or array-like): A number or an array of numbers.
beta (float): The beta parameter of the Swish function. Default is 1.0.
Returns:
float or array-like: Swish activation of x.
"""
return x * sigmoid(beta * x)Swish function implemented in Python
SELU (Scaled Exponential Linear Unit)
The Scaled Exponential Linear Unit (SELU) function, defined as f(x) = lambda * x if x > 0, else lambda * alpha * (exp(x) - 1) where lambda and alpha are predefined constants, represents a significant advancement in activation functions designed for neural networks. SELU is specifically crafted for use in deep feedforward neural networks and is renowned for its self-normalizing properties. These properties enable the activations to maintain a zero mean and unit variance, which helps in preventing the vanishing and exploding gradient problems common in deep networks. The constants lambda and alpha are typically chosen to satisfy these self-normalizing conditions. SELU is particularly effective in deep learning architectures, where maintaining stable dynamics in the layers is crucial. Its ability to automatically scale the activations contributes to faster and more robust learning, making it an excellent choice for deep feedforward networks. The SELU function's unique self-normalizing feature helps in achieving faster convergence during training, making it a valuable tool in the machine learning practitioner's toolkit.
import numpy as np
def selu(x, lambda_param, alpha):
"""
Implements the Scaled Exponential Linear Unit (SELU) activation function.
Args:
x (float or array-like): A number or an array of numbers.
lambda_param (float): The lambda parameter for scaling. Default is 1.0507.
alpha (float): The alpha parameter for negative values. Default is 1.67326.
Returns:
float or array-like: SELU activation of x.
"""
return np.where(x > 0, lambda_param * x, lambda_param * alpha * (np.exp(x) - 1))SELU function implemented in Python
To wrap up, each of the activation functions we've discussed plays a crucial role in neural networks, impacting how these models learn and perform. From the popular ReLU to the innovative Swish, these functions are key in solving different types of machine learning problems. Remember, the choice of activation function can greatly influence your model's success, so it's essential to understand their strengths and uses.
Citation
If you found this article helpful and would like to cite it, you can use the following BibTeX entry.
@misc{
hacking_and_security,
title={Most commonly used Activation Functions},
url={https://hacking-and-security.cc/most-commonly-used-activation-functions},
author={Zimmermann, Philipp},
year={2024},
month={Jan}
}
