AdaptiveActivationFunctionInterface#

Module for adaptive functions.

class AdaptiveActivationFunctionInterface(alpha=None, beta=None, gamma=None, fixed=None)[source]#

Bases: Module

The AdaptiveActivationFunctionInterface class makes a torch.nn.Module activation function into an adaptive trainable activation function. If one wants to create an adpative activation function, this class must be use as base class.

Given a function \(f:\mathbb{R}^n\rightarrow\mathbb{R}^m\), the adaptive function \(f_{\text{adaptive}}:\mathbb{R}^n\rightarrow\mathbb{R}^m\) is defined as:

\[f_{\text{adaptive}}(\mathbf{x}) = \alpha\,f(\beta\mathbf{x}+\gamma),\]

where \(\alpha,\,\beta,\,\gamma\) are trainable parameters.

See also

Original reference: Godfrey, Luke B., and Michael S. Gashler. A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks. 2015 7th international joint conference on knowledge discovery, knowledge engineering and knowledge management (IC3K). Vol. 1. IEEE, 2015. DOI: arXiv preprint arXiv:1602.01321..

Jagtap, Ameya D., Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics 404 (2020): 109136. DOI: JCP 10.1016.

Initializes the Adaptive Function.

Parameters:

  • alpha (float | complex) – Scaling parameter alpha. Defaults to None. When None is passed, the variable is initialized to 1.

  • beta (float | complex) – Scaling parameter beta. Defaults to None. When None is passed, the variable is initialized to 1.

  • gamma (float | complex) – Shifting parameter gamma. Defaults to None. When None is passed, the variable is initialized to 1.

  • fixed (list) – List of parameters to fix during training, i.e. not optimized (requires_grad set to False). Options are alpha, beta, gamma. Defaults to None.

forward(x)[source]#

Define the computation performed at every call. The function to the input elementwise.

Parameters:

x (torch.Tensor | LabelTensor) – The input tensor to evaluate the activation function.

property alpha#

The alpha variable.

property beta#

The beta variable.

property gamma#

The gamma variable.

property func#

The callable activation function.