Base Adaptive Function#

Module for the Adaptive Function base class.

class BaseAdaptiveFunction(alpha=None, beta=None, gamma=None, fixed=None)[source]#

Bases: Module, AdaptiveFunctionInterface

Base class for all adaptive functions, implementing common functionality.

This class extends a standard torch.nn.Module activation function into a trainable adaptive form. It implements the common mechanism used to scale and shift both the input and the output of a given activation function.

Given a function \(f:\mathbb{R}^n\rightarrow\mathbb{R}^m\), the adaptive function \(f_{\text{adaptive}}:\mathbb{R}^n\rightarrow\mathbb{R}^m\) is defined as:

\[f_{\text{adaptive}}(\mathbf{x}) = \alpha\,f(\beta\mathbf{x}+\gamma),\]

where \(\alpha\), \(\beta\), and \(\gamma\) are learnable parameters controlling output scaling, input scaling, and input shifting, respectively.

All specific adaptive functions should inherit from this class and implement the abstract methods declared in the interface.

This class is not meant to be instantiated directly.

See also

Original reference: Godfrey, L. B., Gashler, M. S. (2015). A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks. 7th international joint conference on knowledge discovery, knowledge engineering and knowledge management (IC3K), Vol. 1. DOI: arXiv preprint arXiv:1602.01321..

Original reference: Jagtap, A. D., Karniadakis, G. E. (2020). Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics, 404. DOI: JCP 10.1016.

Initialization of the BaseAdaptiveFunction class.

Parameters:

  • alpha (int | float) – The output scaling parameter of the adaptive function. If None, it is initialized to 1. Default is None.

  • beta (int | float) – The input scaling parameter of the adaptive function. If None, it is initialized to 1. Default is None.

  • gamma (int | float) – The input shifting parameter of the adaptive function. If None, it is initialized to 0. Default is None.

  • fixed (str | list[str]) – The names of parameters to keep fixed during training. These parameters will not be optimized and will have requires_grad=False. Available options are "alpha", "beta", and "gamma". If None, all parameters are trainable. Default is None.

Raises:

  • ValueError – If alpha, when provided, is not a number.

  • ValueError – If beta, when provided, is not a number.

  • ValueError – If gamma, when provided, is not a number.

  • ValueError – If fixed, when provided, is neither a string nor a list of strings.

  • ValueError – If fixed contains invalid parameter names.

forward(x)[source]#

Compute the transformation of the adaptive function on the input.

Parameters:

x (torch.Tensor | LabelTensor) – The input tensor to evaluate the adaptive function.

Raises:

  • RuntimeError – If the adaptive function has not been set.

  • RuntimeError – If the adaptive function is not callable.

Returns:

The output of the adaptive function.

Return type:

torch.Tensor | LabelTensor

property alpha#

The output scaling parameter of the adaptive function.

Returns:

The alpha parameter.

Return type:

torch.nn.Parameter | torch.Tensor

property beta#

The input scaling parameter of the adaptive function.

Returns:

The beta parameter.

Return type:

torch.nn.Parameter | torch.Tensor

property gamma#

The input shifting parameter of the adaptive function.

Returns:

The gamma parameter.

Return type:

torch.nn.Parameter | torch.Tensor