AdaptiveReLU#
- class AdaptiveReLU(alpha=None, beta=None, gamma=None, fixed=None)[source]#
Bases:
AdaptiveActivationFunctionInterface
Adaptive trainable
ReLU
activation function.Given the function \(\text{ReLU}:\mathbb{R}^n\rightarrow\mathbb{R}^n\), the adaptive function \(\text{ReLU}_{\text{adaptive}}:\mathbb{R}^n\rightarrow\mathbb{R}^n\) is defined as:
\[\text{ReLU}_{\text{adaptive}}({x}) = \alpha\,\text{ReLU}(\beta{x}+\gamma),\]where \(\alpha,\,\beta,\,\gamma\) are trainable parameters, and the ReLU function is defined as:
\[\text{ReLU}(x) = \max(0, x)\]See also
Original reference: Godfrey, Luke B., and Michael S. Gashler. A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks. 2015 7th international joint conference on knowledge discovery, knowledge engineering and knowledge management (IC3K). Vol. 1. IEEE, 2015. DOI: arXiv preprint arXiv:1602.01321..
Jagtap, Ameya D., Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Computational Physics 404 (2020): 109136. DOI: JCP 10.1016.
Initializes the Adaptive Function.
- Parameters:
alpha (float | complex) – Scaling parameter alpha. Defaults to
None
. WhenNone
is passed, the variable is initialized to 1.beta (float | complex) – Scaling parameter beta. Defaults to
None
. WhenNone
is passed, the variable is initialized to 1.gamma (float | complex) – Shifting parameter gamma. Defaults to
None
. WhenNone
is passed, the variable is initialized to 1.fixed (list) – List of parameters to fix during training, i.e. not optimized (
requires_grad
set toFalse
). Options arealpha
,beta
,gamma
. Defaults to None.