o ) i@sddlmZddlZddlmZddlmZeddddr%ddd d Zndddd Zeddddr:ddddZnddddZGdddejZ Gdddej j Z GdddejZ GdddejZGdddejZdS)) annotationsN)nn)optional_importztorch.nn.functionalmish)nameFinplaceboolcCtjjj||dSN)r)torchr functionalrxrrb/home/dell461/cl/sdc2/last_ska_mid/HISourceFinder-master-l/src/monai/networks/blocks/activation.py monai_mishrcCs|ttjj|SN)r tanhrr softplusrrrrrssilucCr r )r rr rrrrr monai_swish rrcCs t|Sr)SwishImplementationapplyrrrrr%s cs,eZdZdZd fdd Zd dd ZZS) SwishaiApplies the element-wise function: .. math:: \text{Swish}(x) = x * \text{Sigmoid}(\alpha * x) ~~~~\text{for constant value}~ \alpha. Citation: Searching for Activation Functions, Ramachandran et al., 2017, https://arxiv.org/abs/1710.05941. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['swish']() >>> input = torch.randn(2) >>> output = m(input) ?ct||_dSr)super__init__alpha)selfr  __class__rrr@s  zSwish.__init__input torch.TensorreturncCs|t|j|Sr)r sigmoidr r!r$rrrforwardDsz Swish.forward)r)r$r%r&r%__name__ __module__ __qualname____doc__rr) __classcell__rrr"rr)src@s(eZdZdZeddZeddZdS)rzMemory efficient implementation for training Follows recommendation from: https://github.com/lukemelas/EfficientNet-PyTorch/issues/18#issuecomment-511677853 Results in ~ 30% memory saving during training as compared to Swish() cCs|t|}|||Sr)r r'save_for_backward)ctxr$resultrrrr)Ps zSwishImplementation.forwardcCs,|jd}t|}||d|d|S)Nrr) saved_tensorsr r')r1 grad_outputr$Z sigmoid_inputrrrbackwardVs  zSwishImplementation.backwardN)r+r,r-r. staticmethodr)r5rrrrrHs  rc.eZdZdZd d fdd Zd d d ZZS)MemoryEfficientSwisha%Applies the element-wise function: .. math:: \text{Swish}(x) = x * \text{Sigmoid}(\alpha * x) ~~~~\text{for constant value}~ \alpha=1. Memory efficient implementation for training following recommendation from: https://github.com/lukemelas/EfficientNet-PyTorch/issues/18#issuecomment-511677853 Results in ~ 30% memory saving during training as compared to Swish() Citation: Searching for Activation Functions, Ramachandran et al., 2017, https://arxiv.org/abs/1710.05941. From Pytorch 1.7.0+, the optimized version of `Swish` named `SiLU` is implemented, this class will utilize `torch.nn.functional.silu` to do the calculation if meets the version. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['memswish']() >>> input = torch.randn(2) >>> output = m(input) Frr crrrrrr!rr"rrr|  zMemoryEfficientSwish.__init__r$r%cC t||jSr)rrr(rrrr) zMemoryEfficientSwish.forwardFrr r$r%r*rrr"rr8]sr8cr7)MishaApplies the element-wise function: .. math:: \text{Mish}(x) = x * tanh(\text{softplus}(x)). Citation: Mish: A Self Regularized Non-Monotonic Activation Function, Diganta Misra, 2019, https://arxiv.org/abs/1908.08681. From Pytorch 1.9.0+, the optimized version of `Mish` is implemented, this class will utilize `torch.nn.functional.mish` to do the calculation if meets the version. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['mish']() >>> input = torch.randn(2) >>> output = m(input) Frr crrr9r:r"rrrr;z Mish.__init__r$r%cCr<r)rrr(rrrr)r=z Mish.forwardr>r?r@r*rrr"rrAsrAc@seZdZdZdddZdS)GEGLUaApplies the element-wise function: .. math:: \text{GEGLU}(x) = x_1 * \text{Sigmoid}(x_2) where :math:`x_1` and :math:`x_2` are split from the input tensor along the last dimension. Citation: GLU Variants Improve Transformer, Noam Shazeer, 2020, https://arxiv.org/abs/2002.05202. Shape: - Input: :math:`(N, *, 2 * D)` - Output: :math:`(N, *, D)`, where `*` means, any number of additional dimensions r$r%cCs"|jddd\}}|tj|S)N)dim)chunkrr gelu)r!r$rZgaterrrr)sz GEGLU.forwardNr@)r+r,r-r.r)rrrrrBsrBr>r?) __future__rr r monai.utilsrrrModulerautogradFunctionrr8rArBrrrrs     ("