U Ph@sddlmZddlZddlmZddlmZeddddrNdd d d d Zndd d dd Zeddddrdd d ddZnd d d ddZGdddejZ Gdddej j Z GdddejZ GdddejZGdddejZdS)!) annotationsN)nn)optional_importztorch.nn.functionalmish)nameFboolinplacecCstjjj||dSNr )torchr functionalrxr rU/home/dell461/cl/sdc2/HISourceFinder-master-l/src/monai/networks/blocks/activation.py monai_mishsrcCs|ttjj|SN)r tanhrr softplusrrrrrssilucCstjjj||dSr )r rr rrrrr monai_swish srcCs t|Sr)SwishImplementationapplyrrrrr%scs2eZdZdZd fdd ZdddddZZS) SwishaiApplies the element-wise function: .. math:: \text{Swish}(x) = x * \text{Sigmoid}(\alpha * x) ~~~~\text{for constant value}~ \alpha. Citation: Searching for Activation Functions, Ramachandran et al., 2017, https://arxiv.org/abs/1710.05941. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['swish']() >>> input = torch.randn(2) >>> output = m(input) ?cst||_dSr)super__init__alpha)selfr __class__rrr@s zSwish.__init__ torch.Tensor)inputreturncCs|t|j|Sr)r sigmoidrrr#rrrforwardDsz Swish.forward)r__name__ __module__ __qualname____doc__rr' __classcell__rrr rr)src@s(eZdZdZeddZeddZdS)rzMemory efficient implementation for training Follows recommendation from: https://github.com/lukemelas/EfficientNet-PyTorch/issues/18#issuecomment-511677853 Results in ~ 30% memory saving during training as compared to Swish() cCs|t|}|||Sr)r r%save_for_backward)ctxr#resultrrrr'Ps zSwishImplementation.forwardcCs,|jd}t|}||d|d|S)Nrr) saved_tensorsr r%)r/ grad_outputr#Z sigmoid_inputrrrbackwardVs  zSwishImplementation.backwardN)r)r*r+r, staticmethodr'r3rrrrrHs  rcs6eZdZdZd ddfdd Zddd d ZZS) MemoryEfficientSwisha%Applies the element-wise function: .. math:: \text{Swish}(x) = x * \text{Sigmoid}(\alpha * x) ~~~~\text{for constant value}~ \alpha=1. Memory efficient implementation for training following recommendation from: https://github.com/lukemelas/EfficientNet-PyTorch/issues/18#issuecomment-511677853 Results in ~ 30% memory saving during training as compared to Swish() Citation: Searching for Activation Functions, Ramachandran et al., 2017, https://arxiv.org/abs/1710.05941. From Pytorch 1.7.0+, the optimized version of `Swish` named `SiLU` is implemented, this class will utilize `torch.nn.functional.silu` to do the calculation if meets the version. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['memswish']() >>> input = torch.randn(2) >>> output = m(input) Frr cst||_dSrrrr rr r rrr|s zMemoryEfficientSwish.__init__r"r#cCs t||jSr)rr r&rrrr'szMemoryEfficientSwish.forward)Fr(rrr rr5]sr5cs6eZdZdZd ddfdd Zddd d ZZS) MishaApplies the element-wise function: .. math:: \text{Mish}(x) = x * tanh(\text{softplus}(x)). Citation: Mish: A Self Regularized Non-Monotonic Activation Function, Diganta Misra, 2019, https://arxiv.org/abs/1908.08681. From Pytorch 1.9.0+, the optimized version of `Mish` is implemented, this class will utilize `torch.nn.functional.mish` to do the calculation if meets the version. Shape: - Input: :math:`(N, *)` where `*` means, any number of additional dimensions - Output: :math:`(N, *)`, same shape as the input Examples:: >>> import torch >>> from monai.networks.layers.factories import Act >>> m = Act['mish']() >>> input = torch.randn(2) >>> output = m(input) Frr cst||_dSrr6r7r rrrs z Mish.__init__r"r8cCs t||jSr)rr r&rrrr'sz Mish.forward)Fr(rrr rr9sr9c@seZdZdZddddZdS)GEGLUaApplies the element-wise function: .. math:: \text{GEGLU}(x) = x_1 * \text{Sigmoid}(x_2) where :math:`x_1` and :math:`x_2` are split from the input tensor along the last dimension. Citation: GLU Variants Improve Transformer, Noam Shazeer, 2020, https://arxiv.org/abs/2002.05202. Shape: - Input: :math:`(N, *, 2 * D)` - Output: :math:`(N, *, D)`, where `*` means, any number of additional dimensions r"r8cCs"|jddd\}}|tj|S)N)dim)chunkrr gelu)rr#rZgaterrrr'sz GEGLU.forwardN)r)r*r+r,r'rrrrr:sr:)F)F)F)F) __future__rr r monai.utilsrrrModulerautogradFunctionrr5r9r:rrrr s   ("