U Ph @sRddlmZddlZddlmZddlmZeddd\ZZGdddej Z dS) ) annotationsN)optional_importzeinops.layers.torch Rearrange)namec s<eZdZdZdddddddd d fd d Zd dZZS)SABlockz A self-attention block, based on: "Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale " FNintfloatboolz int | NoneNone) hidden_size num_heads dropout_rateqkv_bias save_attndim_headreturncstd|krdks(ntd||dkr