U ›PÓh ã@sFddlmZddlmZddlmZddlmZGdd„dejƒZ dS)é)Ú annotationsN)ÚMLPBlock)ÚSABlockc s<eZdZdZd ddddddddœ‡fd d „ Zd d „Z‡ZS)ÚTransformerBlockz° A transformer block, based on: "Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale " çFÚintÚfloatÚboolÚNone)Ú hidden_sizeÚmlp_dimÚ num_headsÚ dropout_rateÚqkv_biasÚ save_attnÚreturncsxtƒ ¡d|krdks(ntdƒ‚||dkr