bidirectional

Are Transformers Bidirectional? Are transformers bidirectional? The encoder does not have self-attention masking. Therefore is designed not to have any dependency limitation: the token representation obtained at one position depends on all the tokens in the input. This is what makes the Transformer encoder bidirectional. Why is BERT bidirectional? BERT is designed to pretrain deep