The simulator likely overcounts standard attention though. A fused XLA kernel could, in principle, recognize the causal mask and skip the upper triangle entirely — never compute exp(-inf), never multiply by zero weights. The simulator charges full price for the masked entries; a smart compiler probably wouldn’t. (Without profiling the actual XLA-generated code, this is speculation — but the benchmark gap is consistent with it.)
На Украине захотели заблокировать все соцсети и назвали их абсолютным злом20:42，详情可参考谷歌浏览器
，详情可参考手游
Россия вышла из соглашения с ООН14:29，详情可参考超级工厂
// Inference: no gradients needed