Problem
Research questions and friction points this paper is trying to address.
Transformer
sequence of bits
ordered
Transformer Heads Looking for Order
🤖 AI Summary
研究解决了使用Transformer判断比特序列是否有序的问题，证明单头单层Transformer无法完成，而双头单层可以实现，模型包含输出MLP。
📝 Abstract
In this note, we show that the problem of checking, whether a sequence of bits is ordered, is not doable by 1-head 1-layer transformers but is doable by a 2-head 1-layer transformer. Unlike similar previous results, our results assume the model where transformers have an output MLP.
Innovation
Methods, ideas, or system contributions that make the work stand out.
2-head 1-layer transformer
ordered sequence
output MLP
🔎 Similar Papers
No similar papers found.
