Positional Encoding

cho.sh

researchmini

3,252 notes

260729260729 260727260727 Claude Code Live Renaming30747B 260726260726 260722260722 I prefer TUIBB9A08 MarkRight8B0501 에이전트 런타임 서버 셋업1983DC Bun을 Rust로 재작성하다2BDED3 260721260721 Agent Runtime Server SetupC38C2C 260719260719 AutoBuilder63849A 260718260718 Slack Bots are the new CLIs3E662F 260717260717 WristcomputerD98BDF 260716260716 SalpakanA2940F AGENTS.mdDFE522 260714260714 260713260713 260712260712 260711260711

cho.sh

research mini

3,252 notes

Positional Encoding · cho.sh

Positional Encoding

Warning

This post is more than a year old. Information may be outdated.

Positional encoding augments token embeddings with information about their order so a model that uses only attention can distinguish sequence positions.

For position $p \in \{0,\dots,L-1\}$ and even dimension index $2i$ :

PE (p, 2 i) = \sin (\frac{p}{^{}}

Backlinks1

250529

Comments0

No comments yet. Be the first to share your thoughts.

Comments

10000

2 i / d_{model}

)

,

\text{PE}(p,2i) = \sin\left(\frac{p}{10000^{2i/d_\text{model}}}\right),

PE (p, 2 i) = sin (\frac{p}{1000 0 ^{2 i / d_{model}}}),

\text{PE}(p,2i+1) = \cos\left(\frac{p}{10000^{2i/d_\text{model}}}\right).

Properties

Adds deterministic, continuous vectors; no learned parameters.
Sinusoids at different frequencies make any distance between two positions representable as a linear function of their encodings.

Alternatives

Learned absolute encodings a trainable lookup table of size L × d.
Relative encodings (e.g., Shaw et al., 2018, T5, Transformer-XL): inject pairwise distance directly into attention logits; generalizes to longer sequences.
Rotary positional embedding (RoPE) rotates query/key vectors in complex space by an angle proportional to position; preserves distance under dot product.
ALiBi, xPos, μPPE, etc. linear bias, axis-scaled sinusoids, or mixture-of-experts to extend context or stabilize extrapolation.

Use: add (or concatenate) the positional vector to each token's embedding before the first transformer layer, or incorporate it inside the attention score computation for relative variants.

Comments0

No comments yet. Be the first to share your thoughts.