1. Takeaways: What Should You Learn?
This session is not about proving that Transformer forecasting is universally better than traditional forecasting. It is about understanding a different modeling lens: represent history as tokens, let those tokens exchange information, and map the learned representation to a future horizon.
By the end, you should be able to explain:
- What a generic Transformer does at a high level.
- What tokens, embeddings, position encoding, and attention mean.
- Why token design matters for time series forecasting.
- How PatchTST turns a time series into patch tokens.
- What channel-independence means, and what it does not mean.
- Why PatchTST uses a Transformer encoder and prediction head, not a GPT-style decoder.
- How to read the paper's key figures and tables without getting lost in notation.
Core narrative: PatchTST is not "Transformer magic for forecasting." It is a paper about choosing the right tokens for time series, so that a mostly standard Transformer encoder can learn useful representations of historical patterns.
2. Forecasting Problem Framing: Same Discipline, Different Modeling Language
PatchTST still solves a familiar forecasting problem: use past observations to predict future values. What changes is how the model represents the past.
Traditional forecasting lens
We often reason in terms of trend, seasonality, lag dependence, known covariates, residual structure, changepoints, and rolling backtests.
The modeler often has an explicit story about each component.
Transformer forecasting lens
The model receives a historical window, turns it into tokens and learned vectors, lets different parts of history interact, and maps that representation to a future horizon.
The structure is learned more implicitly from data.
| Familiar forecasting idea | PatchTST version | How to say it in the session |
|---|---|---|
| Lag features | Attention over historical patch representations | Attention is a learned, sample-specific relevance mechanism over pieces of history. |
| Rolling window | Lookback window L | The model reads a fixed amount of historical context. |
| Forecast horizon | Prediction length T | The model predicts the next T future values. |
| Local shape / recent dynamics | Patch token | A patch can represent a ramp, dip, plateau, spike, or noisy fluctuation. |
| Feature engineering | Learned projection into latent space | The model learns local features instead of relying only on hand-built lag features. |
| Model validation | Still time-based validation | Deep learning does not remove forecasting hygiene. |
Important: Deep learning changes the model class. It does not remove time-based splits, leakage control, fair baselines, or careful interpretation of error metrics.
3. Transformer Basics After the Video
We will first spend about 30 minutes watching the 3Blue1Brown attention lesson. After that, we only need a small vocabulary to understand PatchTST.
Video to watch
3Blue1Brown lesson: Attention in transformers, step-by-step.
Corresponding YouTube video: Attention in transformers, step-by-step.
Do not focus on large-language-model scale. Focus on the mechanism: tokens update their representations by attending to other tokens.
| Term | Plain-English meaning | Forecasting translation |
|---|---|---|
| Token | The unit the Transformer reads. | In PatchTST, a token is a time-series patch. |
| Embedding | A vector representation of a token. | A local time-series shape becomes a learned feature vector. |
| Position encoding | A signal that tells the model where the token is. | The model knows whether a patch is recent or far back in the lookback window. |
| Attention | Learned relevance between tokens. | The model learns which historical patches should inform the representation. |
| Encoder | Reads input tokens and creates contextual representations. | PatchTST uses an encoder over historical patch tokens. |
Key clarification: PatchTST is encoder-only. It is not GPT-style next-token generation. The Transformer encoder reads historical patch tokens; a prediction head maps the resulting representation directly to the forecast horizon.
4. Figure 1 First: The Whole Paper in One Picture
Figure 1 should come before the result tables and before the learning-mode details. It is the map of the paper.
How to walk through the figure
- Start with the multivariate historical window: the input has
Mchannels and a lookback lengthL. - Split by channel: each channel is treated as its own univariate series for the forward pass. This is channel-independence.
- Normalize each series instance: the model reduces scale effects so it can focus more on shape.
- Cut the history into patches: a patch is a short local subsequence and becomes the Transformer token.
- Project patches and add position: the model learns what each patch looks like and where it appears in the lookback window.
- Use a Transformer encoder: attention lets patches exchange context and become richer representations.
- Use a prediction head: the encoded history is mapped directly to the future horizon
T. - Lower path: for self-supervised pretraining, some patches are masked and the model reconstructs them.
So what? The paper does not invent a complicated new attention mechanism. It changes how time series are presented to a standard Transformer encoder: split by channel, cut into patches, embed patches, add position, encode the sequence, and predict the future horizon.
5. Key PatchTST Concepts and Intuition
These definitions are the minimum vocabulary needed to read the method and results.
Patch and patching
A patch is a short subsequence of the time series. Patching means cutting each univariate series into patches of length P, usually with stride S.
Why one point is weak
A single value like 120 does not tell us whether we are seeing a ramp, peak, drop, plateau, or noisy fluctuation.
Why a patch helps
A patch like [80, 95, 112, 130, 145] gives the model local shape. This looks like a ramp, which is closer to a meaningful token.
Patching makes time-series tokens more like phrases than characters. A local phrase carries more semantic information than one isolated scalar.
Linear projection and position encoding
A patch is mapped into a latent vector of dimension D, then a learnable position encoding is added.
linear projection = what does this local patch look like?
position encoding = where is this patch in the lookback window?This does not destroy temporal order. It turns raw values into learned features while preserving order information through position encoding.
Channel-independence
Each channel has its own forward pass, but all channels share the same embedding and Transformer encoder weights.
Clarification: channel-independence does not mean the real-world variables are statistically independent. It means the model processes each channel separately while sharing a common temporal pattern learner.
For a dataset with many sensors, customers, or meters, the model can learn reusable temporal motifs such as ramps, peaks, dips, and seasonal repetitions. Each channel can form its own attention pattern, while the temporal feature extractor is shared.
Encoder-only forecasting
PatchTST uses a Transformer encoder plus a prediction head. The encoder reads historical patch tokens and builds a representation of the past. The prediction head maps that representation directly to future T steps.
This is direct multi-horizon forecasting, not GPT-style step-by-step generation. That is why a Transformer decoder is not central to the architecture.
Paper notation
| Symbol | Meaning | Seminar translation |
|---|---|---|
M | Number of channels | How many variables, sensors, stores, or meters are in the multivariate series. |
L | Lookback window | How much historical context the model receives. |
T | Forecast horizon | How many future steps the model predicts. |
P | Patch length | How many consecutive time steps form one token. |
S | Stride | How far the patch window moves each time. |
N | Number of patches | How many patch tokens the Transformer sees. |
D | Latent dimension | The size of each learned patch vector. |
6. What Is Novel?
PatchTST's main contribution is not a new attention formula. Its main contribution is a better input design for applying a Transformer encoder to time series.
Patch tokens
The model reads local subsequences rather than isolated scalar values.
Lower attention cost
Fewer tokens means a smaller attention matrix, because attention cost grows roughly with token count squared.
Longer lookback
Because patching reduces token count, the model can use longer historical windows.
Channel-independence
Each channel has its own forward pass while sharing a temporal representation learner.
Encoder-only design
The paper shows that a vanilla Transformer encoder plus a forecast head can be strong when tokenization is right.
Masked pretraining
The model can learn representations by reconstructing masked historical patches.
7. Main Forecasting Evidence: Supervised Results First
The result discussion should follow the paper's primary task first: long-term forecasting. Representation learning and transfer come after this.
Figure 2: varying lookback length
So what? The point is not just that PatchTST is accurate. The point is that the patching design changes the model's ability to use historical context.
Table 3: supervised long-term forecasting benchmark
So what? Table 3 answers the primary question first: does PatchTST work as a forecasting model on standard long-horizon benchmarks? It is evidence that the architecture is worth understanding and testing, not proof that Transformers always beat traditional forecasting in production.
8. Representation Learning and Transfer: High-Level Only
This section is intentionally shorter than the method and main forecasting evidence. The goal is to make Table 4 and Table 5 readable, not to teach every pretraining detail.
| Mode | What is trained? | Target | Intuition |
|---|---|---|---|
| Supervised from scratch | All PatchTST weights start random. | Future time-series horizon. | Ordinary supervised forecasting. |
| Self-supervised pretraining | The backbone learns from masked historical patches. | Reconstruct masked patches. | Learn time-series representations without future labels. |
| Linear probing | Freeze pretrained backbone; train only the forecast head. | Future time-series horizon. | Test whether the representation is already useful. |
| Fine-tuning | Start from pretrained weights; train the model on forecasting labels. | Future time-series horizon. | Adapt the representation to the forecasting task. |
| Transfer learning | Pretrain on one dataset, adapt on another. | Future horizon on the new dataset. | Test whether representations transfer across datasets. |
Important correction: Table 4 and Table 5 are still forecasting experiments. They are not classification experiments. The metrics are MSE and MAE on future time-series values.
Table 4: self-supervised PatchTST
So what? If linear probing works well, the pretrained representation already contains useful forecasting information. If fine-tuning improves further, the pretrained backbone provides a strong initialization that can adapt to the target forecasting task.
Table 5: transfer learning
So what? The result suggests that PatchTST can learn reusable temporal representations. Channel-independence helps because the pretrained backbone is not tightly tied to the number of channels in the pretraining dataset.
9. Resources and Further Reading
- Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, Jayant Kalagnanam. A Time Series Is Worth 64 Words: Long-Term Forecasting with Transformers. ICLR 2023.
- Official implementation: yuqinie98/PatchTST.
- 3Blue1Brown lesson: Attention in transformers, step-by-step, and the corresponding YouTube video.
- Original Transformer paper: Attention Is All You Need.