CASE STUDY 2026

Replacing Qwen's Vision Transformer with ResNet-50

A feature-distillation pipeline for replacing Qwen's visual encoder with ResNet-50 while keeping the language model unchanged.

PyTorch, ResNet-50, Qwen-VL, Vision Transformers, CNN, Feature Distillation

Code ↗

This project starts from a precise question: can Qwen’s Vision Transformer be replaced with ResNet-50 without changing the language model?

The goal was not a generic CNN-versus-Transformer comparison. I wanted to preserve Qwen’s visual interface and test whether a convolutional backbone, combined with a trainable adapter, could produce embeddings compatible with those generated by the original visual encoder.

Qwen expects a sequence with shape [B, 196, 2048]. ResNet-50 instead returns hierarchical feature maps with different spatial resolutions and channel counts. The task is therefore to transform several CNN feature maps into a token sequence that the language model can consume without being retrained.

Pipeline architecture

The custom pipeline contains three parts:

1. ResNet-50 as the visual backbone;
2. a multi-stage adapter that collects features from several network stages;
3. Qwen’s language model, which remains unchanged.

I did not use only the final ResNet layer. Earlier stages preserve edges, textures and local detail, while deeper stages contain stronger semantic information but lower spatial resolution. The adapter combines both.

Each feature map is first projected into a compatible channel space. The maps are then resized and fused progressively. The final blocks convert the spatial representation into 196 tokens with 2048 dimensions each.

Matching the output shape is required, but it is not sufficient. Two tensors with shape [B, 196, 2048] can encode completely different information. The training objective therefore aims to reproduce not only the interface, but also the latent space used by the original ViT.

Feature distillation

Qwen’s original Vision Transformer acts as the teacher.

For each image, the native visual encoder produces a sequence of embeddings. These tensors are stored and used as targets. The same image is then processed by ResNet-50, and the adapter converts the CNN features into the student sequence.

The initial loss is Mean Squared Error:

L = MSE(E_student, E_teacher)

Here, E_teacher is the output of the original ViT and E_student is the output produced by ResNet-50 and the adapter.

This setup separates representation alignment from the final downstream task. The model is not trained directly on captions or class labels. It is trained to reconstruct the visual representation Qwen uses before text generation begins.

Four adapter versions

I implemented four variants to isolate the contribution of each design choice.

V1 uses 1×1 projections and convolutional fusion. It is the simplest baseline and tests whether ResNet features contain enough information to approximate the ViT embeddings.

V2 adds a Transformer Encoder after feature fusion. Convolutions model local context well, but they do not directly capture relationships between distant image regions. The Transformer introduces this global interaction.

V3 changes the fusion stage by using 3×3 kernels. Unlike pointwise projections, these kernels allow each spatial position to incorporate information from its local neighbourhood.

V4 combines the 3×3 fusion with the Transformer Encoder. It first strengthens local spatial processing and then models global dependencies between tokens.

This sequence of experiments separates the effect of spatial fusion from the effect of global attention instead of hiding both inside one large architecture.

Tensor flow

ResNet produces feature maps at several resolutions. The adapter processes them through:

• channel projection;
• spatial-resolution alignment;
• fusion of shallow and deep features;
• spatial compression;
• final conversion into a token sequence.

The output must preserve both the order and dimensionality expected by Qwen. I did not modify the language model to accept an alternative format; all compatibility is handled by the new visual path.

Training strategy

The first stage keeps ResNet frozen and optimizes only the adapter. This gives the adapter a stable input distribution and makes the training process easier to control.

The second stage jointly fine-tunes the backbone and the adapter using different learning rates. ResNet receives a smaller learning rate to avoid quickly damaging its ImageNet-pretrained features, while the adapter uses a larger one.

A short adapter-only phase follows the joint fine-tuning so that the output can be realigned with the teacher after the backbone has changed.

The repository also separates teacher-embedding extraction, adapter training, MSE evaluation and inference with the custom visual encoder.

Qualitative check

The tank image combines text, sharp edges, tracks, mechanical parts and a complex background. It is useful for checking whether the representation retains both the broad object category and the local evidence needed for a more precise description.

This is not a quantitative benchmark. It is a practical comparison between the original and custom visual encoders on an input containing several different types of visual structure.

Limitations and next steps

MSE is a direct objective, but it does not guarantee that every semantic relationship in the teacher space is preserved. Two embeddings can be numerically close and still lead to different generated text.

The most useful extensions would be:

• cosine-similarity objectives;
• contrastive alignment between images;
• layer-wise or token-wise distillation;
• evaluation on captioning and visual question answering;
• comparison of output quality, parameter count and inference cost.

The main result is not a claim that ResNet-50 is already a complete replacement for the ViT. It is a modular pipeline for measuring how closely a CNN can reproduce the latent interface required by a Vision-Language Model.