Understanding the difference between a CNN and an LLM helps clarify when to use a dedicated classifier versus a general-purpose generative model for image classification. This post examines how a convolutional neural network (CNN) and a multimodal LLM approach the exact same problem—skin lesion image classification—from the perspective of model architecture and training design.
Key summary
A CNN is a dedicated model trained to extract local image patterns hierarchically through convolution filters and pick one of a fixed set of classes, whereas a multimodal LLM is a general-purpose model that converts an image into tokens so a language model can answer in sentences. When a dermatologist compared them on 200 biopsy-confirmed lesions, the author reports that a fine-tuned CNN (Inception v3) achieved 62% accuracy on the first answer among 6 candidate diagnoses, higher than two general-purpose LLMs (42% each). However, the sample is small, each LLM case was run only once, and it is not a comparison against state-of-the-art dedicated models.
What is the difference between a CNN and an LLM?
One-line answer: A CNN is a dedicated image model that picks from fixed classes, whereas a multimodal LLM is a general-purpose model that turns the image into tokens and answers in text.
Comparing dedicated classifiers and general generative models
On an image classification task, a dedicated CNN focuses entirely on extracting visual features from an input photo and mapping them to one predefined label. By contrast, a multimodal LLM takes both images and text as input to generate open-ended sentence responses across many kinds of tasks.
| Aspect | CNN (Dedicated) | Multimodal LLM (General-purpose) |
|---|---|---|
| Input processing | Convolution filters | Image tokens |
| Output | Fixed class probabilities | Sentence text |
| Training data | Target domain | Broad general data |
| Strengths | Narrow, accurate classification | Explanation, conversation, multiple tasks |
How does a CNN see an image?
One-line answer: Small filters slide over the image to detect local patterns, and stacked layers combine them into higher-level features before outputting class probabilities.
Convolution filters and hierarchical feature extraction
In a CNN, a convolution filter acts as a local pattern detector that slides across small regions of the image to spot edges and textures. As layers stack deeper, earlier patterns combine into shapes and structural parts, pooling operations reduce spatial dimensions, and the final layer outputs a probability distribution over the fixed classes.
Transfer learning and fine-tuning
Rather than training from scratch, practitioners typically start from a model pretrained on everyday photos and retrain it on domain images—an approach known as transfer learning and fine-tuning. Retraining on a small domain dataset carries the risk of overfitting to the training examples; for causes and remedies, see What is overfitting? How to fix it when training loss falls but validation loss rises.
How does an LLM see an image?
One-line answer: An image encoder splits the image into patches and turns them into tokens that the language model reads together with the text prompt.
Image patches, tokens, and sentence generation
Instead of feeding the photo straight into convolution filters, a multimodal LLM uses an image encoder to split the photo into patches and convert each patch into a vector (an image token). The language model reads these image tokens alongside the text prompt and generates a sentence answer token by token.
Unless the prompt forces a constrained choice, the model’s output is free text rather than a fixed probability distribution over classes. Given the same photo, a CNN passes it through filters and feature maps to output probabilities over 6 classes, while an LLM turns it into image tokens for the language model to produce a sentence answer, as illustrated below.

What did this experiment compare?
One-line answer: A fine-tuned Inception v3 CNN and two general-purpose LLMs, given the same biopsy-confirmed lesion photos, chose from the same 6 diagnoses.
Experiment setup and results as the author reports
The original author is a dermatologist who designed the test himself. Using Inception v3 (published in 2015) fine-tuned on public dermatology images alongside two general-purpose LLMs, the test presented 200 biopsy-confirmed lesions and asked each model to choose from 6 candidate diagnoses. As the author reports, the fine-tuned CNN achieved 62% on the first answer being correct, compared with 42% each for the two general-purpose LLMs.
Caveats to keep in mind
- The sample is small.
- Each LLM case was run only once and answers can vary between runs.
- Only biopsied, i.e. harder, cases were used.
- The CNN is deliberately not state of the art, and it is not a comparison with the best dedicated models.
Why does a small dedicated model win?
One-line answer: It was trained on exactly this kind of image, it only has to pick one of fixed classes, and it does not carry a general model’s preferences.
Domain fine-tuning and fixed-class optimization
- Domain fine-tuning: The dedicated model sees many images of this exact task during training, tuning its filters to domain-specific visual cues.
- Fixed output classes: Training directly optimizes choosing among the fixed candidate classes rather than generating open-ended prose.
- Favourite answer tendency: The author observed that each general-purpose LLM tended to have a “favourite answer” it chose often.
- Broad optimization target: A general-purpose model is optimized for many tasks rather than this single narrow classification problem.
This does not mean CNNs are always better—with a stronger or newer dedicated model, or a different dataset, results can differ. For another way to build compact task-specific models from larger ones, see What is knowledge distillation? Training a small model from a big one, and how it differs from quantization.
So when should you use which?
One-line answer: Use a dedicated model for classification or detection with a clear label set, use an LLM for explanation, conversation, and summarizing, and often combine both.
Selection criteria by task and combined architecture
- Dedicated model: Best suited for fixed-class classification and object detection where the label set is clearly defined.
- Multimodal LLM: Strong at explaining results, drafting reports, summarizing findings, and handling free-form questions.
- Combined structure: A practical pipeline lets the dedicated model classify the image first and lets the LLM explain or organize that output (or lets the LLM call the classifier as a tool).
Medical decisions must always come from a specialist physician; this post is only a study of model design.
What are the frequently asked questions?
One-line answer: Below are brief answers on how ViT differs from a CNN, what transfer learning means, and when an LLM makes sense for classification.
How does a ViT differ from a CNN?
A ViT (Vision Transformer) splits an image into patches and processes them with self-attention instead of convolution filters.
What is transfer learning?
Transfer learning reuses a model pretrained on a large general dataset and retrains it on a smaller target dataset.
Can you not use an LLM for classification?
It can be useful for quick prototypes or when labeled data is scarce, but for a fixed, well-defined label set with enough labeled data, a dedicated classifier is usually the more natural fit.
What related posts are worth reading?
One-line answer: Read these related posts on knowledge distillation, fixing overfitting during training, and calibrating LLM confidence scores.
- What is knowledge distillation? Training a small model from a big one, and how it differs from quantization
- What is overfitting? How to fix it when training loss falls but validation loss rises
- What is an LLM confidence score? Why you should not just ask the model how sure it is, and what calibration means
What are the sources?
One-line answer: Below are the discussion thread and original article citation for the skin lesion image classification comparison covered in this post.
- Hacker News discussion: a dedicated CNN vs general-purpose LLMs on skin lesion image classification (checked 2026-11-02)
- Original article: lynchlab.net, Dr Magnus Lynch (posted 2026-10-25, read 2026-11-02)