Can Visual In-Context Learning Handle Unfamiliar Defects?

I like the idea of showing a vision model one annotated example and asking it to do the same thing on a new image. In industrial computer vision this would be genuinely useful. You could try out a customer’s problem before spending time on annotation and training.

Test-time adaptation of visual in-context models made me wonder whether the idea holds up on images that look nothing like the training data. So I spent an evening playing with it.

The idea

Visual Prompting via Image Inpainting (Bar et al., NeurIPS 2022) frames a vision task as a picture puzzle. You stitch an example input, its desired output and a new query image into one grid, leave the fourth cell blank, and let an inpainting model fill it in. If the example output is a segmentation mask, the model is expected to paint a mask for the query.

Diagram: a task input, a task output mask and a query image are concatenated into a grid; an inpainting model fills the empty cell with the query's mask. Below are examples for edge detection, colorization, inpainting, segmentation and style transfer. — Visual prompting: the example pair defines the task, and the model fills in the missing cell. Figure from Bar et al., 2022.

Painter (Wang et al., CVPR 2023) builds a generalist model on the same principle. It is trained with masked image modeling on stitched input–output pairs across many tasks. At inference, a single example pair tells it what to do.

The weakness is that one example isn’t always enough, especially when the test image comes from a different distribution than anything the model was trained on. Test-Time Visual In-Context Tuning (VICT) (Xie et al., CVPR 2025) addresses this by briefly fine-tuning the model on each test sample:

Predict the output for the test image as usual.
Swap roles. Use the test image and its predicted output as the prompt, and ask the model to reconstruct the output for the original example.
The original example’s output is known, so the reconstruction error gives a training signal. A few gradient steps on this cycle-consistency loss adapt the model to the test sample.

VICT pipeline: Painter predicts the output for a test image from a prompt pair; the roles are then flipped, the prediction is used as the prompt to reconstruct the original prompt output, and a regression loss between the reconstruction and the original output updates the model. — The VICT loop: predict, flip the roles, reconstruct the known prompt output, and minimize the difference. Figure 2 from Xie et al., 2025.

It’s worth noting what the paper evaluates: six vision tasks under fifteen common image corruptions such as noise, blur and weather effects. A new kind of industrial defect is a much bigger jump than that.

A quick test on cable defects

I adapted the model using images of steel wire ropes damaged by lightning strikes, then tried it on a different defect in the same domain: broken and frayed wires. This wasn’t a benchmark. I only wanted to see whether the approach produced meaningful masks at all.

Four close-up photos of twisted steel wire ropes with localized melted and scorched spots caused by lightning. — Adaptation data: wire ropes with lightning damage. image source needed

Four photos of steel wire ropes with broken, displaced and frayed outer wires, some with rust. — A different defect in the same domain: broken and frayed wires. image source needed

After 30 test-time tuning steps, the model highlighted the two damaged regions in the test image below.

Left: a prompt image of a damaged rope with its binary defect mask, next to a test image of a rope with two damaged spots. Right: the predicted mask, which marks two blobs at the damaged spots. — Prompt image and mask, test image, and the predicted mask after 30 test-time tuning steps. image source needed

That’s an encouraging result for a quick test on unfamiliar data. But a plausible-looking mask on a couple of images says little about recall, false positives or performance across a production dataset. I didn’t measure any of those.

Would I use it in an industrial project?

Not as the default choice yet. General visual models are very good at representing objects, but specific defects may not be distinct concepts in their pretraining data. A surface feature can be obvious to a quality inspector and still be ambiguous to a foundation model.

My practical starting point would usually be a pretrained visual encoder such as DINO, with a lightweight classifier, adapter or segmentation head trained on its features. That still benefits from large-scale pretraining, while letting the application define exactly what counts as a defect. It’s worth benchmarking that against a visual-prompting approach rather than assuming either one wins.

So, did VICT work? Somewhat, on the examples I tried. Enough to make the idea interesting, and nowhere near enough to call specialized defect segmentation solved. For an evening of experimenting, that’s a useful answer.