Hi,
Can anyone point me a model that is able to do Smart Object Swap using a reference object. So the main source is not a prompt but are images. A base image where for example a door needs to be replaced with another door that is presented as a reference object. If that is to difficult a third image with the masked area is also an option. I have tried numerous models but until now no success.
Thanks in advance!
Jos
What about image-to-image models?
You are looking for a model that can do reference-driven object replacement, not just ordinary prompt-based editing.
That distinction is the whole story.
Most image editors are good at:
But you want something stronger:
Take object B from a reference image, and install it into image A, ideally in a chosen region, while preserving the rest of image A.
That is a much narrower and harder task. It sits between inpainting, image editing, reference-guided generation, and object-level compositing. That is why so many otherwise impressive models have not worked for you. (openaccess.thecvf.com)
For your use case, the useful models fall into two families:
These are the ones that are conceptually closest to your task:
These are the best when you can provide:
These are newer, broader, more modern image editors that can often do the job if you set them up well:
black-forest-labs/FLUX.2-klein-4BQwen/Qwen-Image-Edit-2511FireRedTeam/FireRed-Image-Edit-1.1meituan-longcat/LongCat-Image-Editblack-forest-labs/FLUX.2-dev if you can handle a large modelMy real opinion is:
That is the cleanest summary I can give.
If I were doing this myself, I would not hunt for a single “perfect” model first.
I would use this strategy:
black-forest-labs/FLUX.2-klein-4B
because it is:
Qwen/Qwen-Image-Edit-2511
because the Diffusers docs explicitly support multi-image reference workflows, and Qwen has become one of the safest open editing baselines on Hugging Face. (huggingface.co)
FireRedTeam/FireRed-Image-Edit-1.1
because its model card explicitly emphasizes identity consistency and multi-image conditioning, which are exactly the two things that usually break in reference-based object replacement. (huggingface.co)
meituan-longcat/LongCat-Image-Edit
because the card explicitly says it supports local editing and reference-guided editing. That makes it unusually relevant to what you are trying to do. (huggingface.co)
AnyDoor
because the paper is still one of the closest direct matches to “swap object in scene using object reference.” (openaccess.thecvf.com)
MimicBrush
because it is built around source image + selected region + reference image, and is especially good when the edit is more like “make this region become like the reference” than “fully reconstruct a new rigid object from scratch.” (xavierchen34.github.io)
You said a third image with the masked area is an option.
That is not a fallback. That is a major advantage.
In practice, the masked version of your problem is much easier than the unmasked version, because the model no longer has to guess:
So I would strongly reframe the situation like this:
That is one of the biggest reasons people fail with this class of task. They are asking the model to solve localization, object transfer, and blending all at once. (openaccess.thecvf.com)
There are four common failure modes.
It gives you “a similar object,” not the actual reference object.
This is the classic identity-drift problem. It is exactly why I think FireRed is worth testing early, and why AnyDoor still matters conceptually. (huggingface.co)
It changes the surrounding image when you only wanted one region edited.
That is why masked editing and locality-aware models matter so much. (huggingface.co)
The object looks plausible by itself, but does not fit the target scene:
If the reference contains extra background, lighting, or surrounding objects, the model may import the wrong things.
The AnyDoor paper explicitly reports that filtering background information from the reference object helps. (openaccess.thecvf.com)
A research-oriented object-level image customization method designed for tasks like object insertion and object swapping. (openaccess.thecvf.com)
Because its whole framing is basically:
Because the current Hugging Face Space shows a runtime error, so the practical HF experience is not as clean as with the newer families. (huggingface.co)
Excellent conceptual fit.
Not the cleanest beginner-first Hugging Face experience today.
A Diffusers pipeline that explicitly supports:
Because many models say “editing,” but the docs do not clearly spell out the exact local workflow. Kontext does. That makes it one of the most concrete “yes, this really matches your problem” options in the HF ecosystem. (huggingface.co)
If you can provide a mask, this is one of the strongest practical paths, even though newer general models now exist.
A newer FLUX.2 model with:
It hits a rare sweet spot:
It is still a broader general editor family, not a pure object-swap paper architecture. So it may still need strong masking and good setup to shine.
If you ask me for one Hugging Face model to try first today, this is near the top of the list.
A strong open editing model with documented support for multi-image reference workflows in Diffusers. (huggingface.co)
Because it is one of the cleanest current “modern open image editor” stacks:
It is broader than your exact task. It is not a pure object-swap specialist the way AnyDoor is.
This is the model I would use as a strong open baseline. If even this struggles with your case, that is useful information.
A general-purpose image editing model whose card explicitly highlights:
Because the biggest pain in reference-object swap is often:
“The edit happened, but the model did not really preserve the referenced object.”
That is the exact axis where FireRed is trying to improve. (huggingface.co)
If your current attempts produce generic-looking replacements, test FireRed early.
A model card that explicitly says:
That wording is unusually aligned with your problem. It suggests a model that was designed with more structured edit control in mind, not just flashy broad edits.
This is a strong candidate if your problem is mostly:
A project focused on local reference imitation:
Because it directly respects the way you think about the task: not “prompt first,” but “image first.”
It is especially appealing when the edit is about making the target region look like the reference, while preserving more of the original shape. (xavierchen34.github.io)
The current HF Space shows a configuration error. (huggingface.co)
Useful, relevant, but more research/project-flavored than modern HF-native turnkey.
Here is the order I would actually use.
These give the best balance of practicality and relevance:
This wave tells you whether the problem is already solvable with current modern HF-native editors.
If the first wave is close but not quite right:
This wave tells you whether the missing ingredient is more explicit locality/object-level structure, not more raw model power.
Only if you have heavy compute and want to test the ceiling:
These details matter a lot.
Use a tight crop or segmented object if possible.
Do not feed a messy reference image if you can avoid it. (openaccess.thecvf.com)
If possible, the mask should cover the exact region you want replaced, not a huge loose box.
Precise locality is a big part of success. (huggingface.co)
If the region is small relative to the full image, cropping around it helps the model focus. The Diffusers docs explicitly mention this for local inpainting-style workflows. (huggingface.co)
Even in image-led editing, a small prompt helps:
Short, concrete, and visual is usually better than long creative prose.
The FLUX.2 editing app prompt rules are actually quite aligned with that style. (huggingface.co)
If you want the answer reduced to one clear recommendation:
Use a mask and test modern HF-native editors first.
black-forest-labs/FLUX.2-klein-4B
because it is recent, open enough, multi-reference capable, and practical. (huggingface.co)
Qwen/Qwen-Image-Edit-2511. (huggingface.co)
FireRedTeam/FireRed-Image-Edit-1.1. (huggingface.co)
meituan-longcat/LongCat-Image-Edit. (huggingface.co)
AnyDoor. (openaccess.thecvf.com)
Here is the compact, concrete start-here recipe I would use for your task.
Input A — base image
The image you want to edit.
Input B — reference object image
A tight crop of the object you want to insert/transfer. Remove as much background as possible. The AnyDoor paper reports better results when background information around the reference object is filtered out. (openaccess.thecvf.com)
Input C — mask image
A mask of the region to replace. If you can provide this, do it. It makes the task much easier and much more controllable. Diffusers’ FLUX Kontext docs explicitly support image-reference conditioning inside a fixed mask region. (huggingface.co)
Do not always feed the entire full-resolution image first.
If the target region is small, crop around that region plus a little context. The Diffusers docs explicitly note that when the masked region is small compared with the whole image, cropping around it can improve results. (huggingface.co)
FLUX Kontext masked reference workflow
Reason: this is the clearest officially documented Hugging Face path for masked local editing + image-reference conditioning. (huggingface.co)
black-forest-labs/FLUX.2-klein-4B
Reason: it is current, practical, Apache-2.0, and its card explicitly says it supports image-to-image multi-reference editing. (huggingface.co)
Qwen/Qwen-Image-Edit-2511
Reason: it is a strong open baseline, and Diffusers explicitly documents multi-image reference workflows for the Qwen image editing family. (huggingface.co)
meituan-longcat/LongCat-Image-Edit
Reason: the card explicitly says it supports local editing and reference-guided editing. (huggingface.co)
Use something like this:
Replace the masked object with the reference object. Keep the rest of the image unchanged. Match scale, lighting, and perspective.
Do not write a long creative paragraph. Keep it direct and visual.
For each model, do these 3 runs:
This is your baseline.
This checks whether the model needed a bit more freedom around edges.
This checks whether the full image was distracting the model.
Problem: weak locality.
Action: use a better mask, tighter crop, or switch toward Kontext / LongCat. (huggingface.co)
Problem: reference identity drift.
Action: clean the reference crop more aggressively; then try Qwen or FLUX.2-klein-4B again. The AnyDoor paper and newer model cards both point to reference handling as a key issue. (openaccess.thecvf.com)
Problem: geometry / perspective / scene integration.
Action: enlarge the crop around the target area a bit and make the prompt explicitly say match scale, lighting, and perspective.
Problem: blending.
Action: rerun with a slightly larger mask, then do a second cleanup pass.
If you want the fastest sensible path: