Hi! Here's an open-source implementation I released today for masked ControlNet synthesis, where you can specify the region that will be synthesised using a mask. The content of the synthesised region is controlled via textual and visual guidance as shown in the README.

https://github.com/mikonvergence/ControlNetInpaint

Here's an example with a prompt of "a red panda sitting on a bench":

https://preview.redd.it/4vxsg9sc0lna1.png?width=1860&format=png&auto=webp&v=enabled&s=9776369a86043f9420ec8771dd6f9d22308e521c

Comments

You must log in or register to comment.

PuzzledWhereas991 t1_jc4ema6 wrote on March 13, 2023 at 11:23 PM

This is exactly what I have been looking for this days

kross00 t1_jcdmgl4 wrote on March 16, 2023 at 1:49 AM

Do you plan on releasing an Automatic1111 plugin for this?

mikonvergence OP t1_jceoitq wrote on March 16, 2023 at 8:28 AM

Hi! Do you think many people would be interested? I've never used Automatic1111 since I work in research so I need direct access to the source code.

If there is enough interest, I will definitely check out how to release it as an extension!

imaginethezmell t1_jcnj7ia wrote on March 18, 2023 at 3:04 AM

everyone is interested

are you kidding lol

kross00 t1_jcnr2yo wrote on March 18, 2023 at 4:17 AM

How is it different from the inpain already built in controlnet?

mikonvergence OP t1_jco5znu wrote on March 18, 2023 at 7:16 AM

Which implementation do you have in mind?

When putting this together, neither the official implementation (https://github.com/lllyasviel/ControlNet) nor the diffusers pipeline (https://huggingface.co/docs/diffusers/main/en/api/pipelines/stable_diffusion/controlnet) had the inpainting option built-in (and it seems they still don't?).

While this framework still follows the principle of injecting ControlNet features into a core SD backbone, this core backbone had to be changed to an inpainting one, to allow mask input (and since the input also includes a mask, it is not possible to just specify a different backbone source and reuse an existing pipeline). The pipeline provided in this repository StableDiffusionControlNetInpaintPipeline implements this approach and merges the interfaces of StableDiffusionControlNetPipeline and StableDiffusionInpaintPipeline, so you can provide a source image, a mask, and a control image, and also set all possible parameters to the values you like.