Inpaint anything extension in Stable diffusion webUI guide (automatic 1111)

The Inpaint Anything extension. smoothly fixes images using Segment Anything's masks. With Segment Anything, you can point to the areas you want to mask without filling them in manually. This makes creating masks faster and more accurate, resulting in better image fixes while saving time and effort 🤩.

In this blog, we'll learn about stable diffusion painting and how "Inpaint Anything" operates. We'll explore the specialized features available in this extension.

How to install the extension?

To install the extension for inpainting in stable diffusion

• Navigate to the extension tab, click on the available tab,
• Select "load from it". That URL contains numerous JSON files from which the extensions will load
• Search "inpaint anything" then click on the install button
• Next, go to the Installed tab and select "check for updates"
• Ensure you have the latest version of the extension
• click on apply and quit to complete the installation process.

After refreshing the UI, you will see the "Inpaint Anything" tab.

Navigate to that tab to unleash your creativity and craft amazing creations 👀

Inpainting

First, you'll notice the "Segment Anything Model ID" where images are automatically segmented. There's a bunch of models, each model gives a different output. I suggest the "sam_vit_l_0b3195.pth" model because it makes segmentation look more clear. Download the model, upload your image, and click "Run". Your image segmentation appears on the right side, like magic! 🎩✨ This is what is known as AI inpainting.

To select the segmentation object.

Take a look at those colors on the right side. Each color represents an object! Now, when you draw a black line or put dots, it's like a special signal for the segmentation model to apply the segmentation mask in the image. Simply click or draw on the areas you want to modify, then hit "Create Mask"! Easy-peasy😅!

Click on "Expand Mask Region" to increase the size of the mask, which will enhance the results in a super cool way. Alright, I just typed out my prompt "Earth"

My Prompt: Earth

The result is fantastic; it looks real, almost as if it's right in front of you😍

Cleaner

Let me give you the overall idea of how the cleaner works. Picture this: you want to remove an object from a photo or a person in your photo. Well, the cleaner tool does that without messing up the other stuff in your picture. It's like having a digital eraser but without the smudges! Don't forget to hit "Expand Mask Region" for a better result. I picked the lama cleaner model as it provides excellent image output. Check out their GitHub repo 'lama'

Now, take a look at the result - it effectively cleaned up the planet.

Inpainting webui

This tool works a bit like the inpainting tool but with an extra twist - it has a refiner option. In the advanced settings, I can tweak some configurations like sampling methods and sampling steps.

Make sure to enable the check box in the refiner option. You can pick any refiner model and set the Switch at between 0.2 and 0.5. That means the main model does most of the work, and the refiner model adds the finishing touches. It's like having two painters team up to get the job done! 🎨 Let's say, I want to change the earth into Football.

My Prompt: Football

Hit the "Run Inpainting". Wait for the results.

The result is just as you'd predict🤩. The quality of the image is better than the inpainting image. All because of the refiner option.

ControlNet inpainting

Transitioning to ControlNet inpainting. We need to install a few additional extensions. We need to download the sd-webui-controlnet extension in the same manner

• Navigate to the extension tab, click on the available tab
• Select "load from it"
• Search "sd-webui-controlnet" then click on the install button
• Go to the Installed tab and select "check for updates"
• Click on apply and quit to complete the installation process.

It appears like this, indicating the requirement for the ControlNet model.

Click on "ControlNet-v1–1," which will open the Hugging Face page. On the Hugging Face page, copy the download link for the "control_v11p_sd15_inpaint.pth" file.

Open the command prompt and navigate to this directory. Make sure you're in the home directory. Type "cd" to go to the home directory. Paste the below code.

cd stable-diffusion-webui/extensions/sd-webui-controlnet/models.

Use the wget command to download the file, Ensure to rename it with the .pth extension, Navigate to the automatic 1111 dashboard, and refresh the page

wget https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_inpaint.pth?download=true

Use the mv command to rename it.

mv [current_filename] [new_filename]

After that, it'll pop up like a jack-in-the-box, and then we can start messing around with it! 😁

What features are available?

We've got an extra option just like Inpaint WebUI, but with a twist: you can adjust the ControlNet parameters! Here's how it works: it takes the input image as a reference, and you can tweak the ControlNet weight.

Think of it like adjusting the volume on your favorite song: 1 is balanced, but you can adjust it up or down depending on your need. In the ControlNet mode, You get to decide which is more important: your ControlNet reference image or your prompt.

Once you upload the reference image, I usually set the "Reference only" option and choose "Inpaint only" in the ControlNet Preprocessoring option. To get started, I went for the ControlNet v11 sd15 model.

My prompt: "Mountains, Forest , cinematic "

Click "Run ControlNet Inpaint"! And there you have it! The output image is a combination of your reference image and the masked one. It's like mixing two ingredients and seeing what delicious dish you create! 🍳🎨

Result:

Mask only

The "Mask Only" feature is the last but not least tool in this extension. It allows me to extract the mask into separate parts Clicking "Get Mask as Alpha of Image" reveals the unmasked image, while hitting "Get Mask" only gives a mask of the image. From there, we can send it to the img2img tab for image inpainting and cook up something creative on the masked picture! 🎨

If you reading this you've reached the finish line. high five 😅 🙌 ! I hope you figured out some idea how to use the extension.