Dropping by just to cheer you on! I've tried the other methods (SD-CN-Animation and TemporalNet) and your script is still by far the most reliable and flexible method.
Any word on the update to upload custom masks for the frames?
This would allow Higher resolution workflows though tiling sections together and much finer control. Also the batch idea for the guide image would be great to help guide the animation with control net better. fantastic script!! I'm figuring out some workflows will update with results soon.
Would you be able to update your script to work with the latest version of a1111? The one mentioned below that supports your script still didn't support AMD cards so the rendering is unbearably low. Thanks in advance!
Works great but you have to use an older webui version (git checkout a9eef1fbb1dcdce4f0eb0b8e0f79dcd4c96713e1) as described in MonsterMMORPG's video, hope this can get updated to the latest version of webui!
what would be the minimum vram for this i keep getting out of memory even when running control next on low vram, it works fine normally for like batch img2img with control net so confused why
INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 1 images in a total of 1 batches.
Loading model from cache: control_sd15_canny [fef5e48e]
---
I do have JPG as my default save format, I changed it to PNG and still have the same error. I'm going to let it render out, it's going to be a while, When it's done I'm going to disable the dynamic_prompts and see if that does it.
Getting "UnidentifiedImageError: cannot identify image file" error when I click the generate button. Guide frames are selected. I can see the temp image file was created but with 0kb in size. Thanks in advance!
The script doesn't work with the current version of Automatic 1111 web UI. After checkout an old version of web UI, it works well. For more details, you can watch the video bellow posted by MonsterMMORPG, start from 13:53.
Hi Xanthius! Love your work here - I'm the founder of https://kaiber.ai - a generative AI video platform. We recently worked on Linkin Park's LOST music video - was wondering if you were interested in consulting/contract work. Drop me a line at eric@kaiber.ai if so!
Ah yes, it works finally! thanks again for the the code tweak! I would have never figured out how to fix this by myself! I've been using the classic batch img to img and it gave excellent results already but I can't wait to see and share what I can get from Xanthius's script!
Great tutorial! Do you know how this would work with Google Colab? I'm very new to this and haven't found a way to roll back to the earlier commit. I'm also getting the "UnidentifiedImageError" with the latest version.
Same error, I decided to assemble a 1500€ PC because this script's results looked so good (when it works). I even tried to modify the line of code associated with the error in order to remove the "8lqje9jfr" by replacing the 77th line of code by this:
but that's not it. Tried different image sequence renderers as I thought the images where faulty, I doubt it is. Tried jpeg, png, not it. I tried with and without controlnet still no. Maybe it was the resolution I made sure it all matched, not it.
I will use the standard control net for animation for now until it's solved.
I am making a tutorial video right now for this script. I made it work easy but not with latest automatic1111. the biggest problem of the script that it is uploading instead of directory. working on that too
Keep us posted! until then, I'm not giving up. I'm stilll trying to understand what is happening. I stupidly thought that these kind of files "'C:\\Users\\Username\\AppData\\Local\\Temp\\out-00008lqje9jfr.png'" were basically our guiding frames, but evidently not. It's not even in the same directory. When I look at the properties of said file, it's actually an empty file, it says: 0byte file size... that might be why it's unidentified.
Used some deflicker in Resolve to clean it up. Next step would be to create a mask (maybe using depth maps) to do loopback cleanup. In this sample the left arm (on the right side of the video) could definitely be cleaned up with in-painting so it would be cool to script that with a depth-map to inpaint just that section. Way above my paygrade I am just leech reaping the benefits of all you kind open source making souls.
take a look at the console output, it will give you a hint. For me it mostly fails because of memory issues. In that case use less ControlNets and/or enable the Low Vram mode.
I think it's a really great work but I have a problem after trying it. I use recommandation settings to render my frames, then the frame2 and frame3 have a bad color degradation compared with frame1, even if they are same pictures. Fortunately after frame3 there is no longer color degradation. It makes the step1 style transfer become no use, is this an unfixed bug?
I appreciate the desire to tip but unfortunately Paypal is the only thing I have setup that can accept tips like that (unless you deal with crypto or something where I could make a keypair quickly). No need to concern yourself with tipping me though. People have been more generous than I expected already (especially the people that came here after the PromptMuse video dropped)
I havent tested with any 2.X since ControlNet was released for it, but it works great on every 1.5 model I have tested with although I havent tested pix2pix at all
The solution of CGJackB did it for me. Got the error with 330 files but not with 30. I found the lenght check that is throwing the error in line 38 of the multiframe script. Any idea what the limitation here is or how to make it able to run more files at once?
This works pretty well in the beginning. But inside the tunnel it is too dark to give the ControlNet enough info for a consistent animation. I will try again with another sequence inside the tunnel, after the camera adapted to the darkness.
Unfortunately I have a problem with the tool, although I can not understand the cause.
I think I have tried all imaginable combinations for the last 3 days and no matter what settings I use, the frames become more and more "noisy" or "pixelated" over time.
See the attached pictures.
First image is from frame 0
Second image is from frame 10
Has anyone encountered this problem before or have any idea what is causing it?
the Allow other script to control this extension is on. Please find attached the screenshot of the settings in the img2img tab. The controlNet settings are written on the right side in the text. I used the models mlsd, depth-leres, hed and segmentation.
FYI: AntiBurn is off. It also doesn`t matter if I use all, some or none StabeNet Moduls, the issue is allways there.
Thank you for your great work! It is a novel and impressive idea.
I am unfamiliar with SD webui as I do not have a local PC to run it. But technically, I am wondering how this method can use two kinds of image conditions (ControlNet input and inpainting mask). Is the basic inpainting model (like official sd-v1.5-inpainting) works well on the official ControlNet, even if the ControlNet model is not pretrained based on that inpainting model?
An inpainting-mask layer isn't implemented into the script yet but it's planned for an upcoming update so you can do stuff like only modifying areas of the video. As for how ControlNet works on various models, it seems to work on dreamboothed models just fine, so all the ones people download should work. I ran my tests on RealisticVision1.4 just because I like that model in general but I haven't tested if inpainting-specific models do better or worse.
Thank you for your reply! I guess a branch for the inpainting mask is already used in the current version, as the frame-wise mask is used as p.image_mask input. In addition, I just found that the webui will guess the model type by counting the inputs' dimensions. The program seems to automatically choose the inpainting version as the base model. I am wondering that did you download all checkpoints of RealisticVision1.4, including the inpainting version? Thanks!
I do have the inpainting model for RV1.4 and I used that version a couple times, but I generally just used the normal version for the tests and I haven't compared the inpainting vs non-inpainting models yet. As for the inpainting mask I think I misunderstood what you meant. It does use an image mask in order to do the processing, but there isn't currently a way to upload custom masks for the frames themselves and I thought you were asking about that.
help, my images are very dark,I have tried many methods, such as replacing ckpt or vae, checking "Settings: Apply color correction to img2img results to match original colors", and reducing resolution. But the image hasn't changed anything.
What are the settings that you're using? I suspect it might be too low of a denoising strength but it would be helpful to see the settings as a whole in order to help
I usually use 1.0 denoise strength but for most images I'd suggest going with 0.75-1.0, so keep it high. With the black background it also could be more prone to darkening from either the color correction or the loopback (i.e. denoising below 1)
Hello! How can I use your add-on on a remote server? When you click "add files" the download window opens from my computer, but the program itself is running on another server and the files are not uploaded there.
As for the batches, I have been debating how they should work with it. I'm working on a system where you can have it generate multiple versions of each new frame then have it try to pick the best one at each iteration to get better consistency. I believe that would probably be the most useful way to use batch processing for it but it's not quite ready yet.
probably nothing as complex or computationally expensive as that but I'm looking to perhaps use some of the control_net preprocessors to help reduce details like stray hairs from manifesting or disappearing randomly, as well as some color checking to keep color-flickering to a minimum
Hi, I can't figure out why the script give me only the first frame, the subject don't change pose like is not following the controlNet, I followed the guide step by step, and here are my settings. What m I doing wrong? I also clciked on upload guide frames, select all frames and then open.
Did you allow the script to control the extension in Settings/ControlNet? You need to enable the option and do a complete restart of A1111 UI, as it seems to set the permissions on startup. Had the same problem, doing that made it work for me.
In this experiment I used the Dreamshaper model that generates pretty nice illustrations. The ControlNet 'depth' alone, however, is not so great. As we can see it struggles with the details when the fingers are near the mouth and so the AI starts to create a horrible act of chewing finger nails.
Looks great! I think the best method would be if you used something like blender and had it render perfect depthmaps instead of using the preprocessor, but I haven't actually tested that yet. I have heard a lot about https://toyxyz.gumroad.com/l/ciojz for that kind of thing but I have never tried it myself and I don't know how it would work for animations. I would like to implement the ability to add multiple controlnet inputs rather than just the guiding frames but the issue is that, as far as I'm aware, I can only change the "control_net_input_image" property to impact them all at once and I cannot set them individually with a script.
← Return to tool
Comments
Log in with itch.io to leave a comment.
Hello! Sent a few $ over yesterday and grabbed the script. Getting errors thought.... is it still being updated at all?
Dropping by just to cheer you on! I've tried the other methods (SD-CN-Animation and TemporalNet) and your script is still by far the most reliable and flexible method.
Could you make a video tutorial please? I can't seem to get how it's done due to struggles with just text descriptions.
You can check my video:
the most amazing script ever, thank you so much for this
Any word on the update to upload custom masks for the frames?
This would allow Higher resolution workflows though tiling sections together and much finer control.
Also the batch idea for the guide image would be great to help guide the animation with control net better.
fantastic script!!
I'm figuring out some workflows will update with results soon.
Still works fine with my updated a1111 :) though there seems to be a limit to how many init images I can upload.
Would you be able to update your script to work with the latest version of a1111? The one mentioned below that supports your script still didn't support AMD cards so the rendering is unbearably low. Thanks in advance!
Works great but you have to use an older webui version (git checkout a9eef1fbb1dcdce4f0eb0b8e0f79dcd4c96713e1) as described in MonsterMMORPG's video, hope this can get updated to the latest version of webui!
what would be the minimum vram for this i keep getting out of memory even when running control next on low vram, it works fine normally for like batch img2img with control net so confused why
The most up to date and detailed tutorial here. Moreover, I have shared file path instead of upload. Works on runpod as well
im not able to get this to work, i simple get the same image for the output. does anyone know a solution to this error?
allow other scripts to control this extension on controlnet
I'm getting an error. All my extensions are up to date and it's saving the images, but every image save I'm getting -
ERROR:sd_dynamic_prompts.callbacks:Error save prompt file | 20/18320 [01:21<22:37:33, 4.45s/it]
Traceback (most recent call last):
File "F:\stable-diffusion-webui\extensions\sd-dynamic-prompts\sd_dynamic_prompts\callbacks.py", line 24, in on_save
png_info = image_save_params.pnginfo["parameters"]
KeyError: 'parameters'
INFO:sd_dynamic_prompts.dynamic_prompting:Prompt matrix will create 1 images in a total of 1 batches.
Loading model from cache: control_sd15_canny [fef5e48e]
---
I do have JPG as my default save format, I changed it to PNG and still have the same error. I'm going to let it render out, it's going to be a while, When it's done I'm going to disable the dynamic_prompts and see if that does it.
Disabling dynamic_prompts removed the error.
The most up to date and detailed tutorial released. Moreover, I have shared file path instead of upload. Works on runpod as well
I see there is a new third frame option called "GuideImg". What does that do? Can you please update the docs in this page about this? thanks
GuideImg just means that it uses the Guide Image for that frame, which is the same image passed to the ControlNet
thanks!
Getting "UnidentifiedImageError: cannot identify image file" error when I click the generate button. Guide frames are selected. I can see the temp image file was created but with 0kb in size. Thanks in advance!
sxy9699 said:
I found an easy way to solve this problem: simply change the code in multi_frame_render.py
to
Thanks for this quick fix!
In case you want to upload single files instead of a whole folder change the line to this:
The script doesn't work with the current version of Automatic 1111 web UI. After checkout an old version of web UI, it works well. For more details, you can watch the video bellow posted by MonsterMMORPG, start from 13:53.
Hi Xanthius! Love your work here - I'm the founder of https://kaiber.ai - a generative AI video platform. We recently worked on Linkin Park's LOST music video - was wondering if you were interested in consulting/contract work. Drop me a line at eric@kaiber.ai if so!
The most up to date and detailed tutorial released
Thanks!
Ah yes, it works finally! thanks again for the the code tweak!
I would have never figured out how to fix this by myself!
I've been using the classic batch img to img and it gave excellent results already but I can't wait to see and share what I can get from Xanthius's script!
Great tutorial! Do you know how this would work with Google Colab? I'm very new to this and haven't found a way to roll back to the earlier commit. I'm also getting the "UnidentifiedImageError" with the latest version.
Thanks!
Dear developer please add folder path option instead of upload.
With new update script doesn't work anymore, keep reciving this error:
UnidentifiedImageError: cannot identify image file 'C:\\Users\\Username\\AppData\\Local\\Temp\\out-00008lqje9jfr.png'
Same error, I decided to assemble a 1500€ PC because this script's results looked so good (when it works). I even tried to modify the line of code associated with the error in order to remove the "8lqje9jfr" by replacing the 77th line of code by this:
p.control_net_input_image = Image.open(reference_imgs[i].name.replace("\\\\", "\\")[:-12] + reference_imgs[i].name.replace("\\\\", "\\")[-4:]).convert("RGB").resize((initial_width, p.height), Image.ANTIALIAS)
but that's not it.
Tried different image sequence renderers as I thought the images where faulty, I doubt it is.
Tried jpeg, png, not it.
I tried with and without controlnet still no.
Maybe it was the resolution I made sure it all matched, not it.
I will use the standard control net for animation for now until it's solved.
I am making a tutorial video right now for this script. I made it work easy but not with latest automatic1111. the biggest problem of the script that it is uploading instead of directory. working on that too
it will be on my channel https://www.youtube.com/@SECourses
Keep us posted! until then, I'm not giving up.
I'm stilll trying to understand what is happening. I stupidly thought that these kind of files "'C:\\Users\\Username\\AppData\\Local\\Temp\\out-00008lqje9jfr.png'" were basically our guiding frames, but evidently not. It's not even in the same directory.
When I look at the properties of said file, it's actually an empty file, it says: 0byte file size... that might be why it's unidentified.
here tutorial released
here a tutorial
here tutorial released :
I found an easy way to solve this problem: simply change the code in multi_frame_render.py
to
like this:
it works for me!
thanks a lot! It worked.
Same same
take a look at the console output, it will give you a hint. For me it mostly fails because of memory issues. In that case use less ControlNets and/or enable the Low Vram mode.
I think it's a really great work but I have a problem after trying it. I use recommandation settings to render my frames, then the frame2 and frame3 have a bad color degradation compared with frame1, even if they are same pictures. Fortunately after frame3 there is no longer color degradation. It makes the step1 style transfer become no use, is this an unfixed bug?
Hello,
Is there a different way to tip you? I don't use PayPal.
Thanks!
I appreciate the desire to tip but unfortunately Paypal is the only thing I have setup that can accept tips like that (unless you deal with crypto or something where I could make a keypair quickly). No need to concern yourself with tipping me though. People have been more generous than I expected already (especially the people that came here after the PromptMuse video dropped)
Well, except a sincere thank you and a virtual high-five then 🤚
Glad to hear others have been generous
what category to put the plug-in in, i use the web version of StableDiffusion
I can't really make it work, recorded a sceen capture.
https://capture.dropbox.com/QB4hVnHsGhGAjB4B
I havent tested it with pix2pix and I dont know how inpainting works for that model, have you tried it with any other models?
Yes, 1.5 is the other one I have and I still can't make it work.
Do I strictly need SD 2.0?
I havent tested with any 2.X since ControlNet was released for it, but it works great on every 1.5 model I have tested with although I havent tested pix2pix at all
Tks, I'll try again without pix2pix.
Hello, I heep getting this error: TypeError: object of type 'NoneType' has no len()
Any idea of what it can be? Tks
I have only seen that error when guide frames aren't uploaded from the interface
I was getting this error when I was trying to do 120 frames. I tried fewer frames, like 35, to see if that would work and it did.
So try using fewer frames to see.
The solution of CGJackB did it for me. Got the error with 330 files but not with 30. I found the lenght check that is throwing the error in line 38 of the multiframe script. Any idea what the limitation here is or how to make it able to run more files at once?
Did you figure it out since? I have 250 images and it throws the same error
Here's what the source material looks like: http://twitter.com/finefingames/status/1388535835985948681?cxt=HHwWksC93Y3_iMUmA...
And here you can download a 614 frame sequence of this clip as 512x512px PNGs and try for yourself: http://finefin.com/tmp/TunnelSequence-614frames-512x512.zip
Unfortunately I have a problem with the tool, although I can not understand the cause.
I think I have tried all imaginable combinations for the last 3 days and no matter what settings I use, the frames become more and more "noisy" or "pixelated" over time.
See the attached pictures.
First image is from frame 0
Second image is from frame 10Has anyone encountered this problem before or have any idea what is causing it?
Is the "Allow other script to control this extension" setting in ControlNET ON? Show us your settings in the img2img tab.
the Allow other script to control this extension is on. Please find attached the screenshot of the settings in the img2img tab. The controlNet settings are written on the right side in the text. I used the models mlsd, depth-leres, hed and segmentation.
FYI:
AntiBurn is off.
It also doesn`t matter if I use all, some or none StabeNet Moduls, the issue is allways there.
I have an GeForce GTX 1080
Denoising strength should be around 1.
check this vid:
Thanks for your reply. I think, this was my problem. Wasn`t quite shure, which denoise strength should be set to 0. First tests looking promising.
I discovered that removing the background helps with maintaining consistency. I'm using this extension:
https://github.com/AUTOMATIC1111/stable-diffusion-webui-rembg
A short video about it:
Thank you for your great work! It is a novel and impressive idea.
I am unfamiliar with SD webui as I do not have a local PC to run it. But technically, I am wondering how this method can use two kinds of image conditions (ControlNet input and inpainting mask). Is the basic inpainting model (like official sd-v1.5-inpainting) works well on the official ControlNet, even if the ControlNet model is not pretrained based on that inpainting model?
An inpainting-mask layer isn't implemented into the script yet but it's planned for an upcoming update so you can do stuff like only modifying areas of the video. As for how ControlNet works on various models, it seems to work on dreamboothed models just fine, so all the ones people download should work. I ran my tests on RealisticVision1.4 just because I like that model in general but I haven't tested if inpainting-specific models do better or worse.
Thank you for your reply! I guess a branch for the inpainting mask is already used in the current version, as the frame-wise mask is used as p.image_mask input. In addition, I just found that the webui will guess the model type by counting the inputs' dimensions. The program seems to automatically choose the inpainting version as the base model. I am wondering that did you download all checkpoints of RealisticVision1.4, including the inpainting version? Thanks!
I do have the inpainting model for RV1.4 and I used that version a couple times, but I generally just used the normal version for the tests and I haven't compared the inpainting vs non-inpainting models yet. As for the inpainting mask I think I misunderstood what you meant. It does use an image mask in order to do the processing, but there isn't currently a way to upload custom masks for the frames themselves and I thought you were asking about that.
help, my images are very dark,I have tried many methods, such as replacing ckpt or vae, checking "Settings: Apply color correction to img2img results to match original colors", and reducing resolution. But the image hasn't changed anything.
What are the settings that you're using? I suspect it might be too low of a denoising strength but it would be helpful to see the settings as a whole in order to help
"FirstGen" and "InputFrame".I have tried 0.2~0.9 of Denoise Strength
I usually use 1.0 denoise strength but for most images I'd suggest going with 0.75-1.0, so keep it high. With the black background it also could be more prone to darkening from either the color correction or the loopback (i.e. denoising below 1)
I use a white background,I tried 1.0 denoise strength
still incorrect. Looks like it's lost color
what are all the settings and prompt you used?
Hello! How can I use your add-on on a remote server? When you click "add files" the download window opens from my computer, but the program itself is running on another server and the files are not uploaded there.
For future updates, is it possible for your script to have batch img2img as part of the process as well?
Looks fantastic!
As for the batches, I have been debating how they should work with it. I'm working on a system where you can have it generate multiple versions of each new frame then have it try to pick the best one at each iteration to get better consistency. I believe that would probably be the most useful way to use batch processing for it but it's not quite ready yet.
A content-aware system? Wow, that'd be fantastic. Is that something like what SD/DD Warpfusion does on their implementation, ie optical flow?
probably nothing as complex or computationally expensive as that but I'm looking to perhaps use some of the control_net preprocessors to help reduce details like stray hairs from manifesting or disappearing randomly, as well as some color checking to keep color-flickering to a minimum
i put the script in correct folder, the mult frame appears in my scripts tag, but the controls of the script dosent... i need to seetings something?
what happens when you run it?
Hi, I can't figure out why the script give me only the first frame, the subject don't change pose like is not following the controlNet, I followed the guide step by step, and here are my settings. What m I doing wrong? I also clciked on upload guide frames, select all frames and then open.
And this is the result
Did you allow the script to control the extension in Settings/ControlNet? You need to enable the option and do a complete restart of A1111 UI, as it seems to set the permissions on startup. Had the same problem, doing that made it work for me.
yeah option was already enable, I figure out is related with denoise strenght.
Try to select "input frame" at last option
input frames, gave me different poses. but I would like to know why previous frame doesn't work for me.
here's another one
In this experiment I used the Dreamshaper model that generates pretty nice illustrations. The ControlNet 'depth' alone, however, is not so great. As we can see it struggles with the details when the fingers are near the mouth and so the AI starts to create a horrible act of chewing finger nails.
Looks great! I think the best method would be if you used something like blender and had it render perfect depthmaps instead of using the preprocessor, but I haven't actually tested that yet. I have heard a lot about https://toyxyz.gumroad.com/l/ciojz for that kind of thing but I have never tried it myself and I don't know how it would work for animations. I would like to implement the ability to add multiple controlnet inputs rather than just the guiding frames but the issue is that, as far as I'm aware, I can only change the "control_net_input_image" property to impact them all at once and I cannot set them individually with a script.
thank you, I will have a look at the Character bones thing.
and sd-webui-controlnet has an API that allows you to target individual ControlNet models:
https://github.com/Mikubill/sd-webui-controlnet/wiki/API
example is here: https://github.com/Mikubill/sd-webui-controlnet/blob/main/scripts/api.py