Struggling similarly to others, while it seems to be batch outputting photos, it's really just a slight redrawing of the same frame over and over again. No matter what ControlNet or settings I use, it refuses to actually follow the guide frames at all.
Just a heads up for anyone using mediapipe controlnet with this. Make sure you set max faces to 3. If you don't you, will find the results of the animation to be a mess because it will only produce a face map for one of the images its using in the process
Do you know any edits to the script that can be done to upload more guide frames (e.g. 2000 frames). If i upload too many guideframes, the script doesn't generate anything. I can only upload approx 80 frames
when you upload something to gradio, it creates a temporary copy of the file and works with that, see https://www.gradio.app/docs/file#behavior so maybe it's just lack of disk space on your system partition?
However, I never tried the script with that many frames. And I also wouldn't recommend it because of color degradation that WILL happen over time.
It takes a very, very long time to render with this script. If the usual generation with ControlNet takes 1 minute, then with the script, the first frame is 2 minutes, the second frame is 5 minutes, the 3rd frame is 25 minutes already, and so on. I can't wait for it to finish (res is 576/1024, 1controlnet)
Hello there! I got some issues with this tool First time when i used it - works perfectly, generate what i lookin for and all that. But after that, it starts to generate first pic over and over again, no matter what settings i use, no matter how much pic i selected as animation, every time same results with first generated picture, only with little differences between them Did somebody deal with that issue?
Greetings and beofre anything thanks for that much appreciated script ! From what I've read in the comments, there is a limit to the amount of images we can feed at once... I've tried the script update tips as indicated by @sxy9699 but to no avail, I still get the same error when trying to upload longer sequences (above 1000 images).
Is there anyway to get around this limitation and be able to feed it 1000's of pictures, much like in Deforum ?
I am enjoying the quality of and coherence I get with your script and that technique but I'd like to be able to throw huge sequences at it...
Thanks for your time, patience and, hopefully, help !
I use the script to try to generate a video, the first image is fine but the second image takes forever to come out. Is there a way to solve the problems? thank you!
I have seen a number of folks use your script for great result, and I am excited to use it, however I am having some challenges. I found your script from a video by Geek At Play, and he uses different settings than you do. I have tried both and still don't have satisfactory results. When I follow your instructions, using the 0 denoise for the first frame and any of the loopback settings, I get a great first frame, but the second and subsequent frames are completely different. I imagine that it has something to do with controlnet, but I'm not sure what is wrong. Any thoughts would be very helpful.
First of all make sure you select "pixel perfect" in your ControlNet(s).
I use the A1111-fork by Vlad and in some cases I have to de- and reactivate ControlNets and LoRAs after sending my first frame from txt2img. Sometimes only a complete re-start of the GUI helps. I usually do a few test renders before I activate the video script in order to see if it's working.
You should set the initial denoise to 0 if you want to keep your 1st frame, otherwise it will be re-rendered.
The normal 'denoise' setting should be something between 0.9 and 1, tho. If you set it below that you will generate garbage and if you set this to 0 you get the same frame over and over again.
Sadly, I'm just not getting there. I have explored MANY settings! Although I have had varying and interesting results, I have yet to come close to anything I want. When I in fact use denoise like you suggest of .9 to 1.0 (full on baby) I get frames that look nothing like the original except in pose. I have tried all three loopbacks and the second frame is always dog poop. I have followed your instructions to the letter and I have also followed "Geek at Play's" tutorial and my results are very different than either. I am clearly missing something very basic, but I have no clue what it is.
I would think that what I am try to accomplish should be doable: I have 489 frames of a man playing violin, I need to make him into a Lycan (wolfman) playing violin. I can test with any frame in Image to Image and get the output that I desire, But when I run the script, I get all kinds of weirdness. The strangest was when it put glasses and a gas mask on him. Nothing in the prompt, nor the guide frames that would suggest that!
Is there anything that you can think of that may solve my issues, or should I just give up and go through the longer and more arduous process of doing this in Deforum with hybrid video.
Thanks again for developing a tool that is working well for so many!
Evidently I'm a glutton for punishment! I spent several hours that I just didn't have to spend trying to get this to work, mainly because my best alternative SD-CN-Animation, is broken and therefore not an option. What is happening every time, regardless of any settings I use, is that in the second generation generation, the subject is moved to the left edge of the frame. Maybe you can help me to understand this weirdness. . . .
oh yes, I had a "move to left by half the width" issue once or twice and I don't really know why that happens. I think it was caused by the checkpoint or a LoRA that I used. And did you take a look at the console output? It sometimes gives you a hint on what is maybe missing or not working correctly.
Getting "UnidentifiedImageError: cannot identify image file" error when I click the generate button. Guide frames are selected. I can see the temp image file was created but with 0kb in size. Thanks in advance!
Hey! I saw that Xanthius didn't reply to your comment for a month now and wanted to offer my expertise in this field, if you are still interested. I experiment a lot with AI stylization, so if you want to get in touch with me [aleksejfifa15@gmail.com] is my email adress.
You mentioned in this tutorial that you've done non faces using this technique, and I'm really interested in locking in a decent rotoscoping method using stable diffusion, hopefully one with only slight alterations/flickering. Right now I'm trying out the "loving vincent" style van gogh rotoscoping using this technique with some drone footage I took in switzerland, however I ran into several issues. When I follow these exact steps in the tutorial, it seems each frame is getting darker and darker and the lines eventually become super dark and take over the image, all of the brush work gets flattened out and it starts essentially glitching after like 10 frames.
So I tinkered around a bit and found that if I change the last setting (loopback source) and set it to "firstgen" instead of "previous frame", the issue is resolved, however it seems like its not really moving, but just kind of looping the first frame over and over again, no more darkening, it gets additive for about 10 frames and then pretty much just stays still and doesn't move with slight flickering.
My videos camera pans slightly, and that isn't being reflected when I render this. Basically I'm just trying to batch the process in a way where Stable diffusion is painting over ever frame of an image sequence, but using control net in a way where it locks in the style for subsequent frames in the sequence. Any thoughts or recommendations using your method?
← Return to tool
Comments
Log in with itch.io to leave a comment.
How to write the request json when I call the img2img interface of the sd by Multi-frame rendering script, my original request json was as follows:
payload = {
"init_images": [image_base64],
# "prompt": prompt,
"prompt": prompts[i],
"negative_prompt": negative_prompt,
"styles": [],
"seed": 2694589603,
"subseed": -1,
"subseed_strength": 0,
"seed_resize_from_h": height,
"seed_resize_from_w": width,
"sampler_index": "Euler a",
"sampler_name": "Euler a",
"batch_size": 1,
"n_iter": 1,
"steps": 30,
"clip_skip": 1,
"cfg_scale": 8,
"width": width,
"height": height,
"restore_faces": True,
"tiling": False,
"do_not_save_samples": False,
"do_not_save_grid": False,
"eta": 0,
"denoising_strength": 0.4,
"s_min_uncond": 0,
"s_churn": 0,
"s_tmax": 0,
"s_tmin": 0,
"s_noise": 0,
#"override_settings": {
# "sd_model_checkpoint": "chosenMix_chosenMix.ckpt [dd0aacadb6]",
# "sd_vae": "pastel-waifu-diffusion.vae.pt"
#},
"override_settings_restore_afterwards": True,
"refiner_checkpoint": "",
"refiner_switch_at": 0,
"disable_extra_networks": False,
"comments": {},
"resize_mode": 0,
"image_cfg_scale": 1.5,
"mask": None,
"mask_blur_x": 4,
"mask_blur_y": 4,
"mask_blur": 0,
"inpainting_fill": 0,
"inpaint_full_res": True,
"inpaint_full_res_padding": 1,
"inpainting_mask_invert": 1,
"initial_noise_multiplier": 1.0,
"latent_mask": None,
"include_init_images": False,
"script_name": "",
"script_args": [],
"send_images": True,
"save_images": False,
"alwayson_scripts": {
"ControlNet": {
"args": [
{
"input_image": image_base64,
"enabled": True,
"lowvram": False,
"pixel_perfect": True,
"mask": None,
"module": "canny",
"model": "control_canny-fp16 [e3fe7712]",
"weight": 2,
"guidance_start": 0,
"guidance_end": 1,
"threshold_a": 100,
"threshold_b": 200,
"control_mode": 2,
"resize_mode": 1,
"processor_res": width
}
]
},
"face editor ex": {
"args": [
{
"prompt_for_face": "best quality, highly detailed, extremely detailed, shiny skin, shiny clothes, perfect face, beautiful face, white face, anime, cute, manga, green eyes, red lips",
"affected_areas": ["Face", "Hair"],
"use_minimal_area": True,
"save_original_image": True,
"face_size": width,
"upscaler": "R-ESRGAN 4x+"
}
]
}
}
}
Struggling similarly to others, while it seems to be batch outputting photos, it's really just a slight redrawing of the same frame over and over again. No matter what ControlNet or settings I use, it refuses to actually follow the guide frames at all.
Just a heads up for anyone using mediapipe controlnet with this. Make sure you set max faces to 3. If you don't you, will find the results of the animation to be a mess because it will only produce a face map for one of the images its using in the process
Do you know any edits to the script that can be done to upload more guide frames (e.g. 2000 frames). If i upload too many guideframes, the script doesn't generate anything. I can only upload approx 80 frames
when you upload something to gradio, it creates a temporary copy of the file and works with that, see https://www.gradio.app/docs/file#behavior so maybe it's just lack of disk space on your system partition?
However, I never tried the script with that many frames. And I also wouldn't recommend it because of color degradation that WILL happen over time.
It takes a very, very long time to render with this script. If the usual generation with ControlNet takes 1 minute, then with the script, the first frame is 2 minutes, the second frame is 5 minutes, the 3rd frame is 25 minutes already, and so on. I can't wait for it to finish (res is 576/1024, 1controlnet)
ps 3060 12gb
Hey Xanthius! I did some changes to the script that I want to share with the community - are you okay with that?
find my modified script here:
https://github.com/finefin/SD-scripts
Thank you!
will test it!
Error loading script: multi_frame_render-beta-fine-version.py
File "D:\SDiff\webui\scripts\multi_frame_render-beta-fine-version.py", line 91
<title>SD-scripts/multi_frame_render-beta-fine-version.py at main · finefin/SD-scripts · GitHub</title>
^
SyntaxError: invalid character '·' (U+00B7)
How to install it properly?
Error loading script: multi_frame_render-beta-fine-version.py
Traceback (most recent call last):
File "D:\SDiff\webui\modules\scripts.py", line 257, in load_scripts
script_module = script_loading.load_module(scriptfile.path)
File "D:\SDiff\webui\modules\script_loading.py", line 11, in load_module
module_spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 879, in exec_module
File "<frozen importlib._bootstrap_external>", line 1017, in get_code
File "<frozen importlib._bootstrap_external>", line 947, in source_to_code
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "D:\SDiff\webui\scripts\multi_frame_render-beta-fine-version.py", line 7
<!DOCTYPE html>
^
SyntaxError: invalid syntax
You downloaded the wrong thing. You need to get the raw file: https://raw.githubusercontent.com/finefin/SD-scripts/main/multi_frame_render-bet...
Thx Bro, i re download it, and it helps.
But when i'm starting generate with script nothing happens
(AttributeError: module 'modules.shared' has no attribute 'log')
What i,m doing wrong now?;))
I did not change the log output at all. Maybe you need to update your SD install? I use SD.next by Vladmandic and don't have any problems.
You could try to comment out that function call on line 69: put a "//" in front of the line like this:
// shared.log.info (....
If you do that, you wont see any progress in the UI, tho.
Hello there! I got some issues with this tool
First time when i used it - works perfectly, generate what i lookin for and all that.
But after that, it starts to generate first pic over and over again, no matter what settings i use, no matter how much pic i selected as animation, every time same results with first generated picture, only with little differences between them
Did somebody deal with that issue?
I think i fixed that by choose "Balanced" in Control mode in ControlNet. Previously ControlNet was more important
Greetings and beofre anything thanks for that much appreciated script !
From what I've read in the comments, there is a limit to the amount of images we can feed at once... I've tried the script update tips as indicated by @sxy9699 but to no avail, I still get the same error when trying to upload longer sequences (above 1000 images).
Is there anyway to get around this limitation and be able to feed it 1000's of pictures, much like in Deforum ?
I am enjoying the quality of and coherence I get with your script and that technique but I'd like to be able to throw huge sequences at it...
Thanks for your time, patience and, hopefully, help !
I use the script to try to generate a video, the first image is fine but the second image takes forever to come out. Is there a way to solve the problems? thank you!
there are many reasons why this can happen - take a look at the console output to narrow them down ;)
looks like you're running out of memory. you can try to lower the resolution, use less ControlNets and enable low vram mode.
I have seen a number of folks use your script for great result, and I am excited to use it, however I am having some challenges. I found your script from a video by Geek At Play, and he uses different settings than you do. I have tried both and still don't have satisfactory results. When I follow your instructions, using the 0 denoise for the first frame and any of the loopback settings, I get a great first frame, but the second and subsequent frames are completely different. I imagine that it has something to do with controlnet, but I'm not sure what is wrong. Any thoughts would be very helpful.
First of all make sure you select "pixel perfect" in your ControlNet(s).
I use the A1111-fork by Vlad and in some cases I have to de- and reactivate ControlNets and LoRAs after sending my first frame from txt2img. Sometimes only a complete re-start of the GUI helps. I usually do a few test renders before I activate the video script in order to see if it's working.
You should set the initial denoise to 0 if you want to keep your 1st frame, otherwise it will be re-rendered.
The normal 'denoise' setting should be something between 0.9 and 1, tho. If you set it below that you will generate garbage and if you set this to 0 you get the same frame over and over again.
Sadly, I'm just not getting there. I have explored MANY settings! Although I have had varying and interesting results, I have yet to come close to anything I want. When I in fact use denoise like you suggest of .9 to 1.0 (full on baby) I get frames that look nothing like the original except in pose. I have tried all three loopbacks and the second frame is always dog poop. I have followed your instructions to the letter and I have also followed "Geek at Play's" tutorial and my results are very different than either. I am clearly missing something very basic, but I have no clue what it is.
I would think that what I am try to accomplish should be doable: I have 489 frames of a man playing violin, I need to make him into a Lycan (wolfman) playing violin. I can test with any frame in Image to Image and get the output that I desire, But when I run the script, I get all kinds of weirdness. The strangest was when it put glasses and a gas mask on him. Nothing in the prompt, nor the guide frames that would suggest that!
Is there anything that you can think of that may solve my issues, or should I just give up and go through the longer and more arduous process of doing this in Deforum with hybrid video.
Thanks again for developing a tool that is working well for so many!
Evidently I'm a glutton for punishment! I spent several hours that I just didn't have to spend trying to get this to work, mainly because my best alternative SD-CN-Animation, is broken and therefore not an option. What is happening every time, regardless of any settings I use, is that in the second generation generation, the subject is moved to the left edge of the frame. Maybe you can help me to understand this weirdness. . . .
oh yes, I had a "move to left by half the width" issue once or twice and I don't really know why that happens. I think it was caused by the checkpoint or a LoRA that I used. And did you take a look at the console output? It sometimes gives you a hint on what is maybe missing or not working correctly.
Your script is the best for SD at the moment, thank you very much!
same problem, can you do so we can just add FOLDER name for UPLOAD? why upload 5000 files, ?? please ?
Getting "UnidentifiedImageError: cannot identify image file" error when I click the generate button. Guide frames are selected. I can see the temp image file was created but with 0kb in size. Thanks in advance!
you can fix this error by editing the python script as described here: https://itch.io/post/7576730
thanks, but not working, I find out , that i can drop images. but that is also not working , I am using google drive colab
please can you add url insted like a code? so I can add the dir of the google drive and dont upload?
oh, you're trying to run in on Collab? sorry, I can't help you with that, I only run this script locally.
@Xanthius. Good work on the script. I am working on a new AAA film and I have got a job for you. How do I get in touch?
Hey! I saw that Xanthius didn't reply to your comment for a month now and wanted to offer my expertise in this field, if you are still interested. I experiment a lot with AI stylization, so if you want to get in touch with me [aleksejfifa15@gmail.com] is my email adress.
here's another short animation that I made :)
Warning: loud noise! turn down audio volume!
You mentioned in this tutorial that you've done non faces using this technique, and I'm really interested in locking in a decent rotoscoping method using stable diffusion, hopefully one with only slight alterations/flickering. Right now I'm trying out the "loving vincent" style van gogh rotoscoping using this technique with some drone footage I took in switzerland, however I ran into several issues. When I follow these exact steps in the tutorial, it seems each frame is getting darker and darker and the lines eventually become super dark and take over the image, all of the brush work gets flattened out and it starts essentially glitching after like 10 frames.
So I tinkered around a bit and found that if I change the last setting (loopback source) and set it to "firstgen" instead of "previous frame", the issue is resolved, however it seems like its not really moving, but just kind of looping the first frame over and over again, no more darkening, it gets additive for about 10 frames and then pretty much just stays still and doesn't move with slight flickering.
My videos camera pans slightly, and that isn't being reflected when I render this. Basically I'm just trying to batch the process in a way where Stable diffusion is painting over ever frame of an image sequence, but using control net in a way where it locks in the style for subsequent frames in the sequence. Any thoughts or recommendations using your method?