DMP plate rework with GenAI

By -
I have been reworking plates with GenAI to complete DMP tasks such as season changes and time of day changes.



---

Process:

I start with my plate, making sure it's cleaned up and ready to work with. My plate will be the starting raster layer in InvokeAI:


I would like to change the overall overcast lighting to a brighter sky and warm sunlight with long cast shadows.

I am using an image to image approach, together with ControlNets and IP adapters, which allows me to preserve the plate as much as possible while controlling the changes I want to make to the plate. 

For example, using a canny edge detection and depth map will retain the structure.


After some tweaking and finding a good balance between the denoising strength value, the weight of each ControlNet and my prompt, I was happy with my result:




A season change to autumn:





A season change to winter:


Quick note that the style referencing takes into account the mood and texture of the image reference. You can add more than one reference and fine tune the weight of each one to get it closer to what you want.




My final summer to winter result using Invoke AI:




Day to night:


For this night matte painting, I had a specific look I was going for, so I generated a reference image with Leonardo AI that I could then use to get to my result faster:



I also wanted to change certain regions only, such as the sky and bounce light on the trees.



For the trees, I first upscaled my result image, and then placed the bounding box at around 1k resolution around the area where I wanted the bounce light for a more detailed result. I painted a regional guidance mask and had a reference image for the tree bounce light specifically.



I can then control how much of the layer I want by adjusting its opacity. 

With InvokeAI's node-based workflow editor, I can always tweak individual elements in an undestructive manner.




Final day to night re-styling while keeping the original plate intact:



I wanted to take the same plate and do a season change on it to winter.
I generated the exact references I wanted with Leonardo AI again:



InvokeAI has a neat tool 'Select Object' which allows you to isolate an element as a mask for inpainting.
In this case I separated the castle as an object and tweaked it on it's own for the snow. 


I find this very useful, and you can also invert the selection or save it out as a mask.
I also got a mask for the lake as I wanted to have more control over it.




Final winter DMP:


Comments

Post a Comment

Popular posts from this blog

Creating Styleframes with LeonardoAI and InvokeAI

By -
Image
I decided to take screenshots of one of my favourite animated movies, The Road to El Dorado, and create some live action styleframes out of them. --- Process: To get an initial image out, I'm using Leonardo AI. It's workflow helps me quickly establish the style and composition I'm looking for. In this case, I have a screenshot to match to: Quick definition before continuing: Control Nets These are AI tools that guide image generation by providing additional structural input, such as sketches, depth maps, or edge detection. They help maintain consistency in composition, pose, or perspective while still allowing for creative freedom in generating an output image. Since I want to match my image as closely as possible to the screenshot, I'm using a combination of ControlNets with the screenshot as my input image. The Edge to image ensures my  generated images follow the specific structure. The Depth to image acts as my depth map. Image Prompt Adapter IP Adapters  extract sp...
Read more

ComfyUI IC-Light relighting exploration

By -
Image
I've started using  ComfyUI  to gain more control over my workflow.  Its  node-based system  is similar to InvokeAI. I'm exploring IC-Light (image relighting) and it's a great tool in visual development for composition, lighting, mood and style. Since it runs on an SD1.5 model, it's not quite ready for a final DMP output yet, but I see lots of potential for visual development. For example, it is quick at giving you a location with different lighting scenarios. I was also able to exactly control the direction of the lighting, the colour and intensity. Using my own photos, I'm able to quickly create multiple different moods for art direction. For my workflow, I mixed different colour gradients with my image, controlling the light direction and tone of the scene. Original photo: My IC-Light generations: Original photo: My IC-Light generations: Example of light direction control with a spline: Adding some colour to the gradient: Output example: Original photo: My IC...
Read more