Tag: lora

LCM and multiple versions of LoRas on the AI Horde

Db01 Comment

Finally 2024 is here and this allowed me a bit of free time to work on some of my NLNet tasks. The first thing on my list to tackle was adding LCM support on the AI Horde as it provides massively reduced steps, which for a crowdsourced service like ours, it makes all the difference in how much we can deliver.

For those who don’t know, LCM is a new breakthrough in Stable Diffusion that allows to “finetune” the model in such a way where an image can be generated using 10% of the steps previously required. So an image which would require 30 steps to converge, now needs just 3! That is a massive boost for lower-range GPUs. For high range GPUs, it starts avenues such as video generation as an image can happen at millisecond speeds!

Given the benefits, I wanted to work on this as soon as possible, and given the flexibility of the FOSS GenAI technology enthusiasts, we already had a great way to use LCMs, by using LoRas to turn any SD model into an LCM version.

However there was a snag. You see while the AI Horde already supports all LoRas on CivitAI, we never supported different versions of each, as we never expected anyone would want to use more than the latest. Unfortunately people on CivitAI started using the versioning system as “alternative” versions. And the LCM LoRa was using the same approach, where there was a version for each different sampler.

So the first order of business had to be to allow the AI horde to understand and support all LoRa versions of each LoRa! This took the better part of a full work-week of development and debugging, and then another week of troubleshooting and fixing in beta.

The good news is that this lead to us also identifying and squashing a very frustrating long-running bug where workers would rarely return previous images they’d generated instead of the ones requested. Getting someone else’s image is something we definitely don’t want to ever happen so we’re very happy we figured it out.

With that out of the way, I simply had to update the AI Horde itself to be able to handle the payload for specific LoRa versions, and then add support for the LCM sampler and then some ways to urge users to switch to it.

If you’re an AI Horde integrator, we strongly suggest you change your default settings to utilize LCM LoRas in your generations. You can get them from the same API you receive the model details, under the modelVersions key. To use them, you need to send the exact version ID as a string (found in modelVersions[#]['id]) this won’t accept a version name. You will also need to set is_version: true for the LoRa payload. This will tell the worker to look for a version instead of a LoRa ID.

Sending the LoRa name or ID will continue working as usual, grabbing the latest version (modelVersions[0]) from that list, so you existing implementations should continue working as usual.

Also we recently added AlbedoXL in our model list, to provide a better baseline for SDXL generations than basic SDXL 1.0 which requires a refiner to work. Using Albedo you can get generations that do not require a refiner in your workflow at all and get much less “fuzzy” generations in the process!

Magnagothica and my first LoRa!

Db0

A week ago or so, a new game came out from the creator of Kill Six Billion Demons (one of my most favourite webcomics that you should all go to read right now): Magnagothica: Maleghast

It’s a great world-building and incidentally it’s an idea for a type of setting that I have been thinking to make myself for years, and now someone plucked it straight out of my mind and made a really great version of it which just oozes style!

I have been enamored with this game since I first saw it. It hits ALL my buttons! Heavy Metal, Fantasy, Aesthetics, Boardgames. Only thing it could have done more is include even more metal genres as I’m not too keen on the extreme stuff and I’d love to see styles on Power or Prog Metal etc. But nothing we can’t extend ourselves!

But ye, as I mentioned, I’ve been hyperfocusing on it a bit too much. First I created a lemmy community for it. Then I started extending the music playlists for each house. And lastly I wanted to find a way to be able to not only create my own warband, but also to give individual art to each of my units. And for this, I finally had some necessary skills and hardware. It was time to train my first LoRa!

It took me a good 3 days to figure out how things work, create the necessary datasets, experiment with creation and retry until I got the hang of it. It didn’t help that the original artwork was, let’s say, challenging to work with. But I think in the end I am very proud to have managed to make something which seems to be capable of following the style well enough to create leaders and units.

I’ve found the best use of this lora is for Leader illustrations. To create a leader simply use the “necromancer, leader” tokens in your prompt. For leaders I suggest you set your Lora strength between 0.5 and 0.8. Remember to mention which house you want the leader to be in. The LoRa is pretty decent at copying the style of each house.

magnagothica maleghast, gargamox, drawing, necromancer, male, skinny leader, plain background, toxic green highlights, by tom bloom, sidecut haircut, chains, dripping, white coat###halftone

To generate units, you can choose to either generate unit portraits styles, or full unit illustrations. You can try to replicate the maleghast style with the halftone, or put “halftone” in the negative prompt to get a more sharp result. Personally I prefer to avoid the halftone look and try to make them look more full. You can also use the name of the unit to try to lead the lora to draw something like that unit. You can also use “ghoul” for humanoid undead and “abomination” for more monstrous looking units. For units I suggest you use model strength between 0.3 and 0.6

magnagothica maleghast, abhorrers drawing, full unit illustration, penitent, ghoul, scarred, spiky flail, plain white background, ochre highlights, by tom bloom###halftone
magnagothica maleghast, C.A.R.C.A.S.S., drawing, unit portrait, enforcer, ghoul, plain black background, fuchsia highlights, by tom bloom###halftone

You can check out the new LoRa on CivitAI, and as you’d expect you can also just try it out directly on the AI Horde!

Try it out and let me know what other uses you can find using this style. I think it has plenty of potential.

The AI Horde now seamlessly provides all CivitAI LoRas

Db0

Demand for AI Horde to support LoRas was high ever since they were discovered and unleashed into the Stable Diffusion ecosystem. About 2 months ago we were really close to releasing support for them but alas “stuff happened” which culminated in us having to completely rewrite out inference backend.

Fortunately, ComfyUI already supported LoRas, so once we switched to using it as our inference library, we just needed to hook into its LoRa capabilities. Myself along with the extremely skilled hordelib devs, Jug and Tazlin have been working constantly on this for the past half a month or so and today I’m proud to announce that the Lora capability has been released to all AI Horde Workers!

You might have seen earlier that some Client like Artbot, the Krita plugin and Lucid Creations were already supported LoRas. This was however only using the few beta testing workers we had. Now those clients will start seeing a lot more speed as worker with the power to do so start running them.

This also signifies our tightest integration with CivitAI yet! We we were already utilizing them, but mostly to just pull stable diffusion checkpoint files. The LoRa integration however goes far far further than that, using them as an integral part of our discovery and control process.

Through the very useful REST API provided by CivitAI, we have developed a method where we can discover, download and utilize any LoRa that is hosted there. No more do you need to search, validate compatibility, download etc. The AI Horde and its Worker will handle that automatically!

To showcase this point, I’ve made a small video showing off this capability in Lucid Creations which I’ve tightly integrated into CivitAI, so that it can look for any LoRa based on ID or partial name, display all relevant information and even warns you of compatibility issues.

Sorry about the sound quality. I don’t have a professional streamer setup, just a shitty webcam 😀

I’ve put a lot of effort into making finding and using LoRas as painless as possible, because there’s literally dozens of thousands of the things. Manually searching through the website and copy-pasting IDs is just a complete PITA. Never mind downloading and trying them out one-by-one.

Through the AI Horde, you can simply type the partial name of what you want, and keep mix and matching them until you achieve the desired outcome! Changing and trying new LoRas now takes literally a few seconds!

Also many kudos to the CivitAI devs which quickly implemented one request I put through to make this even easier and faster for everyone.

As this has just been released, currently only a few UIs support this. Lucid Creations which has full discovery, and ArtBot which requires you to know the ID. The Krita plugin is also supporting them. I expect the other UIs to soon follow in the coming days.

Hopefully this is another step in unleashing your creativity, without requiring a powerful GPU of your own!

Stay tuned for the next update adding access to all Textual Inversions as well.

Lucid Creations is the first client to support LoRa on the AI Horde!

Db01 Comment

I don’t have time to keep Lucid Creations well updated but sometimes I just need to show others how it’s done!

So I managed to the first UI to support LoRa on the AI Horde!

In case you don’t know what LoRa are, the short version is nothing less than a breakthrough in Generative AI technology, by allowing to condense the time and power a training (or “fine-tuning”) needs!

This kind of breakthrough, along with the achievements done by new models such Stable Diffusion and Llama is what is causing the big tech companies to scramble, to the point that OpenAI had to run to the government to ask them to regulate future breakthroughs (but please, please don’t regulate what they’re currently doing, OK?)

But I digress. I haven’t yet officially announced LoRa support on the AI Horde yet, as we’re still trying to squash all bugs on the workers, but I hope the addition to Lucid Creations will help other integrators see how it can be done