State of the AI Horde – May 2023

I’ve been meaning to write one of these posts every month, but the events since I wrote the last piece have been fairly disruptive. With the loss of our maintainers of nataili, we’ve been forced to put our heads down to alleviate the backend issue, and I didn’t want to write another “State of the AI Horde” until that business was completed. More about that later. First, let’s look at the basics.

(In case you’re new here and do not know what is the AI Horde: It is a crowdsourced free/libre software Open API which connects Generative AI clients to volunteer workers which provide the inference.)

More Requests, More Power!

The total amount of images generated has stayed relatively stable since the last month, with only ~12M images generated. However the total amount of Terapixelsteps is up to 4.2TPS compared to 3.7TPS last month. This shows that people are looking for more details at higher resolution instead of just grabbing more images. Unfortunately we can’t capture the impact of ControlNet as easily, but suffice to say, its demand is pretty significant.

On the LLM front, we’ve generated 3.5M texts, for a total of 374 Megatokens. We have effectively tripled the LLM output! This makes sense as we see plenty of traction in the LLM communities since Google Colab started banning Pygmalion.

Also, did you know Cubox from our discord community has gone and setup a whole public Grafana server to track our stats? Head over and check it out yourselves.

Top 10 Stable Diffusion models

No significant changes since last month in this chart, with Deliberate still proving its worth and further solidifying its position with a 25% use rate throughout the entire month. Stable Diffusion 1.5 sees plenty of use too.

Anime models seem to be losing some popularity across the board, and Dreamshaper kicked Abyss OrangeMix off the board so as to settle on a decent 5%. Good showing!

  1. Deliberate 25.8% (3067246)
  2. stable_diffusion 14.4% (1707729)
  3. Anything Diffusion 9.0% (1067386)
  4. Dreamshaper 5.0% (596651)
  5. Realistic Vision 4.7% (563079)
  6. URPM 3.6% (428415)
  7. Hentai Diffusion 2.9% (348985)
  8. ChilloutMix 2.6% (314301)
  9. Project Unreal Engine 5 2.6% (307907)
  10. Counterfeit 2.2% (260739)

Top 10 Text models

As is to be expected, Pygmalion 6b is still leading the charts with a significant margin, but its big brother, the recently released Pygmalion 7b, which is based on the Llama models, has already secured 3rd place and is only set to further cannibalize 3b’s position.

Erebus and Nerybus continue to fill up the rest of the board with a few new 4bit models starting to finally come into the fray.

  1. PygmalionAI/pygmalion-6b 44.5% (1571109)
  2. KoboldAI/OPT-13B-Erebus 9.8% (345983)
  3. PygmalionAI/pygmalion-7b 5.2% (183413)
  4. KoboldAI/OPT-2.7B-Nerybus-Mix 3.8% (135223)
  5. KoboldAI/OPT-13B-Nerybus-Mix 3.8% (132564)
  6. bittensor/subnet1 2.5% (86859)
  7. Pygmalion-7b-4bit-32g-GPTQ-Safetensors 2.3% (79577)
  8. gpt4-x-alpaca-13b-native-4bit-128g 2.0% (72358)
  9. pygmalion-6b-gptq-4bit 1.8% (64750)
  10. OPT-6.7B-Nerybus-Mix 1.8% (62300)

Image Ratings keep flowing in

Adding new ratings continue unabated but we’re still finding people trying to bypass our countermeasures and just automatically rate images to poison the dataset. Please don’t do that. If you want kudos, you can easily get more by simply asking in our discord instead of wasting our time 🙁

One interesting note is that the Stable UI has finally bypassed the Artbot on amount of images rated

 count  |        client        
--------+----------------------
   6270 | AAAI UI
 110826 | ArtBot
 279112 | Stable UI
   3197 | Unknown
   2700 | ZeldaFan Discord Bot

A total of 400K ratings over the past month. Very impressive.

I have also finally on-boarded the full DiffusionDB dataset of 12M images, into the ratings DB, we we’ll have enough images to rate for the foreseeable future.

A whole new backend

As I mentioned at the start, the big reason this state of the AI horde was delayed, was the need to switch to a completely new backend. It’s a big story, so if you want to read up on some details on that, do check out the devlog about it.

New Features and tweaks

Not much to announce since all our effort was in the backend switch, but a few things nevertheless

  • We have added support for Shared Keys, so that you can share your account priority, without risk.
  • A new kudos calculation model has been put live, instead of the manual way I was calculating kudos for each request based on magic numbers. The new kudos model is a Neural Network trained by Jug, and it takes into account the whole payload and figures out the time it would take to generate empirically.
  • Kudos consumption has been removed for weights, and a tiny kudos tax of 1 kudos has been added per request to discourage splitting the same prompt into multiple separate requests.
  • You can now change your worker whitelist into a worker blacklist so that you avoid specific workers, instead of requesting explicitly which ones you want.
  • A new “Customizer” role has been added to allow you to host custom Stable Diffusion models. This is not possible on the Worker yet, but once it is, these people will be able to do it. Getting this role is a ping to the AI Horde moderators.
  • We had to fight back another raid, on the LLM-side this time, which forced me to implement some more countermeasures. Scribes (i.e. LLM workers) cannot connect through VPN unless trusted, untrusted users can only onboard 3 workers, and there’s an worker-limit per IP now.

Discord Updates

I’ve been doing some work to improve the social aspects in our discord server, one of them is onboarding 3 new volunteer community managers through and hoping we can start doing more events and interactions with the community.

Another is the addition of a new channel per Integration I know of. I give the developers/maintainers of these services admin access to that channel so they can use it to provide support to their users, or redirect to their own discord servers if they prefer.

If you have a service integrating into the AI Horde API, please do let me know and I’ll be happy to open a new channel for you and give you admin access to it.

Infrastructure Improvements

Our demand is increasing significantly, as we have way more and more concurrent web sessions every day. Unfortunately a few database downtimes this month convinced me that hosting it in a Contabo VM with 30% CPU steal is not feasible anymore.

So I finally finalized switching the Database to a dedicated server. It’s a lot more expensive but massively worth it as our response times have increased 5-fold! A worker payload pop that used to take 1-3 seconds, now can take 0.2 – 1 second! As a result the whole AI horde should feel way snappier now.

I have also added our first dedicated box API front-end. Initially it didn’t look to be doing too good, having some of the worst performance, but once I switched to the dedicated database server, it suddenly became the fastest by far, making very obvious just how much impact latency has.

I have also finally deployed a central log server based on Graylog, which should help me track issues across all servers and look them up historically as well.

Funding and Support

All of the above now means the AI horde is significantly more expensive to run and it’s almost a second full-time job for me at this point. Currently all of this is funded via my patreon subscribers only but that is not scaling quite as fast as my infrastructure costs :-/

To add to this, the stability.ai support seems to have run dry and the GPUs they were providing to the AI Horde have been removed and I haven’t been able to arrange to bring them up again.

So I think I need to more consistently promote the way to help me sustain the AI Horde as a truly free and open API for everyone.

So, please if you see the potential of this service, consider subscribing as my patron where I reward people with monthly priority for their support.

If you prefer, you can instead sponsor me on github.

I am also looking for business sponsors who either see the value of a Crowdsourced Generative AI Inference REST API or might want to use it for themselves. I basically have no contacts in the AI world so I would appreciate people forwarding this info to whoever might be interested.

Final Word

The last month has been very difficult for the AI Horde but fortunately I’ve had the help of some really valuable contributors from both the AI Horde community as well as the KoboldAI community and we managed to pull through.

Now with our new backend I expect new features to come much faster and of more quality. We’re already working hard on LoRa support horde-wide for one, so stay tuned!

State of the AI Horde – 26/03/2023

Things are progressing very rapidly in this dawn of the AI and likewise for the AI Horde. I thought it would be a good idea to post about all the things that changed and improved in recent days for our service.

More Requests. More statistics.

I’ve deployed endpoints to measure the usage of the AI horde. Now that one month has passed, we can take a look.

  • Per day, we are averaging 356,378 images (3.7 terapixelsteps) and 45,248 texts (4 megatokens)
  • In the past month, we produced 11,475,183 images, generating a staggering 127.6 terapixelsteps. Text has also picked up significant speed since merging the hordes with 1,241,895 generated texts for a total of 112.8 megatokens!

Top 10 Stable Diffusion models

The AI Horde offers close to 200 models at the same time. Our statistics allows us to see how the popularity of the various models changes day to day and month to month. The below are just the top 10 models being used.

  • Deliberate 22.2% (2550591)
  • stable_diffusion 15.1% (1730426)
  • Anything Diffusion 11.0% (1257688)
  • Hentai Diffusion 4.1% (468473)
  • Realistic Vision 3.0% (338742)
  • Counterfeit 2.7% (310337)
  • URPM 2.6% (297853)
  • Project Unreal Engine 5 2.5% (289006)
  • waifu_diffusion 1.8% (211572)
  • Abyss OrangeMix 1.8% (205268)

For the longest time SD 1.5 (stable_diffusion above) was king, but in the past month, Deliberate has confidently taken the lead and has been leading the pack with a staggering 20% of all image requests passing through the AI Horde! This speaks very highly for the popularity of the model

Top 10 Text models

Almost as many text models exist for the AI Horde, but they’re more varied. However last months saw the release of two big milestones, the Pygmalion models for chat-like generation, which happened after the gimping of the Character AI models. The new Llama model was also released, bringing unparalleled miniaturization of the model size, allowing consumer GPUs far more coherence.

  1. PygmalionAI/pygmalion-6b 52.4% (651566)
  2. KoboldAI/OPT-13B-Erebus 14.0% (174393)
  3. KoboldAI/OPT-6.7B-Erebus 6.7% (83249)
  4. KoboldAI/OPT-6.7B-Nerybus-Mix 3.8% (46747)
  5. KoboldAI/OPT-13B-Nerybus-Mix 2.8% (35110)
  6. KoboldAI/OPT-13B-Nerys-v2 2.7% (33667)
  7. Facebook/LLaMA-13b 1.9% (23367)
  8. KoboldAI/OPT-6B-nerys-v2 1.9% (23232)
  9. OPT-6.7B-Nerybus-Mix 1.6% (19268)
  10. KoboldAI/OPT-2.7B-Erebus 1.0% (12464)

We can see Pygmalion has immediately dominated text generation, with Mr.Seeker’s storytelling models mopping up the rest, but the Llama Ascendancy is just beggining!

Ratings, botting and counter-measures

A few months ago we started collecting ratings for the LAION non-profit to help improve the models existing in the commons, as the success of midjourney has a lot to do with them training their models with the best images their previous generation created.

The initial design was very simple to allow integrators to onboard it fast and giving good kudos rewards for those helping us. Unfortunately people almost immediately started abusing this by creating bots to rate randomly, therefore poisoning our collection’s accuracy.

I always knew this was a possibility but I was hoping I wouldn’t be forced to add countermeasures quite so soon. So I spent quite a few days adding a captcha mechanism (along other things) to block at least the low hanging fruit.

It immediately led to a drop in ratings per day which automatically shows just how much damage botted ratings were doing

New Features

We are fortunate enough to have gathered some great collators for the inference aspect of the AI Horde. So I wanted a big shout-out.

  • ResidentChief has stepped up strongly to help add new features and squash bugs in the nataili library. As a result the AI Horde now supports inpainting on many more models, a lot more post-processors, such as more upscalers and background removers, controlnet improvements, and so many other stuff too numerous to mention. They’re a beast!
  • Jug has been working on improving the AI Horde worker practically non-stop. Giving us a great terminal control, and improving the webui. Plus a lot of bugfixes and improvements in the bridge part of things
  • Tazlin who’s been doing a great deal of tech support in the channels as well as helping me detect and figure out malicious ratings. And also sending some code improvements as well!
  • Aes Sedai who’s been putting a ton of work on improving the moderation capabilities of the AI Horde with a custom frontend.

And of course all the frontend integrators like rockbandit, aqualxx, sgt.chaos and concedo, who’ve been keeping the frontends up to date, with a lot of features smartly using the capabilities of the AI Horde in ways even I had not expected!

CI/CD and pypi

I finally got around to adding CI/CD pipelines for AI Horde Worker and nataili. Now they will be automatically versioned when the right tag is applied to a PR. The Nataili package has also been republished to pypi and will also automatically receive new versions whenever we publish a new release on GitHub.

The notifications also automatically publish a notification on discord, so people can be aware when something new is up.

Alchemists

Using the new post-processing improvements from ResidentChief, I’ve expanded the interrogation worker so that it can now perform post-processing on images, as well as img2text operations. Unfortunately the previous name didn’t fit so well, so now I’ve renamed it to “Alchemist”, to signify it’s capability to convert images to something else.

Likewise, the official names for image worker is now “Dreamer” and text worker is now “Scribe”. Why not 🙂

Final Word

The pace of progress in this space is mind-blowing. I can’t wait to see what we achieve together in the coming days!