I’ve been meaning to write one of these posts every month, but the events since I wrote the last piece have been fairly disruptive. With the loss of our maintainers of nataili, we’ve been forced to put our heads down to alleviate the backend issue, and I didn’t want to write another “State of the AI Horde” until that business was completed. More about that later. First, let’s look at the basics.
(In case you’re new here and do not know what is the AI Horde: It is a crowdsourced free/libre software Open API which connects Generative AI clients to volunteer workers which provide the inference.)
More Requests, More Power!
The total amount of images generated has stayed relatively stable since the last month, with only ~12M images generated. However the total amount of Terapixelsteps is up to 4.2TPS compared to 3.7TPS last month. This shows that people are looking for more details at higher resolution instead of just grabbing more images. Unfortunately we can’t capture the impact of ControlNet as easily, but suffice to say, its demand is pretty significant.
On the LLM front, we’ve generated 3.5M texts, for a total of 374 Megatokens. We have effectively tripled the LLM output! This makes sense as we see plenty of traction in the LLM communities since Google Colab started banning Pygmalion.
Also, did you know Cubox from our discord community has gone and setup a whole public Grafana server to track our stats? Head over and check it out yourselves.
Top 10 Stable Diffusion models
No significant changes since last month in this chart, with Deliberate still proving its worth and further solidifying its position with a 25% use rate throughout the entire month. Stable Diffusion 1.5 sees plenty of use too.
Anime models seem to be losing some popularity across the board, and Dreamshaper kicked Abyss OrangeMix off the board so as to settle on a decent 5%. Good showing!
- Deliberate 25.8% (3067246)
- stable_diffusion 14.4% (1707729)
- Anything Diffusion 9.0% (1067386)
- Dreamshaper 5.0% (596651)
- Realistic Vision 4.7% (563079)
- URPM 3.6% (428415)
- Hentai Diffusion 2.9% (348985)
- ChilloutMix 2.6% (314301)
- Project Unreal Engine 5 2.6% (307907)
- Counterfeit 2.2% (260739)
Top 10 Text models
As is to be expected, Pygmalion 6b is still leading the charts with a significant margin, but its big brother, the recently released Pygmalion 7b, which is based on the Llama models, has already secured 3rd place and is only set to further cannibalize 3b’s position.
Erebus and Nerybus continue to fill up the rest of the board with a few new 4bit models starting to finally come into the fray.
- PygmalionAI/pygmalion-6b 44.5% (1571109)
- KoboldAI/OPT-13B-Erebus 9.8% (345983)
- PygmalionAI/pygmalion-7b 5.2% (183413)
- KoboldAI/OPT-2.7B-Nerybus-Mix 3.8% (135223)
- KoboldAI/OPT-13B-Nerybus-Mix 3.8% (132564)
- bittensor/subnet1 2.5% (86859)
- Pygmalion-7b-4bit-32g-GPTQ-Safetensors 2.3% (79577)
- gpt4-x-alpaca-13b-native-4bit-128g 2.0% (72358)
- pygmalion-6b-gptq-4bit 1.8% (64750)
- OPT-6.7B-Nerybus-Mix 1.8% (62300)
Image Ratings keep flowing in
Adding new ratings continue unabated but we’re still finding people trying to bypass our countermeasures and just automatically rate images to poison the dataset. Please don’t do that. If you want kudos, you can easily get more by simply asking in our discord instead of wasting our time 🙁
One interesting note is that the Stable UI has finally bypassed the Artbot on amount of images rated
count | client
--------+----------------------
6270 | AAAI UI
110826 | ArtBot
279112 | Stable UI
3197 | Unknown
2700 | ZeldaFan Discord BotA total of 400K ratings over the past month. Very impressive.
I have also finally on-boarded the full DiffusionDB dataset of 12M images, into the ratings DB, we we’ll have enough images to rate for the foreseeable future.
A whole new backend
As I mentioned at the start, the big reason this state of the AI horde was delayed, was the need to switch to a completely new backend. It’s a big story, so if you want to read up on some details on that, do check out the devlog about it.
New Features and tweaks
Not much to announce since all our effort was in the backend switch, but a few things nevertheless
- We have added support for Shared Keys, so that you can share your account priority, without risk.
- A new kudos calculation model has been put live, instead of the manual way I was calculating kudos for each request based on magic numbers. The new kudos model is a Neural Network trained by Jug, and it takes into account the whole payload and figures out the time it would take to generate empirically.
- Kudos consumption has been removed for weights, and a tiny kudos tax of 1 kudos has been added per request to discourage splitting the same prompt into multiple separate requests.
- You can now change your worker whitelist into a worker blacklist so that you avoid specific workers, instead of requesting explicitly which ones you want.
- A new “Customizer” role has been added to allow you to host custom Stable Diffusion models. This is not possible on the Worker yet, but once it is, these people will be able to do it. Getting this role is a ping to the AI Horde moderators.
- We had to fight back another raid, on the LLM-side this time, which forced me to implement some more countermeasures. Scribes (i.e. LLM workers) cannot connect through VPN unless trusted, untrusted users can only onboard 3 workers, and there’s an worker-limit per IP now.
Discord Updates
I’ve been doing some work to improve the social aspects in our discord server, one of them is onboarding 3 new volunteer community managers through and hoping we can start doing more events and interactions with the community.
Another is the addition of a new channel per Integration I know of. I give the developers/maintainers of these services admin access to that channel so they can use it to provide support to their users, or redirect to their own discord servers if they prefer.
If you have a service integrating into the AI Horde API, please do let me know and I’ll be happy to open a new channel for you and give you admin access to it.
Infrastructure Improvements
Our demand is increasing significantly, as we have way more and more concurrent web sessions every day. Unfortunately a few database downtimes this month convinced me that hosting it in a Contabo VM with 30% CPU steal is not feasible anymore.
So I finally finalized switching the Database to a dedicated server. It’s a lot more expensive but massively worth it as our response times have increased 5-fold! A worker payload pop that used to take 1-3 seconds, now can take 0.2 – 1 second! As a result the whole AI horde should feel way snappier now.
I have also added our first dedicated box API front-end. Initially it didn’t look to be doing too good, having some of the worst performance, but once I switched to the dedicated database server, it suddenly became the fastest by far, making very obvious just how much impact latency has.
I have also finally deployed a central log server based on Graylog, which should help me track issues across all servers and look them up historically as well.
Funding and Support
All of the above now means the AI horde is significantly more expensive to run and it’s almost a second full-time job for me at this point. Currently all of this is funded via my patreon subscribers only but that is not scaling quite as fast as my infrastructure costs :-/
To add to this, the stability.ai support seems to have run dry and the GPUs they were providing to the AI Horde have been removed and I haven’t been able to arrange to bring them up again.
So I think I need to more consistently promote the way to help me sustain the AI Horde as a truly free and open API for everyone.
So, please if you see the potential of this service, consider subscribing as my patron where I reward people with monthly priority for their support.
If you prefer, you can instead sponsor me on github.
I am also looking for business sponsors who either see the value of a Crowdsourced Generative AI Inference REST API or might want to use it for themselves. I basically have no contacts in the AI world so I would appreciate people forwarding this info to whoever might be interested.
Final Word
The last month has been very difficult for the AI Horde but fortunately I’ve had the help of some really valuable contributors from both the AI Horde community as well as the KoboldAI community and we managed to pull through.
Now with our new backend I expect new features to come much faster and of more quality. We’re already working hard on LoRa support horde-wide for one, so stay tuned!