Those who’ve followed my progress in developing Hypnagonia would know by now that I’ve been working on the integration of KoboldAI with Godot for the automatic generation of stories. This solution while working never quite satisfied me, as it required quite a lot of investment on behalf of a player who wanted to generate stories.
That approach left me unsatisfied. As it happened I had an idea to use the new KoboldAI API to create a distributed cluster based on consumer GPUs in people’s PCs. The concept being that one would connect their PC in some fashion to a server software and whenever someone sent a request for generation, an available PC would pick it up, generate it, and send it back to the requester.
The requester themselves would therefore not need a powerful GPU (or skill to use Google Colab) to use KoboldAI, while people who’s PC is otherwise sitting idle, would be supporting the other members of the community.
So after some discussions with members of the community, I decided this idea had legs and I wanted to build it before I continued with Hypnagonia. I’ve been working non-stop on this for the past week and I now feel it’s at a pretty good state.
Introducing the KoboldAI Horde!
This is a python server which you run on a server somewhere and it provides an interface with which people can request GPT writing generations. The second part is the bridge, which is what people who have their own KAI instances run, in order to connect the KAI server to the server. The last piece of software of course is the KoboldAI client which runs the models generating the text.
Until now, the only option for people without such powerful hardware would typically to use a service like NoveAI or AI Dungeon, where one has to pay for the service. They also provided a fancy interface and tools to help people write their individual stories. Aside from the cost however, there were also limitations. Dungeon AI recently imploded due to their own actions, and a lot of people were left without a home.
And of course, for a free software video game, such services are prohibitive, as I can’t pay out of pocket for the text generations of dozens of players (or imagine if my game ever became actually popular).
After a bit of hacking of the KoboldAI client itself, I also added functionality to allow someone to use their own KoboldAI client to talk to the Horde directly. Which means someone with a potato PC, can still use KoboldAI directly.
Now there are some obvious limitations as you might expect. First of all, the availability of the models in the horde is dependent on who is supporting it at the moment. If only people with weak GPU are in, the best you’ll find is a Nerys 2.7B. If however some of the big chonkers are in, you might be able to use KoboldAI as if you have a 20B model behind it.
The other problem is of course questions of privacy. Even though the text is encrypted during transfer, and even though the horde does not store any prompts or generations, at the end destination, the owner of the server can see all your prompts if they so wish, even if they don’t know who you are. So for those wanting the really steamy stuff, or perhaps those who write stories using real people and locations, this might be a deal breaker.
However for more normal stories, or for more generic generations such as for utilities and apps, this is hardly a concern.
Finally we’re still new, so currently everyone is playing nice. But there’s always the potential of bad actors, who might food the horde with garbage generations just to shut it down. There’s nothing I can do about someone dedicated to do that except go into a whitelist mode, and have the horde become an invite-only service, much like a private bittorrent tracker. In fact, anyone could set up their own private instance for their own community, for those situations where they want only trusted people to use it.
Eventually my plan is to follow the bittorrent approach for usage and contributions as well, where people will only be able to use the horde if they contribute somehow to the KAI community. Whether by adding their own GPU to the horde, or by writing documentation, etc.
But ultimately I want this to be a mutual aid based service. Where those who have the capacity help those who do not, and the latter find some other ways to make it up, for the uplifting of everyone in the KoboldAI community.
And before anyone suggests to tie this to some web3 token: Fuck that shit! I will work integrate with any cryptocurrencies! That is poison for our communities AND for our ecosystem.
The owner of the KAI client repo was also kind enough to redirect their domain to my KAI Horde instance, so now we have a very fancy url as well.
Already this is working amazingly well and I’m working on improvements daily. I’m excited to see what kind of doors this will open.
