With the recent updates to the Fediseer, it is now possible to use it more efficiently for programmatically adjusing one’s blocklist using the built-in censures, so I wanted to add this capability to the Divisions by zero by updating my update_blacklist.py script to utilize this method.
While I could have done this by using simple http requests, I thought it was enough features there that creating a python SDK package around the fediseer API would be worthwhile. So I’m excited to announce the release of pythonseer!
It is built very similar to Pythörhead, so if you’re already using that in your scripts, it should be trivial to add support for the Fediseer via pythonseer as well.
I have already updated the provided script to automatically maintain your blocklists to allow it to make use of the fediseer censures as well. Simply censure the domains you would like to defederate from, and the script will automatically add them to your list along with the suspicious instances. If you have the script to run on a schedule, you can then simply add new censures on the fediseer API and your blocklist will be automatically kept up to date.
Likewise, if you don’t want to maintain all the blocklist yourself, find other like-minded instances and combine their censures with yours. This way whenever they censure someone, you also benefit.
Enterprising fediverse admins could also do other fancy things with the pythonseer, such as creating polls to democratically add instances to their endorsements or censures etc. So do let me know if you come up with new ideas!
One can now request all endorsements from multiple domains:
This allows instances to quickly discover a “list of friends” based on other instances. Use cases for this might include scripts which auto-approve comments in moderation, or automatically update a fediverse instance’s whitelist based on common endorsements.
Censures
The current fediseer guarantees are meant to apply only to consideration of spam. As such we do not have a way to mark instance that many would consider terrible in all other ways except spam (e.g. pro-nazi instance, or an instance allowing loli content) as such.
To solve this a new feature has been added: Censures
An instance can now apply a censure to any other instance domain (whether it’s guaranteed already or not) for any reason. This extreme disapproval can come from any subjective reason, but like endorsements it doesn’t, by itself, have any mechanical impact.
In fact, because endorsements and censures are explicitly subjective, I have taken the decision to not display censure counts on instance details to avoid people using them for “objectively” rating instances which is not their intended purpose. This is because one’s instance could easily be censured by a lot of, say, fascist instances, but this should have no overall impact on how non-fascists percive that instance.
Instead, one can see combined censure lists from multiple instances, like one can see combined endorsements. You simply pass a comma-separated list of domains to the /api/v1/censures_given endpoint and you get a list of all the instances they have censured collectively.
This endpoint can then be consumed to make collective blocklists among instances that trust each other, without one having to manually discover and parse different instance federation blocklists all the time.
Likewise, by not being explicitly tied to a blocklist, the censure list could also be used to enforce a softer approach. For example by having an auto-moderator script which flags comments from censured instances for manual review, etc. This could allow an instance to retain a less-restrictive blocklist, but still allow for more rapid response to users coming from potentially problematic instances.
As always, the point of the fediseer is to allow a way to provide the information easily, rather than to dictate how to use it. I excited to see in which ways people will utilize the new abilities.
Just one day ago I released my initial release of the Overseer and I was annoyed by the implementation almost immediately. The requirement to register on another Lemmy instance using a custom username and wait for manual approval (which could also lead to someone sniping that username, and forcing me to manually delete it), and THEN register your instance on the Overseer was just too clunky.
While a few people did register, I realized almost immediately it’s very likely this would never take off. So I started rethinking how I could streamline this process so that it would require far less steps.
My initial plan was to simply register all available instances on the fediverse by default, and allow admins to claim them later. That would require me somehow being able to contact those admins directly. This led me to try to figure out the best way to do that where I discovered I could theoretically talk to any fediverse instance directly, and not have to rely on the a specific lemmy instance with all its limitations.
So I spent many many hours figuring out how to do that. The documentation is frustratingly sparse and incomplete on this point, with my best guide was this blogpost, from half a decade ago for a different language than python. Fortunately this day and age, I had access to ChatGPT, so I could ask it to translate the code on that page to python and then with some trial and error and digging in the official documentation, I had a working DM system for mastodon!
Then I set out to do the same for lemmy which is where the big frustration was waiting, as the documentation is practically non-existent, the messages are cryptic and Rust and the way it’s coded was completely impenetrable to me. Lots and lots of digging in the code, trial and error and asking around in desperation, and I managed to figure out that the main blocking point was a “type” key in my activitypub instance, instead of a fault in my payload.
Unfortunately figuring out how to “speak activitypub” took me the better part of my day. But the good news is that once I had that down, the rest was just a matter of time. I just had to refactor everything. And hell, while I am doing that, why not change the name and domain as well
I have already updated my previous post with the new workflow, but the big difference is that it completely removes the need for an extra lemmy instance one has to manually register. Instead one just has to “claim” their instance and they will get a PM directly in their mailbox!
Likewise, you can guarantee for a different instance even if their admins haven’t claimed it first and they will get a helpful PM informing them about it. If the instance doesn’t exist yet, all you need to do is search for it in the whitelist and it will be automatically added, and then can be guaranteed. My next step is to automatically import all known instances by pulling them from the federation, which should avoid the manual step of searching for them first.
So let’s see how it goes. Now with the ability to talk to fediverse instances directly, it also opens up some really fascinating doors for automation! Stay tuned!
The instance has been growing nicely and holding up very well indeed. but there’s dark clouds forming on the horizon, as more of more of the early adopters and people with principles are leaving that service and are looking for alternatives.
The first signs of trouble appeared when I noticed that the top instances in the Fediverse Observer were growing by thousands of users per day, but had very little activity to speak of, with few threads and barely any comments. This was a clear sign of botted accounts being generated by the thousands.
My initial reaction was to cook up a quick REST API and a complementing script which would allow instance admins to quickly de-federate from instances with this amount of botted accounts, as it would point to an instance with insufficient protection, and those account could easily be used in the future to spam others. A small pre-emptive measure. It’s not particularly sophisticated, but I wanted to get something out before trouble occurs.
The Fediverse Fediseer was born, but I wanted something more. I feel like a big issue with the Fediverse as it stands right now is the same as email. It’s trivial for someone to set up hundreds or thousands fake fediverse domains and start spamming other instances. All it requires is that any account on those instances to follow their spam domains, to open the door. Sure, instances with manual user approval are somewhat more protected, but this approach does not scale well, and will lose us the opportunity to capture the people looking for alternatives outside of corporate control. Likewise any place with open registrations is available to botters to inject their fake accounts in order to follow their spam instances. In general, it’s a big problem to solve through de-federation alone.
My thoughts then was to find a way utilize the “whitelisting” capabilities of lemmy and the fediverse to ensure that only trusted instances are federated. But this has it’s own share of problems. Particularly, it’s also difficult to scale, and it easily excludes people running their own instances.
However the main benefit of whitelists, is that one does not have to be constantly vigilant. A dedicated bot network can spawn thousands of new domains and sleeper accounts to flood the fediverse and de-federation lists would easily grow exponentially to try and fruitlessly fight against this. A whitelist should theoretically remain relatively short and there’s natively protects against domain flooding.
So I wanted to come up with something that would both make it easy to compile and maintain whitelists, but also not lock out people from individual accounts.
After a day of work, I’m excited to unveil the new Fediseer functionality which works around a Chain of Trust!
This all works via a REST API (which I hope some enterprising people will make a fancy UI for it)
The first step is to use the Fediseer API to register your instance’s domain. You can do it from the provided interface, or use a curl call, But I hope in the future the community will develop way fancier UIs to handle this. You will need to provide a username which is an admin on the instance you want to claim (preferably yours).
If all goes well, the user you provided will then receive an API key via Private Message on their own instance, which they can then use to guarantee or endorse other instances.
A new account on the Fediseer however is not visible on the whitelist API by default. This is because only accounts which are “Guaranteed” by other instances are visible. The Fediseer starts with an core instance, the fediverse.com, which functions as the root for the Chain of Trust. Either this account, or another account previously guaranteed, will have to guarantee your instance, before it is available for the whitelist.
Using the Guarantee API endpoint you can now guarantee other instances with your own instance.
Once your instance is guaranteed, it also get its first endorsement from its guarantor, and now will be displayed on the whitelist using the default settings
This list can be exported also in csv format, for easy injection into lemmy whitelists
This naturally forms a network of trust with instances guaranteeing each other down the chain. The purpose of this guarantee is to prevent bad actors from sneaking in. So let’s pretend that a spam instance sneaks in, and starts causing trouble. What do we do?
Very easy, instead of having to waste time going around asking everyone to defederate that instance, and reminding new users to do likewise, we simply withdraw our guarantee for that instance. Once a guarantee is withdrawn, that instance is not anymore visible in the whitelist. Which means any servers which automatically pull and deploy the whitelist from the Fediseer, will automatically reject such instances.
But this goes even further. Let’s say someone pretend to be nice in order to start letting spam instances into the whitelist. When people shout about it, they say the right words and withdraw their guarantees, but keep adding new ones in. Eventually, we’re going to notice that all guarantees for spam seem to be coming from the same instance. When that happens, we just withdraw our guarantee for that instance which does 3 things. A) It prevents that instance from being in the whitelist. B) It prevents that instance from guaranteeing or endorsing others C) It removes the instances lower in the chain branch of that instance as well! So if instance A faked being nice, then guaranteed for 20 spam instances once in, all it would take for those 20 spam instances to be removed from thew whitelist, for the guarantor of instance A, to withdraw their guarantee! If they don’t want to do that, then whoever guaranteed them might withdraw it, until we find someone who does.
Now, I won’t claim that this system is perfect. Human nature being what it is, I expect power groups will form which might not agree with who else is guaranteed. This is where the fediseer being FOSS helps. If there’s a core disagreement between big groups of fediverse projects about who should be guaranteed in the first place, I expect other Fediseers to spawn with their own Chains of Trust which are more or less strict than other. An instance could very well be registered to multiple Overseers and thus be part of different whitelists. I am perfectly aware that I will not be able to satisfy everyone limits but I hope to provide a tool that can!
The main point here is to create the system which can start building Chains of Trust, which have a manual human control but are easy to adjust as the environment changes.
There’s more here that I haven’t mentioned yet, such as the endorsements, where guaranteed instances can endorse others, and anyone can set their whitelist to require more of less endorsements. Or being able to whitelist only instances with endorsements from another instance. Or how the Fediseer will PM you with changes to endorsements and guarantees etc
There’s plenty of ideas to add. This is merely the first step. And I’d love you all to help me take more of them!
Close
Ad-blocker not detected
Consider installing a browser extension that blocks ads and other malicious scripts in your browser to protect your privacy and security. Learn more.
