posted in Fediverse
Fediverse Observer is offline and no longer being developed
The domain https://fediverse.observer redirects to sedo now.
On October 11th these two commits where made to the repo:
this will be my final release, glad to have contributed since 2011 to some fediverse stats, https://fedidb.com/ is a great resource with more interest
this software is no longer developed
Sad to have lost such a great resource.
Replying to @squirrel@cake.kobel.fyi
@vicinus@piefed.zip fyi, this will impact !fedilytics@piefed.zip right?
Yeah. I saw the dead website earlier today and have been trying to figure out how to keep the posts working.
The fediverse software post, I could rewrite some code to gather fedidb daily (to my knowledge, they don’t have all time series data available), but their MAU math is suspect. So, not a great option, but my only other thought is basically remaking fediverse-observer and I do not have a server or potentially relevant server skill set beyond cron jobs.
For the threadiverse instances post, I can keep gathering data from the instances I already have in my database, but don’t really have another way to find new ones. Maybe, I could see if I could scrape it from fedidb (I couldn’t get their API working in the past).
In conclusion, the posts are not in a good place :(
Replying to @Vicinus@piefed.zip
but their MAU math is suspect
Do they do any math? I assumed it's simply additive. They get the MAU from individual instances via their crawler, then display a total?
Differences to other sites is probably due to the set of instances checked, not anything math related.
You’d think it’s straightforward and additive. I added up all their listed instances for a couple different softwares. The math was wrong and overestimated the MAU count. I checked it a few times across a couple weeks. Couldn’t quickly figure out a pattern to how the math was wrong each time.
Maybe they fixed it in the last ~6 months, but I wouldn’t count on it. I believe it’s the same Dev as pixelfed and loops. So, not sure they’ve done any work on fedidb in years (to my knowledge, they’ve focused on Loops and Pixelfed).
Edit: to clarify, adding the MAU of all their listed instances for a particular software was lower than the total MAU they reported for the software.
Edit 2: Just checked their Piefed listings and it’s still wrong. Fedidb states: ~7.0k, actual addition: 6858 MAU.
Replying to @Vicinus@piefed.zip
Have you contacted them about the discrepancy? Ideally FediDB could just become the new Fediverse Observer.
Missing out on accurate data about Fediverse growth would really suck. 😕
I just confirmed the problem still exists and messaged them via mastodon about it.
Probably didn’t let them know originally because I didn’t have a mastodon account at the time and already had a good option (fediverse-observer).
Replying to @Pamasich@kbin.earth
Differences to other sites is probably due to the set of instances checked
fedidb.com/software/lemmy?search=lemmy.world&stat…
Fedidb doesn’t seem to include lemmy.world, which seems like a substantial omission, as I believe that it’s the largest Lemmy instance.
EDIT: And my own home instance, lemmy.today, also doesn’t seem to be on there:
fedidb.com/software/lemmy?search=lemmy.today
The largest user count server that they seem to query is lemmy.ml:
fedidb.com/software/lemmy?search=lemmy.ml
EDIT2: Your home instance, kbin.earth, does seem to be on there:
Replying to @tal@lemmy.today
That is interesting to know. It doesn't look like lemmy.world has opted out, judging by their robots.txt.
Replying to @Pamasich@kbin.earth
Oh, that’s a thought. I didn’t think of checking robots.txt.
Lemmy.today does have a blanket ban on all bots in its robots.txt, which I assume was put into place to help try to reduce (sporadically severe) scraping load. I’m not sure that the admin, @mrmanager@lemmy.today, realized that it might have cut it out of Lemmy statistics gathering.
Replying to @tal@lemmy.today
Yeah I know, but its worth it. :)
Replying to @tal@lemmy.today
I knew to check robots.txt because of fedidb's Data Sources page, which specifies what data they get, their user agent, and how to opt-out via robots.txt.
One thing I'm noticing in regards to lemmy.world, I wonder if they ever disallowed access to the nodeinfo endpoint in their robots.txt in the past. They don't currently, but the wording suggests that having that rule at any point will forever remove the instance from fedidb. Which could explain what we're seeing for that instance.