← Back to post

Edit history

Most recent

probably all the posts in the “fediverse” have been taken already

I don’t doubt this, as it’s trivial to do (by design). You don’t even have to scrape anything.

New posts and comments in a Lemmy community (or Mastodon profile or any other ActivityPub-based Fediverse app) get pushed to every server where federation is allowed and at least one person is subscribed to it. You can spin up your own server, subscribe to a bunch of communities, wait a while, and over time your PostgreSQL database will be populated with profiles, posts, comments, etc.

Edited

probably all the posts in the “fediverse” have been taken already

I don’t doubt this, as it’s trivial to do (by design). You don’t even have to scrape anything.

New posts and comments in a Lemmy community (or Mastodon profile or any other ActivityPub-based Fediverse app) get pushed to every server where federation is allowed and at least one person is subscribed to it. You can spin up your own server, subscribe to a bunch of communities, wait a while, and over time your PostgreSQL database will fill be populated with profiles, posts, comments, etc.

Original

probably all the posts in the “fediverse” have been taken already

I don’t doubt this, as it’s trivial to do (by design). You don’t even have to scrape anything.

New posts and comments in a Lemmy community (or Mastodon profile or any other ActivityPub-based Fediverse app) get pushed to every server where at least one person is subscribed to it. You can spin up your own server, subscribe to a bunch of communities, wait a while, and then just read all the posts and comments from your local PostgreSQL database.