Hi! Welcome to the app!

This is an extremely early preview with a lot of bugs and incomplete features.

The intent of this thing is to make it a federated social front interface to the Web. For now that includes an RSS reader, a web search engine, and limited ActivityPub capability.

You don't have to read all this stuff if you don't want to. While you're using the site, I just want you to focus on one question: "Why would I not use this?"

Jump to the help section for some quick starting tips. Also check out the changelog for more recent info.

What is it and why

fedi or tbd is a placeholder name while I work out what to actually call it. Current top choices are "Xoro" and "Anansi". Still not sure, so feel free to suggest.

This app is a front interface for the Web, created to replace the poor-quality, bad-faith, or no-longer-available legacy tools we're used to. Its goal is to prioritize sites and services that practice good Internet citizenship, without biases toward SEO or mass surveillance.

The code will be opened up and released under the AGPL Real Soon Now. At least as soon as it's worthy of being looked at, and not full of load-bearing FIXME comments

The features

Obviously we're not feature complete yet. Here's what's on the roadmap, with existing stuff checked off:

  • RSS / Atom reader
  • Web search
  • ActivityPub integration (currently minimally functional with lots more left to do)
  • Macroblogging
  • Administrative federation

Let's look at each one in turn:

RSS / Atom

Syndicated content still exists, but the death of Google Reader and the dominance of algorithmic news feeds have suppressed it quite a bit, leaving users to hope they'll be spoon-fed their desired content by aggregators like Google News

There isn't a lot to say about this feature if you're already familiar with RSS. Subscribe to a site, it shows up in your feed in the order the posts were created, so that you can catch up and move on with your day.

There's a toxin filter on the feed aggregator that strips out stylesheets, Javascript, iframes, and other horrors.

Web search

Yep, we're a search engine.

Google's original best feature was how well it indexed the entire Web, but now thanks to SEO, surveillance capitalism, and AI content, the overwhelming majority of web search results are trash. And a large multinational corporation doesn't have a financial incentive to change that.

Indexing the entire Web is no longer desirable. Here, instead, is the model we're using:

  • A site's administration team chooses sites to index based on local interests and storage capacity, and curated on content quality.
  • Users of the site can tag and rate indexed (and not-yet-indexed) sites to indicate interest, value, quality, etc.. Admins can choose to add or remove sites with this information in mind
  • Admins of different sites can create a trusted peer connection with one another, to share index responsibilities (see Administrative Federation, below)

ActivityPub integration

RSS and Search have always felt like they were missing a social function. Here, we can not only subscribe to an ActivityPub user's feed just like an RSS feed, but also re-post interesting articles or search results the same way we can with posts on traditional social sites.

Macroblogging

Yes, the actual term is simply "blogging". Making the distinction here because ActivityPub tends to be used primarily for short-format microblog content.

Here we will be focusing on the ability to federate longer-format, rich-text content via ActivityPub (and RSS!). This is something you can already get in a few other places, and is being included here just for the sake of making it a complete social application. If it proves useful it will be developed in detail

Administrative federation

Here we're creating a new (?) type of federation. Not just from user to user, but from site to site. This will get us:

  • Cooperative search. When a user sends a query, peer servers pitch in and provide results based on what they index.

    • Peered search goes into a lower priority queue so that helping your friends doesn't result in a hug of death
    • Second (or more) degree search will be possible depending on peer configuration

  • Index data sharing. Peers can optionally transfer entire search indexes to one another, to let them index the same sites without having to crawl the same targets

  • Split indexes. For very large index targets (think Wikipedia), share the compute and storage cost of indexing with one or more peers

  • Shared blocklists. Defederation needs to be a first-class feature, and peers can opt to track what's being blocked on each other's servers, or a certain number of degrees away.

  • Satellite sites. If you're hosted in a public cloud or VPS provider where disk space and IOPS are at a premium, create a trusted peer connection with the multi-terabyte data-hoarding NAS we know you already have at home.

How can I help out?

For now? Test! Finding bugs is great. The hope is that this can be the tool we all use for our main access point to the Web, and if there are reasons why you're not yet inclined to use it that way, help identify them. That stuff is at least as important of a defect as any software crash

Boring nerd stuff

I'm going to provide better detail when we get closer to having the ability to receive community contributions

This is the most ambitious code project I've produced, and is built on a pretty great tech stack I created years ago for a radically different project that fizzled out.

At its core are several (micro? mini?) services written in async Python that use NATS JetStream, a great little Golang-based message bus, worker queue, and key-value store

We're using SQLAlchemy via asyncpg for relational data storage and alembic for schema tracking

Web search is managed by Meilisearch, which is a search engine written in Rust that is lightning fast and uses a tiny fraction of the resources required for a tool like Elasticsearch

In development mode, there's a Docker Compose stack that launches one of each service plus a pytest container. Each app container and the pytest container monitor the code using Nodemon and every time a change is saved, the services restart and the full synthetic/integration test suite runs

Web crawling supports different modules. One of the included modules is a direct crawler written specifically for this application, and the other abuses CommonCrawl's S3 data storage rather than crawling sites directly, which is nice but very bandwidth intensive unless you're running in EC2. Everything uses BeautifulSoup because BeautifulSoup is love.

On the front end we're fully invested in FastHTML, making fairly heavy use of htmx and hyperscript, with no React / Node horrors anywhere. Read the HTML source. It's simple!

There's even a custom CLI for the application, called feditool, which runs commands inside the Docker stack or lets you manage the message bus

In retrospect this could have all probably been done in Django.