Adding another Scheme.org server + better compartmentalization
Lassi Kortela 24 Nov 2023 19:45 UTC
The main server running most Scheme.org subdomains has been at full
capacity for a while.
For example, if you've witnessed Gitea crashing, it's because the
Postgres database runs out of RAM (and swap) at intervals. This does not
cause data corruption, but does cause service disruption.
A lot of the resource usage is due to botnets hitting the site, which is
probably unavoidable on today's internet. This also causes log bloat.
It's common for one log file to take nearly a gigabyte of disk. We have
some log rotation, but apparently not enough. To avoid disruption, logs
should probably be stored on a dedicated file system.
Even if the above is taken care of, the main server still isn't beefy
enough to host everything. I'd like to take this opportunity to advance
the "microkernelization" of Scheme.org (as explained in the original
announcement) by keeping the front page and some administrivia on the
current server, and moving the community subdomains to a new server.
This would also make server configuration and git repos easier to figure
out, as the main server would correspond to the github organization
https://github.com/schemeorg and the community servers would correspond
to https://github.com/schemeorg-community. The current layout is quite
confusing.
This being Black Friday, there are some great deals at
https://lowendtalk.com/categories/offers and I already rented one new
VPS from RackNerd which is a reputable provider in that community.
Barring objections, I could move much of the current stuff over to that
server. Arvydas' server, ironwolf.servers.scheme.org, may also have some
spare capacity.
Scheme.org is fundamentally DNS-based, git-sourced, and automated using
Ansible, so it is quite easy to experiment with different servers and
move subdomains between them with little disruption.
Let me know if there are any suggestions or objections. If I don't hear
anything by the end of the weekend, I'll go ahead with the plan. I
emphasize that this is just a server issue and should not cause
user-visible changes.
As before, ssh and git access is available to people who'd like to work
on the site.