Deployment - microservices for Kubernetes

(no problem)

Ok, to me they seem a bit like Docker-Compose config files, but each container gets its own manifest file.

Directly comparable to docker-compose manifests. One manifest can define multiple containers (and which images they are based on), the namespaces that they reside in, the services that they access and how the ingress is defined. Or you could write a single manifest, for a single container, that runs in the same namespace as other containers (which interact through ports and/or shared storage). You could manage these individual manifests individually, or all as one deployment manifest (and potentially trigger updates automatically if linked to something like a git repository - see later). It's very flexible.

Services/applications within the same namespace can 'see' each other. You can write a service manifest to allow containers within that namespace to see other namespaces.

Helm is a way of using variables, in a values file, that defines the deployment (so automates, to a degree - there are defaults that might be overridden in defined ways, the deployment through an auto-generated manifest).

So, you might have the same helm file, that (depending on how the values are defined) could either run a postgresql server image within the namespace as part of the application deployment, or access an existing (say HA) postgresql server cluster located somewhere else.

You might have default values file that causes helm to create a manifest that deploys a single application ('talkyard') container (based on a defined image), with both a postgresql and redis container within the deployment, and uses default storage with RWO and only scales to one replica (because you don't know how the default storage works for that environment - it might be nfs, it might be local storage on a single node that the application itself must reside on).

Altering the values file might cause the same helm chart to create a manifest that uses a clustered HA postgresql (within that k8s cluster, or maybe even somewhere else), a specified storageclass that supports distributed RWM access, and thus allows the application ('talkyard') to have up to 'n' replicas (scale up).

Updating the helm chart, when applied using the same values file, can manage container/image updates.

Exactly the same deployment can be achieved by writing a deployment manifest (which may be more complex from a user perspective).

You could define a container image ('talkyard') that was an image (defined by 'dockerfile') that bundled application, database, cache, etc all into the same image, but you would lose the scale out and platform specific flexibility.

Didn't know about OpenFaaS — a few things in Talkyard might indeed be well suited for "serverless" functions, e.g. converting CommonMark to HTML, or maybe custom plugins, or sanitizing HTML.

The problem with microservices, is that the functions called have to be available in every deployment scenario (bare metal server, docker/podman, k8s), in a consistent manner. Unless you are writing an application from the ground up, I would caution against this. It seems to me that it would be much easier to call functions from the application container ('talkyard'), but allow that and the underlying services (database, cache, storage) to be scaleable. A half way house.

The reason I say this is that the same code could then be deployed easily to every platform - it would just be the environment packaging that changes (an install.sh script for server, a docker-compose for docker/podman, a helm chart for k8s).

Yes, it seems such version numbers, i.e. semantic versioning, is required by Helm:

I'm not really qualified to have an intelligent informed opinion on how a project should be versioned. But there's no reason you can't have a separate image project "talkyard-container(docker?)" that manages tagged (auto-build?) builds, or releases, with conforming versioning. Also true for a helm chart based on the images. Images don't have to follow master branch releases, and might lag them for stability and packaging reasons.

It works as follows: The application server container stops, it gets a new (upgraded) image. Then it starts — and this image includes the database migrations, and, when starting, the app server itself applies the migrations (and saves the state in a special database table).

If I were you, I'd retain this scheme. when a container starts, it tests for required migration, performs (or not) as required (as part of init), then runs the app. Scaling up replicas add containers of the same image, which perform the same tests, but tests yield no database migration required (because an equivalent version level container has already performed the migration). I suggested the scaling down to 1 replica prior to upgrade (enter the board into maintenance mode?), so that multiple concurrent migrations don't clusterfuck the database. It depends on the app, and how the migration works. It might be possible.

Down version as well as up? Not impossible, but I would imagine complicated. I'm out of my depth here. (the previous helm chart/images need advance notice of how to strip out features, and adjust the schema, of the next version upgrade. Which hasn't been written yet. Time machine required?). This might limit what you can do in future versions, to maintain up and downward compatibility, and may not be worth it. Alternatively duplicate the database, spin up a new version in a new namespace/server. Test/deploy. Provide a script to add new version contributions, to a previous version database. Revert adding new content to the old version database if you want to downversion? I'm just spitballing. ).

Is that a K8s pod "init container"? Which runs before the "real" containers in the upgraded pod, starts — whilst an old versions pod might still be up & running, handling traffic

Could be. I think I'd be tempted to just include the code in the new version image, and restrict image update to no further than the next major version (where the schema might change in understood ways). Should be a simple test that's quickly skipped past (once the initial migration is complete), and makes the image and deployment multi-scenario (upgrade, run, scale up additional replicas).

Hmm, what I meant was, without any human intervention. Currently Talkyard runs a cron job each day, which looks for new images, downloads and starts them. But in k8s, where would such a cron job run? (or something comparable)

You can run cron jobs in k8s (for example you might use them to trigger something like a borgbackup container to execute a user defined backup scheme, rclone a copy to s3, and push notify result by xmpp - I use this example, because I do exactly this). you could also use something like argoCD (https://argoproj.github.io/cd/) to auto-run updates (could be helm, could be a manifest) from a git repository. So you increment the image version number in git, argo monitors the changes, and auto-updates.

The advantage of this is that I could link to your repository, so auto-update when you issue...or I could use my own repository and control when I do updates, after testing.

I've been thinking about Podman and Systemd (instead of Docker and Docker-Compose), as a default "light weight" installation for Ty.

My personal confidence in docker is decreasing, my confidence in podman is increasing. It's reaching a level of maturity and feature completion. I believe it is less subject to licensing whims, and less exposed to commercial vulnerability. Not to mention the whims of support and licensing. I've been running some test servers, with podman, and a cockpit portal for easy maintenance and visualisation. It's lean, efficient, and does what I want for a single standalone server. It builds containers okay. I think me and docker are finished. Once I get rid of all my legacy cruft.

podman also has a 'pod' construct, where multiple containers can exist in a 'pod' - broadly equivalent to a 'namespace'. It can also be used to automatically create a kubernetes deployment manifest - so test in podman, deploy to k8s.

Longhorn
Hadn't heard, about, sounds interesting. Seems to me it's for self hosted K8s (?).

Can be. Can also be used by a service provider. If I make a cluster I have to make provision for storage, loadbalancing, databases....where a service provider might offer these as services, that you can access by your applications, and pay for in a different way to the compute for k8s. They might put these services in place using similar, or identical tools to what you can use in your own cluster. It comes down to balancing cost and management.

I've been thinking that the Ty customers could choose themselves if they want low budget hosting, or more expensive & more stable hosting. — Maybe Civo some day could power Ty's low budget price plans, hmm. (Or some other similar thing.) Also, there's https://fly.io which looks nice too I think (more expensive bandwidth though).

I could see that you might offer talkyard customers managed k8s based hosting. The backend infrastructure might depend on plan level. Deploying a new instance simply using helm with a template. Access through a simple service manifest linking to customers FQDN.

Kilo https://github.com/squat/kilo sounds nice — I wonder, was it hard to configure all that? What about IP addresses — if a HTTP web server pod, is in Amazon US East-1, and other pods in GCE EU West, then, ... let's say AWS US East-1 goes offline, then, I'm thinking the traffic would still get sent to the now offline pod? Hmm, maybe Kilo is more for backend services? Or also for frontends, with more fancy IP rounting? (so traffic gets re-routed to EU West?)

Kilo is just the CNI. So I have distributed servers, making up a k3s multi-master cluster. Intra-cluster traffic being encrypted over wireguard using kilo. All the normal cluster operations work (but slower, because the servers are not on the same subnet). I can use a loadbalancer (metallb) to expose a service on any of the nodes (the applications don't necessarily run on that exposed node). In addition I can, for example, spin up a vps (or indeed another cluster) somewhere, connect it with wireguard to the cluster, and access services on the cluster. So the cluster I have, for example, could do HA postgresql, and simple vps could run an application (or just a web-proxy) accessing the cluster.

It's not 'high end' performance, but it is low budget.

Hmm, yes, and the old namespace could be read-only, during the migration, I suppose. So, there wouldn't have to be any downtime, just some read-only time.

This should be entirely possible.

Aha :- ) Then, if you open the editor, try the "Place left" button. Then you'll get even more "collapsible" side things. & simpler to reply to long replies. — "Soon" the edits preview, too, will be visible, side by side with the editor and the comment one is replying to. (But that'd be for somewhat wide screens, >= 1900 px maybe.)

I like what has happened here. Following the instruction I have an editing pane, and a discussion pane, that I can easily work with.

Pressing "show preview" renders a live preview in position in the thread.

This works for me. I don't think I would like another preview pane. I don't need it, and it would reduce the clarity of what I am working on.

If I navigate away from the thread (for example to check something, or copy and paste something from elsewhere), I have to manually navigate back to the thread I am replying to, in order to get back to the thread/reply.

I would prefer a button on the editor, that navigated me back to the thread and my reply, more than an additional form of preview. (the "show preview" button taking me back there would be ideal, but currently I do not believe that it does).

If that were possible.