Filesize limitations to uploads

I've started looking into integrating with Google Cloud Storage and / or AWS S3 and DigitalOcean and Scaleway's object storage solutions.

Since Talkyard .net uses Google already, probably initially I'll add support for Google Cloud Storage.

Hopefully this'll be available within ... 2 weeks? Thereafter your users will be able to upload large images and videos. (How large are the largest files they've wanted to upload this far?)

Cool! We're also on GCP so this would be perfect for us :)
How were you thinking to do the implementation?

The way I thought of it was, that we should be able to somehow create a storage bucket in our GCP account and then create TY as a member and then in the TY admin interface we would enter the API key or whatever credentials are needed.
I don't know the entire workflow for how to enable 3rd parties into storage buckets but that's the general idea of how I thought it should work.

Another thing to consider is the ability to use fine grained permissions within storage buckets. There's something to consider for the case where users would post files to closed off portions of the forum and whether or not those files should be in a publicly available storage bucket or not.

Yea wrt spending limits, it definitely is a complicated subject and DDoS attacks could become very expensive if this is not taken into account for sure. But if we're hosting content on our own GCP project at least TY doesn't need to care about this.
It would still be helpful to configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server (I think there would be a case for using JS to upload directly to the Storage bucket, then TY would never see this traffic).

Hi again @chrscheuer

How were you thinking to do the implementation?
we should be able to somehow create a storage bucket in our GCP account [...] Ty as a member

This sounds good to me. I'll look into this now this week. Probably I'll add a default cloud storage, + CDN with spending limits, for everyone who uses Talkayrd's hosting — and then you can override this, by uploading your own Cloud Storage credentials for your Talkayrd user (and, later on, configure your own CDN too should you want to). I had a quick look at how GCP & identity and access management works, and seems to me all this will work fine.

fine grained permissions within storage buckets

Yes that sounds like a good feature to have, ... and, personally I won't have time to look into this, not the nearest half a year at least. — Uploaded files have SHA256 hash names, so they're essentially hidden, unless one has a direct link.

(I noticed that CDN:s also seem to support access permissions per URL, so maybe fine grained permissions can work also when a CDN is connected.)

configure some sort of threshold that was enforced by TY both at the clientside (before uploading, should be possible with the File class in Javascript) and potentially serverside - if directing through the TY server

Having a look at the files client side, is a good idea. Thanks for linking to Web/API/File/size. And server side too of course. Also, maybe one would want to specify if and how much the files should be compressed — I'd think in most cases, a 10 MB macOS Retina screenshot, is "overkill"? and maybe such large images should be downsize / lossy-compresed to 1 or 2 MB? (there're tools for lossy-compressing PNG images too).

Maybe won't implement all this in the initial version — I'll start with just connecting to Cloud Storage & CDN.

All of this sounds great!
And yea completely agree on the hashed file names/URLs being enough "security by obfuscation" for most purposes (including all of ours).
And yes auto-downsizing images is a good feature to put in the backlog for some time in the future :)

Sorry this got a bit delayed. (Code reviewing some code took days longer than what I expected. B.t.w. if you type a reply — you'll notice that now there's a live preview of your response, so you can see exactly how it'll look, in the page. And maybe the preview in the editor itself hereafter in most cases isn't needed.)

I started looking into this now this morning ... & this week instead of last.

Thanks for the update :) IMO it's a little confusing right now to see your text 3 places, 2 I think is the maximum of what makes sense to look at, so while the experiment is great in itself my personal opinion is that it makes the editing experience slightly worse right now.
Perhaps the eventual best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components.

This is getting even more delayed, sorry. (Last week, backups suddenly stopped working: gzip started causing "soft lockup"s and kernel panic and the Linux server shut itself down, until I added nice -n19 + I finished some unimplemented backup script things. And now, this week, I've been fixing some unfinished things related to migrating away from Talkyard .net to a self hosted server.)

best editing experience is something we could look at - and talk about soon - while also considering being able to plug in custom made editor components

That would be interesting, yes. — For now, probably I'll hide the in-editor preview, and keep only the in-page live preview. (Except for when one posts a new topic and there is not yet any page.) The thing is, people in some cases reply to the wrong post, so I'm thinking it'd be good if they can see precisely where their reply will appear, e.g. in-reply-to which other reply. (B.t.w. I've seen this happening (people replying to the wrong comment) relatively often with Facebook's semi threaded comments too.)

Gotcha.. Yea you're right replies often end up in the wrong places. Will it be possible to roll back the design change though if it doesn't work? I see lots of flashing of the entire page with the loading bar etc. when editing whenever I have a slight pause -- so until the design change is more complete I feel like it still has the potential of making the editing experience worse. Perhaps you can make the change opt-in to begin with, ie. behind a feature flag until any bugs and design issues can be fixed?
Ways that could improve the experience:

• Never show a loading progress bar while I'm editing. It's very very annoying and almost driving me crazy just as I'm typing this response.
• As a first step, just show a placeholder (eg. the text "Your reply to ... will appear here"), not the entire text of the reply up in the thread - and keep the current preview system. It makes sense to view your markdown / preview side by side so you can verify your formatting works and not up by the thread. It's annoying to have to look up there to see how it looks.
• As a general rule, don't make large design changes without some sort of beta test, opt-in or ability to opt-out.

And no worries with the filesize limitations.
I just really hate the design change with the previews. It's very distracting from the experience of typing the answers, which is something I often spend hours on every day.
I understand the problem you're trying to solve and I agree that that problem is important to solve, but IMO the current proposed solution is not an improvement but a quite serious degradation of the UX.

Ok. I find it interesting that we have different opinions-about / feelings-for the inline preview.

Then I'd like to add a feature toggle for this, before releasing the new version, so you can disable it (for your whole forum).

I suppose it's good with different ways to tweak the UX, depending on one's forum's audience and how tech savvy they are.

I'm not ruling it out permanently. It may at one point be better than the current UX, but I'm surprised you're not seeing that it isn't ready for prime time yet. My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done. It's incredibly distracting and definitely not a finished feature. If you updated the server for everybody without a switch to turn it off it would be enough for us to seriously start considering hosting our own fork of TY.
All I'm saying is that for text editing systems, the user experience is an absolutely central element and any change to it should be A/B tested and thoroughly vetted until you change it for end users. This doesn't feel tested to me.

Thanks for the feedback! I'm not experiencing these things myself, still, I definitely neeed to fix this — I could reproduce the problems you're seeing:

My screen is constantly flickering whenever I pause my typing, sometimes putting up a loading graphic for seconds so I can't continue typing until it's done

by enabling Slow 3G network emulation in Dev Tools. Ok that's an enormously annoying editing experience.

Actually, I think the things you mentioned, are related to 1) you now being far away from the server. (But I'm fairly close to it.) And 2) a bug that pops up the loading indicator, although that shouldn't happen in these cases. But not related to the in-page preview.

I'll try to fix this before the next release. I feel pretty certain other people have the same issues (without telling me about it). ( https://news.ycombinator.com/item?id=21427996 — "They Might Never Tell You It’s Broken").

Great you were able to repro it. Yea that's a common issue (edit: people not reporting issues) which is why having a great beta team/process is a critical part of long term success with software IMO.
Loading popup errors in strange scenarios is one of the most common issues I have with React myself as well, and often it can be quite difficult to repro why it happens for some customers but not for one-self. I hope with their new work with async rendering stuff like that should be easier to fix in the future.

(I'd just like to menion that I've fixed this problem, with the flickering preview & loading-graphics, in my work-in-progress GitHub branch. Probably I can deploy the fix the first days next week.

often it can be quite difficult to repro why it happens

In my case, now, in Dev and Test mode, if a hostname includes "slow-3g" the server adds 2 seconds latency to each request to that hostname. That made this particular problem reproductible in the auto tests too)

Cool :)

@chrscheuer Now i've upgraded this Ty .io server so the flickering preview & editor and loading indicator problem, should be gone now. — Would you like to test?

B.t.w. there're UI config prefs that you can specify for the Everyone group,
which removes the in-page preview, and instead shows a split-view preview inside the editor, like before.

Testing testing... 1 2 3
Yea this seems to have fixed the flicker :)

Ok, I just upgraded the server. — If you have feedback / want to change the UI prefs, maybe you'd like to open another topic about that?
(I think I slightly derailed this topic about file sizes, by starting talking about the editor & UI :- ))

Wait you changed the UI without an option for us to opt out? I thought that was the plan. Ugh...

You can opt out by going here: /-/groups/everyone/preferences/ui and copy-pasting this into the text field:

{ "inp": 2 }

There's going to be user friendly buttons and dropdowns for this later. ("inp" means "In-Page Preview".)
(Don't forget to reload the page afterwards.)