Talkyard API: Upserting categories

@chrscheuer Now I just deployed the new version (v0.6.41) with the upsert API. Feel free to try it out. B.t.w. there are rate limits:

  object UpsertSimple extends RateLimits {
    val key = "UpSm"
    val what = "Upserted things too many times"
    def maxPerFifteenSeconds = 10
    def maxPerFifteenMinutes = 30
    def maxPerDay = 90

I you'd like to try out the API on a test site first (instead of your real live site), then go here: https://www.talkyard.io/choose-demo-forum, and click Create my own test forum. Then send me a link with the URL to the test forum, so I can enable the upsert API for that test forum too.

Awesome! Looking forward to trying it out. Thank you for pushing this!
Wow - the rate limits definitely will make sure that we can't use this as the primary method for users to get to the forum (that would limit us to 90 page views of a forum per day). So with this we'll need to implement our own caching for sure.
I'm a bit afraid to hit that daily or 15 min limit just in debugging when implementing the system. Since we'll need to do our own caching this also means I'll need to run a bulk operation once we have completed the integration, one that runs over all the packages in our system. I'm pretty sure we could hit those limits pretty easily once we want to run the bulk operation.
This essentially means we'll need to implement queueing on our end.

As I see it there are a couple of ways this can work:

1. Talkyard has a rate limited API. This forces us to have a database trigger (when user publishes a package, we register a slug, we talk to TY to get a forum url for that slug, this is stored)
   When user wants to navigate to the forum, we'll know it already works. If the TY API call failed, we'll need to either have it add the request to a queue that manages to keep us below the rate limit (a lot of extra work), or build in workarounds to access the API when the user tries to go to a forum that didn't get properly created. This could in turn mean even more API access leading to cascading errors of the whole system.
2. Talkyard has a non-cached, non-rate-limited API (or the limit is high enough we won't see problems). We still have a database trigger - so we access the API only once per package publishing. Effectively the queueing concern will be gone so no need to build in that or any workarounds. The effective number of requests to the API would be the lowest with this approach.
3. Talkyard has a cached, non-rate-limited API. Since SoundFlow still needs to provide the slugs, that's a potentially slow operation, so we'd still like to do this not on a pageview level but as part of the database trigger.
   So the Talkyard internal caching would here just be an implementation detail of the TY system and lead to potential better usage of server resources. However we'd have the exact number of API calls as in #2, just 1 per package creation.
4. (Rather 3b) SoundFlow splits up our system so that we make sure to define unique slugs as part of our database trigger (when user publishes a package), but then utilize a Talkyard caching endpoint to use it for every pageview/redirect operation. This would have TY API see the most traffic. We would send the same data as now but instead of for every package creation, we'd send for every page view / redirect from main site package to forum package category (same style as SSO).

If we did not need to provide slugs, the system could be simplified for the API consumer, and a potential cached solution from TY would have even more value than #4. Because this would allow us to use the API just like SSO and we wouldn't need to have our own separate algorithm for slugs. Essentially we could use it then more or less use it as part of a redirect for the user, whenever such a forum url would be accessed. We would in this case just send an extId and a name (and parentRef) and get back an url. All stuff that wouldn't require us to have any queueing or database triggers. This would be almost like embedded categories.

The issue here IMO is in putting a good distinction between what's the role of the API and the API consumer. With these pretty low limits it's making some scenarios that should be simple for us to implement much more complex. I'm a bit reluctant to go too far down that rabbithole if we can come up with a better solution.

In essence what I'm afraid is that whereas this API would be called as often as SSO and have no bigger cost on the system than SSO when used as a Category upsert function, the API was designed to be able to do bulk operations, which of course you'd probably need to rate limit.
One idea: Would it make sense to lift the limit (or put it same as SSO) for simple queries that just contain a single category?

Sorry if any of this sounded too critical :) I'm very happy we've gotten this far! And this should definitely be something we can work with. Via our Firebase setup we could do queueing etc if really needed. So we will get something great eventually, no matter what. This is good as a 1st API iteration.
But since what we're trying to do is essentially a specialized version of embedded categories - I think it doesn't hurt to talk about if the API can move towards something equally simple to that (which would be essentially completely URL based I suppose).

whereas this API would be called as often as SSO and have no bigger cost on the system than SSO when used as a Category upsert function, the API was designed to be able to do bulk operations, which of course you'd probably need to rate limit.

One idea: Would it make sense to lift the limit (or put it same as SSO) for simple queries that just contain a single category?

Yes, this sounds like a good idea. Sometimes the /-/v0/upsert-simple requests wouldn't need to write to the database at all. Probably the requests can be optimized to access only the in-memory cache. — Maybe there could be different rate limits: 1) one for requests that upsert just a few items (e.g. one single category) and that actually didn't need to upsert anything, didn't write to the database. And 2) another rate limit, for requests that do write to the database, but just a few items (like, one category). And 3) a third rate limit, for bulk upserting lots of things.

What rate limits do you think would work for you, for the initial import of everything?

So you won't need to spend time writing your own queue to stay below the limits, and so you won't need to do a bulk operation, ... But instead can upsert everything, one request at a time (that's simplest for you?) as fast as you want?

(Maybe this would be approximately the number of packages you have? Per 15 seconds?) ...

... Maybe I can just bump the limits a bit (or a lot) for now, since currently one needs to ask for permissons to use this API endpoint anyway. And I can restrict /-/v0/upsert-simple to only accept a single category — and reject bulk uploading lots of things. And then later on, I can add more restrictive rate limits, for API calls that bulk uploads lots of things, and start allowing that.

(I'll read everything you wrote in more detail a bit later)

I've created a demo site now here:
https://test--soundflowtest.talkyard.net

Would be great to test the integration here first :)

Is this still the definition for slugs?

"Which characters are allowed in the slug" — I have in mind lowercase alpahumeric + hyphen - ok inside (but not starting or ending with -) and at least one letter (not only digits).

Looking at this for a slugify function. Do you see any problems with it?

function slugify(string) {
  const a = 'àáäâãåăæąçćčđďèéěėëêęğǵḧìíïîįłḿǹńňñòóöôœøṕŕřßşśšșťțùúüûǘůűūųẃẍÿýźžż·/_,:;'
  const b = 'aaaaaaaaacccddeeeeeeegghiiiiilmnnnnooooooprrsssssttuuuuuuuuuwxyyzzz------'
  const p = new RegExp(a.split('').join('|'), 'g')

  return string.toString().toLowerCase()
    .replace(/\s+/g, '-') // Replace spaces with -
    .replace(p, c => b.charAt(a.indexOf(c))) // Replace special characters
    .replace(/&/g, '-and-') // Replace & with 'and'
    .replace(/[^\w\-]+/g, '') // Remove all non-word characters
    .replace(/\-\-+/g, '-') // Replace multiple - with single -
    .replace(/^-+/, '') // Trim - from start of text
    .replace(/-+$/, '') // Trim - from end of text
    .replace(/^\d+$/, m => `category-${m}`) //Prepend "category-" if we had a number-only slug
}

I appear to have it working now!

Was just created via the upsert API :)
Edit: and it was triggered by our full package publishing pipeline. Our cache layer and slug generator etc. seems to work as expected :) Yay!

I'm inserting all these without setting the "position" property.
Is there any chance automatic sorting could be implemented on the TY side, in case categories don't have positions?

And here it is now working in the app :)

Hi @chrscheuer,

automatic sorting could be implemented on the TY side, in case categories don't have positions?

I think so — does that mean alphabetically? Or by most-active-first? (I'll look into the SSO-top-window-redirect issue first)

And here it is now working in the app :)

Thanks for the screenshot, I think it looks nicely integrated :- ) (Hmm possibly some minor things to maybe consider some time later, like: the title "soundflow" on the top nav bar with the blue background — maybe that title isn't needed? And the forum title a bit below, again. Maybe some URL parameters could hide the forum title & intro text, + top nav bar? Or a CSS class on the html elem, if is embedded. )

If one navigates to a different category, then clicks a different app tab (say, the "Save Version" tab), and then clicks "Package Forum" again — then, does the forum view get recreated, and one starts in the Online/Offline package again? Or is the forum view remembered, and one is still in that different category?

the SSO process seems to redirect the top window at some point in the process

I think there're two problems: 1) The Package Forum tab link has a redirect url that sends the browser to Talkyard's forum, rather than back to the page at Soundflow. This page: https://soundflow.org/store/dialog-editing-izotope/forum somehow redirects the browser to: https://soundflow.org/forum?returnTo=https%3A%2F%2Fforum.soundflow.org%2Flatest%2Fdialog-editing-izotope — and look at the returnTo=... parameter. It tells SoundFlow to send the browser to Talkyard's domain.

Something like this instead, could work: https://soundflow.org/forum?returnTo=https://soundflow.org/store/dialog-editing-izotope/forum

Then there's problem 2: Because of maybe-security-issues, Talkyard strips the url origin from the_return_to_url in: http://localhost/-/v0/sso-login?oneTimeSecret=nnnnn&thenGoTo=the_return_to_url. So even if Talkyard gets an url with the correct origin (i.e. https://soundflow.org/store/dialog-editing-izotope/forum), Talkyard will refuse to use it, and only looks at the url path (/store/dialog-editing-izotope/forum) instead. — I previously didn't want to allow other url origins than the Talkyard server itself, because of maybe-pishing attacks.

Here's Talkyard's code that runs when calling http://localhost/-/v0/sso-login?oneTimeSecret=nnnnn&thenGoTo=https://soundflow.org/store/dialog-editing-izotope/forum

        // Remove server origin, so one cannot somehow get redirected to a phishing website
        // (if one follows a link with an "evil" go-next url param to the SSO login page,
        // which then redirects to this endpoint with that bad go-next url).
        val thenGoToUnsafe = getOnly("thenGoTo")
        val thenGoToHashEncoded = thenGoToUnsafe.flatMap(Prelude.stripOrigin) getOrElse "/"

That protects against someone who crafts an evil link:

https://soundflow.org/forum?returnTo=https%3A%2F%2Fwww.evil.com

Then, the user would login over at soundflow.org — the real domain — and believe s/he is safe? But would then get sent to an evil server, possibly phishing. Well, unless one or both of SoundFlow and Talkyard detects and rejects unexpected redirect-to destination domains.

Maybe there could be a whitelist of allowed go-to-after-SSO-login origins? There's a setting allowEmbeddingFrom, intended for embedded comments. Maybe Talkyard could allow redirecting to the allowEmbeddingFrom origins after SSO login?

One little note - maybe there should be a limit UI wise on the number of sub categories shown (once we have 57 packages here it could become crowded):

Hmm, yes that seems like a probably good idea. Maybe a max height, + a button "Click to show" ? I could look into this a bit later, after the iframe SSO redrect, and rate limiting issue, and returning the category url path.

Another approach could be to place the Packages category last, so it'll be at the bottom of the page. Then maybe doesn't matter if there're a lot of sub packages and the Packages section gets really tall? Since there'd be nothing below

Is this still the definition for slugs?

Yes. I'm not sure about foreign langage chars, e.g. Chinese? But maybe for now, non-ASCII chars can be forbidden, ... until after a bit research. I'm thinking only [a-z0-9-] is safer, for a start.

Looking at this for a slugify function. Do you see any problems with it?

I had a not-thorough look ( + I'm tired :- )), and I think it looks fine, mainly because of this line: .replace(/[^\w\-]+/g, '') which sort of deletes all "weird" things ... since \w = [A-Za-z0-9_] says MDN. B.t.w I have a slugify function in Talkyard, which converts titles to slugs. Here it is:

Seems there's a bug in my slugify — it allows pure numeric category slugs. Hmm your slugify accepts _, Talkyard's doesn't. Maybe it's better if I change Talkyard and allow _ in slugs.

(Talkyard's slugify is based on an AngularJS plugin or something. I was using AngularJS long ago before moving to React.)

Maybe a max height, + a button "Click to show" ? I could look into this a bit later, after the iframe SSO redrect, and rate limiting issue, and returning the category url path.

Yea that sounds great. Haha yea this did turn up quite a list of follow up items :)
For now we're completely good with the current solution, everything seems to work as expected.

I had to hard program the URLs around in our site and since we're caching them and prerendering the URLs to the site's static content (which means Google is also picking it up) we'll likely experience a little down time on forum links once you make the change. We may also have customers bookmarking or deep linking to the current URLs. So maybe before the URLs change it would make sense to look at redirects (history). We can live without it, but of course it would make the user experience a little better (and make sure we don't take SEO hits). Implementing redirects would be more important to us than returning URLs in the API.

Another approach could be to place the Packages category last, so it'll be at the bottom of the page. Then maybe doesn't matter if there're a lot of sub packages and the Packages section gets really tall? Since there'd be nothing below

Yea I'll look into that if we get an explosion of packages. But actually I prefer it to be up at top for the moment, since we primarily want users to use either the "How to" section or the "Packages" section, so hiding the content below is not a big issue for now - just something to think about in the future.
I think your max height + "click to show" sounds like a good solution and I agree with your prioritization to take this task last :)

I just realized something.. Since you were talking about the slugs right now being global (not scoped).
Is there any potential for us to accidentally hit slugs that were created by talkyard in other categories, pages or posts (not synced from external)? We of course guarantee the slugs we use are unique, but if you just have a single index for everything, won't we risk accidentally using slugs that were already used by others?

Ok, so I'll implement category redirects from old names to the current name, before moving sub category url paths to [below their parent category's url path].

Sounds great!

What if, when one upserts a category, if one specifies a title only, but no slug, then Talkyard can auto generate the slug?

This would be great. And I trust when Talkyard auto-generates it would always be valid, right? Meaning if there's a collision, it itself handles incrementing a counter at the end?

So, after having upserted a category, you can send the browser to:
serverOrigin + responseJson.categories[0].urlPaths.activeTopics.

Sounds good!

Great stuff!
I've been offline for a week, and will be for another, so won't get a look at this until early September. We're launching SoundFlow 3 on September 5 so right now we're very zoned in on the launch process.

Thanks for asking my opinion about the order.

I would say the order in your items right now makes sense (the first 3 ones are in the proper dependency order IMO), except for the iframe redirect problem which I would address earlier (it feels more like a bug where the rest is more feature work).

One thing I'm not sure if you addressed yet is the TY slug generation if slug is null/undefined. I would probably place that somewhere around the top of the list (right after fixing the iframe problem).