API request: Extend page creation to private threads + Notification API

Hi @chrscheuer I hope to start with this at the end of this week — currently reviewing the drafts fixes I wrote some weeks ago.

Now started with this

Amazing :) Quick question: I don't remember if we talked about this, but would this change include the ability for us to also append posts to existing public threads? (In addition to private threads).
Asking this since we could use that feature for when users want to add feedback to bugs (instead of having to create a new thread every time which we have to with the current upsert API for pages).

would this change include the ability for us to also append posts to existing public threads?

I think so yes. Maybe this'd almost work by default and I just need to write auto test cases, I'll have a look.

Should these posts 1) get appended at the bottom of the page,
or 2) become replies to the Original Post sorted by Like votes (best-first)?
(or 3) appear a replies to some other reply?)

It would probably be best if they got appended to the bottom of the page or became replies to the original post.

Ok. I think you'll be able to choose, by specifying Post Type.

I'm thinking when listing users by username prefix, then you want the users extId or ssoId included in the response? So you know what ext id to specify in subsequent API requests. So I'll add JSON fields for that ( + Talkyard id, username, full name).

The list-users requests, they'll be sent directly by the SoundFlow clients, without any API secret or Talkyard session cookie? Or they'll be sent via the SoundFlow server(s)?

That sounds great.
Ideally we could send the list-users requests directly from SoundFlow clients, as the roundtrip via our server would not add any value, only make it slower.
Usernames are already public in the sense that anybody can use the tag feature on the forum. We would of course do some caching on the clients so that multiple identical requests from the same client wouldn't re-fetch from the server every time.
Generally it should probably work pretty similarly to how you've designed the @ tag API now (not that I've looked at it, but just from how it seems to work, being fast, and just returning a limited number of results).

I'm using the word "tag" here where I should maybe be calling it "mention"

(Ok, thanks for the explanation. Yes I'm actually reusing the @mention internal API endpoint, — just added another /-/v0/... endpoint, + added comments in the source code about being part of the API, so won't accidentally break the API)

This is mostly done now — here's an end-to-end test you might want to have a look at; it shows how you can use the API:
https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/specs/api-private-chat-two-pps-impl.test.ts

— maybe you'll notice something I should change?

Actually now I think you don't need the extid or ssoid, but only the username, when upserting a chat message. Because you can use refs like username:someones_username to tell Talkyard which people are talking.

Here's how to use the list-users endpoint: https://github.com/debiki/talkyard/blob/375fbede4a521be2d5e3fb8f98e25dd5b45aec32/tests/e2e/utils/server.ts#L491
Here is the /-/v0/list-users API response: https://github.com/debiki/talkyard/blob/w-km7e/tests/e2e/pub-api.ts )

(Some of the links above will break in some weeks, when I reuse the w-km7e work-in-progress branch for something else)

That sounds amazing Magnus :) These are not live yet anywhere right?
Love the ref in username:blabla form, the response looks right too.
I forgot what the difference between ssoid and extid is.. Can you remind me?
The end to end test looks great as well :)

Great work!!!

This would actually allow us to post messages to existing threads as well right? With threads I mean pages that are not private chat type, but normal "questions" etc.

Yes this works for posting to existing threads too. There's another e2e test that API upserts replies to Question / Idea / Problem pages; here's the reply that gets upserted:

const michaelsReplyToMajasApiTopic = {
  extId: 'michaelsReplyToMajasApiTopic extId',
  postType: c.TestPostType.Normal,                 <—— instead of ChatMessage
  parentNr: c.BodyNr,
  pageRef: `extid:${majasApiTopicUpsData.extId}`,  <—— already existing Question/Idea/etc... topic
  authorRef: `username:michael`,
  body: 'michaelsReplyToMajasApiTopic hello Maja',
};

These are not live yet anywhere right?

That's right, this is not live yet. First I'll need to code review everything (will take a day at least).

the difference between ssoid and extid

In your case, for users, you have only ssoid, never mind extid :- )   (and sso-id it's the id you specify when calling sso-upsert-user-generate-login-secret).

And pages, posts, etc don't have any ssoid, they have only extids.

( ExtId for users: One use case: Let's say an organization imports external users into Talkyard — then Talkyard can remember their ids in the extId field. Later, maybe these users get different Single Sign-On ids via a SSO integration (that uses different ids than from the import source) — and since the ssoid is kept in a different field, this won't overwrite the extId. So if the organization later on re-imports the external users (maybe because even more external users got created somehow and they want to re-import all of them), then the external users imported the very first time, will not get duplicated (because they remember their external ids from the original import, and know they're the same users). )

Brilliant - and thanks for the reminder about extid & ssoid - I remember now.
Can't wait to play with all this. We launched 3.5 yesterday, so now is the perfect timing to start the work on forum sharing :)

Please let me know when you bring this live so I can start playing :)

Yes I'll let you know. I'm done with the first code review — but not the next step: The code review of the code review.
(I'll do tomorrow ... plus some more minor things,
I think the new version will be out on Monday or Tuesday next week.)

B.t.w. someone submitted Talkayrd to HackerNews: https://news.ycombinator.com/item?id=22702311 (in case you're curious).
(After I mentioned it a topic about remote work tools.)

Hi @chrscheuer, Now this is live. I renamed the list users endpoint to /-/v0/list-users and it returns { users: ... }. (There'll be /-/v0/list-groups too, some time later.)

( B.t.w., just FYI: I think some time later, this /-/v0/upsert-simple endpoint and the JSON format will need to change to something like /-/v0/do-action-batch and JSON like { upsertCategory: ..., editPost: ..., deletePage: ..., addChatMessage: ... } — it'll be more a "batch actions" API, than an upsert-things API. Because the endpoint name and JSON format we use right now, doesn't work so well for also deleting and editing things, plus, the notification generation code got a bit complicated. — But this can wait. )

The end-to-end test: https://github.com/debiki/talkyard/blob/master/tests/e2e/specs/api-private-chat-two-pps-impl.test.ts

Great to hear it's live!
Yea makes sense to rename it a bit later. I would recommend that you add a new endpoint first, so that we can migrate to it and test it, before you delete the old endpoint. When that time comes.
By the way, do the APIs log errors by any chance? I'm still seeing quite a few users who can't use the page creation (but since it fails, I also don't get the error, I just see that they have to manually send me the files).

add a new endpoint first

Yes

APIs log errors by any chance?

I had a quick look in the logs, didn't see anything that looks as if it's from the API or SoundFlow. I'm thinking it'd be good if I reviewed how error logging works and make log messages from your site available to you, so you can also see the logs.

Thanks, yea it would be wonderful to have some sort of log viewer for API errors :)
I'll also see how we can better catch them on our end.