Post creation (upsert) API

2019-10-31 16:11:13.767Z

Thank you again for all your help designing & implementing the upsert API for categories.
Our users are so happy with this feature :)

I have another request in the same general area.
We're building a feedback integration inside SoundFlow, and the idea is to allow users to report errors, bugs or to ask for feedback directly from within our app.
Since we use the forum for such feedback, the idea is that we could programmatically create new threads on behalf of users.
By doing it this way, we can ensure that their feedback includes all the relevant info we need (link to upload of compressed log files, system info, and possible state of the app etc.) - while not having to use an entirely different system for that (Talkyard is extremely good for handling feedback/support forums IMO).

Would it be possible to extend the general upsert API to include creating new threads?

Reply

32 replies

There are 32 replies. Estimated reading time: 18 minutes

KajMagnus @KajMagnus
core-dev
support-team
2019-11-01 18:23:09.896Z
Yes, & I like this idea. Something like this?:

POST /-/v0/upsert-simple

With a JSON payload like:

{ pages: [ extId: ..., pageType: PageType.Problem, categoryRef: "extid:the-category's-external-id", createdByRef: "ssoid:the-single-sign-on-id-for-the-user", title: "Error when ...", bodyHtmlUnsafe: "<p>There was this problem when ... Details:</p><pre> .... system info, app state .... </pre>", ] }
Reply
1. C Christian Scheuer @chrscheuer
  2019-11-01 18:25:47.516Z
  Yay! Yea sounds like a good start :)
  And then we'd get a response back with a URL or enough info to construct a URL (I imagine many of the same thoughts that went into the category upsert API design will also apply here).
  One great thing here is we won't have to worry about the upsertion and slug creation - this will be more simple one-off inserts 99.99% of the time.
  Reply
  KajMagnus @KajMagnus
  core-dev
  support-team
  2019-11-07 12:36:31.498Z
  Yes, what about the server returning this:
  
  { pages: [{ id: ..., ..., urlPaths: { canonical: '/path/to/page', redirects: ['/old-path-now-redirects-to-canonical-path', '/another/old'] } }] }
  
  Similar to the upsert-category request, which returns:
  
  { categories: [{ id: ..., urlPaths: { activeTopics: ..., newTopics: ..., topTopics: ..., } }] }
  
  (Sorry for the late reply — I've been working with more complete exports & imports of one's site, ... + moved to a different place here where I am now.)
  
  Reply 1 Like
  C Christian Scheuer @chrscheuer
  2019-11-12 16:02:45.958Z
  Yes this looks great :)
  
  Reply
  C Christian Scheuer @chrscheuer
  2019-11-13 23:42:02.144Z
  A related API (we should probably discuss this in a separate thread but thought I would first mention it here for context) would be that we'd like to be able to query for a user's most recent contributions.
  
  The feedback workflow we'd like for users in our app would be:
  
  Send feedback
  
  Choose between creating a new issue or adding to an existing one.
  
  Now if they choose to create a new issue, we'd need the page creation API that we're discussing here.
  If they choose to append feedback to an existing issue, we'd need a 3rd API to be able to add a post/reply to an existing page/thread.
  
  The reason why we would need this, is that sometimes issues will have been manually created in the forum, or we'd need more than one set of logs for the same issue.
  
  The option to add to an existing issue is what would require an API for us to list most recent issues that a user has contributed to or has been mentioned in. Maybe this could eventually evolve into a generic search API (I actually would probably prefer that). If it were a generic search API, then we could also use it for stuff like users searching for help with an issue, and then listing potentially matching issues. Close to what happens today inside Talkyard when creating a new post.
  
  Reply
  KajMagnus @KajMagnus
  core-dev
  support-team
  2019-11-14 13:50:31.332Z
  I like the idea of having such a find-topics-by-user / generic search API.
  
  The add-post/reply API:
  
  Would you want to reply to existing posts? Then you'd need the parent post's nr or extid: ... to refer to, in addition to the page's id.
  Or would you like to append posts at the bottom of the page? (Then no parent post nr needed)
  
  what about POST /-/v0/upsert-simple and this JSON:
  
  { posts: [{ extId: ..., pageRef: "extid:the-page's-external-id", parentNr: 123, <— if replying to a specific post createdByRef: "ssoid:the-single-sign-on-id-for-the-user", text: "<p>I forgot to mention that ... </p>", postType: PostType.Discussion or PostType.Progress }] }
  
  And a response like:
  
  { posts: [{ id: ..., urlPath: ...., nr: ..., parentNr: ..., postType: ... text: ..., }] }
  
  Reply 1 Like
  C Christian Scheuer @chrscheuer
  2019-11-23 05:01:45.771Z
  This sounds great. Can you remind me what the ssoid: would be - would this be our external ID (ie. our own customer ID)?
  I love how easy it seems to review this API given all the thorough work we did on the previous one :)
  
  Given the amount of feedback we're receiving from users right now it is a very high priority for us to get automated feedback working so here's my vote for getting the page creation API up and running soon :) IMO the generic search API (I'm starting to gravitate towards using a TY API for search instead of our own engine as we previously discussed) could be 2nd in priority, and 3rd would be post addition to existing pages.
  
  Reply
  KajMagnus @KajMagnus
  core-dev
  support-team
  2019-11-25 07:31:41.667Z
  ssoid: is the SIngle Sign-On id. You can use that, or you can use extid: (external id) — whatever works best for you. I'd guess the ssoid and extid would be the same, in your case, and maybe most / almost-all cases for other communities too.
  
  Yes I think most of the hard work has been done.
  
  here's my vote for getting the page creation API up and running soon
  
  I started looking into the iframe problem and people who got error messages when they posted comments. ... At the same time, making this page upsert API work, wouldn't take that long.
  
  Reply
  C Christian Scheuer @chrscheuer
  2019-11-25 08:12:57.310Z
  Sounds great! We were able to switch back from the iframe - there was a feature switch in Electron I hadn't noticed yet. So problem with iframe is averted for now :)
  
  Reply
  KajMagnus @KajMagnus
  core-dev
  support-team
  2019-11-27 14:35:49.761Z
  The page upsert API works now (on my localhost), ... Now I'm writing e2e tests, this took longer than what I thought (+ was away most of the day yesterday). I think I'll release a new server on Friday (sorry for the delay).
  
  Reply
  C Christian Scheuer @chrscheuer
  2019-11-27 16:31:04.464Z
  Yay, awesome!! I'm still working on the feedback integration on our end here. It's gonna be super sweet :)
  
  Reply
  KajMagnus @KajMagnus
  core-dev
  support-team
  2019-12-02 06:14:29.267Z
  Wow, this looks nice :- ) I'm thinking text people type in these different fields will get converted to HTML?
  
  Currently the Upsert API assumes the text one upserts, is formatted in HTML. Maybe there should be a way to choose CommonMark instead? There're the fields title and body, and, there could be another field bodyMarkupLang that you could set to CommonMark?
  
  Reply 1 Like
  C Christian Scheuer @chrscheuer
  2019-12-04 10:54:11.301Z
  Thank you :) Wonderful news - no worries about the delay, happy you're all fine again now!
  
  Oh yea that's a good point about the markdown.
  We have a rich text editor for SF but for simplicity I didn't turn that on in these fields yet.
  I'm fine that we have to do the conversion to HTML.
  It appears you're normally using markdownit with this setup right?
  
  markdownit({ html: true, linkify: true, breaks: true });
  
  It makes sense to me that we just use our own conversion to HTML. If we at some point switch to our own rich text editor (the one that powers our CMS and internal documentation system, based on slatejs), we would still need an HTML endpoint from TY. So it's not a big deal to not have a markdown endpoint.
  
  Reply
Progress
with doing this idea
@KajMagnus marked this topic as Planned 2019-11-11 14:11:32.497Z.
KajMagnus @KajMagnus
core-dev
support-team
2019-12-02 06:12:21.232Z
Hi Christian, now I've upgraded this server, Talkyard .io, to a version with the upsert-pages API. I'll upgrade the Talkyard .net server tomorrow morning probably.

Here's an End-to-End test that shows how you can use the usert-pages API, combined with Single Sign-On:
https://github.com/debiki/talkyard/blob/master/tests/e2e/specs/api-w-sso-upsert-pages.2browsers.test.ts

(Here's without Single Sign-On, normal password login instead:
https://github.com/debiki/talkyard/blob/master/tests/e2e/specs/api-upsert-pages.2browsers.test.ts )

(This took a bit longer than to last Friday — I was away one whole day, and then I caught a cold and was in bed. Now all fine again :- ))
Reply 1 Like
@KajMagnus marked this topic as Started 2019-12-02 06:12:31.424Z.
KajMagnus @KajMagnus
core-dev
support-team
2019-12-04 08:34:37.270Z
Now I've upgraded your server.

Let me know some time later when you have had time to try out the API :- )
Reply 1 Like
C
Christian Scheuer @chrscheuer
2019-12-04 11:06:01.831Zreplies toKajMagnus⬆:
NICE - thank you! Can't wait to try it out :)
Reply 1 Like
C
Christian Scheuer @chrscheuer
2019-12-04 12:44:26.283Zreplies tochrscheuer⬆:
Hi @KajMagnus.

I've tried it out now.
Here's the first result:
https://forum.soundflow.org/-1372/test-issue-5

I'm sending HTML to the "body" string, stuff like <h4>...</h4> but those seem to be stripped out by the API.
Any thoughts on what we should do instead?
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-04 16:32:01.905Zreplies tochrscheuer⬆:
I'll fix in the next version. This is probably me using basicWithImages (sorry) like so:

Jsoup.clean( approvedSource, Whitelist.basicWithImages)))

because now when I had a closer look, I noticed that basicWithImages doesn't allow things like <h1> <h2> and <table>.

I'll change to: Whitelist.relaxed, which allows all that. Note:

Links do not have an enforced rel=nofollow attribute, but you can add that if desired.

So probably you want to inject rel=nofollow into any links people insert?

Docs: https://jsoup.org/apidocs/org/jsoup/safety/Whitelist.html
Reply 1 Like
C
Christian Scheuer @chrscheuer
2019-12-04 17:31:42.250Zreplies toKajMagnus⬆:
So probably you want to inject rel=nofollow into any links people insert?

Good point :) We'll add a nofollow rewrite rule to the markdownit renderer to process this.

And thanks, would be great to have header support (and table too).
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-13 19:57:10.483Z
Hi Christian, probably I'll upgrade your server tomorrow.

I just upgraded this server Talkyard .io with the next version, which keeps <h1> <h2> ... etc, + debug friendly SSO login secret error messages. Have a nice weekend :- )
Reply
C
Christian Scheuer @chrscheuer
2019-12-13 20:07:31.140Zreplies toKajMagnus⬆:
Awesome! Thanks for the update :)
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-14 12:50:01.422Z
Now I've upgraded the server, ... Let me know when you've tried out the <h4> and SSO error messages :- )
Reply
C
Christian Scheuer @chrscheuer
2019-12-15 16:17:57.812Zreplies toKajMagnus⬆:
YAY!!!!!! IT WORKS :D :D :D
https://forum.soundflow.org/-1408/test-issue-reporting
Reply
C
Christian Scheuer @chrscheuer
2019-12-19 11:31:01.173Zreplies tochrscheuer⬆:
One important thing I'm noticing is that I don't get notifications from new issues created with the API.
Not sure if that's by design (I can understand the arguments why) but on the other hand right now I am missing the threads people create via the API since I'm used to using the emails as my reference for what I should check.
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-19 19:03:14.992Zreplies tochrscheuer⬆:
Hi Christian, I'll make Talkyard send notifications for new issues created with the API.

I'm thinking about if this should be the default behavior, or not? Some thoughts:

Maybe /-/upsert-simple could generate notifications by default.

Whilst other endpoints for importing a complete site dump with 9999 items, would not do that.

And/or Talkyard could look at timestamps: If the imported things have timestamps from long ago, then no notifications? And if timestamps are missing or shows the current time, then the items are new, and notifications get sent? (Or maybe this'd make API users confused)

There could be an URL parameter, like &sendNotifications=true/false? Or a sendNotifications: true/false field in the JSON payload somewhere.

Any thoughts?

Glad to hear that it works :- )
Reply
C
Christian Scheuer @chrscheuer
2019-12-19 20:16:25.039Zreplies toKajMagnus⬆:
Good thoughts...
I think having a discrete parameter to control whether to send notifications is better than "magic logic".
I'm fine if it defaults to be off - since right now the API supports inserting many items already.
I would generally think it would be weird to have to specify options via query parameters when also sending JSON payload data. It would be cleaner IMO to have it all be part of the JSON.
Maybe have an options object in the root, next to the categories and pages arrays, which could hold stuff like sendNotifications?
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-20 11:01:40.459Zreplies tochrscheuer⬆:
an options object in the root

That sounds like a good idea to me. It could be named upsertOptions so one won't confuse it with some settings or options object getting inserted into the settings database table. I'll look into this over the weekend and early next week (some days will disappear because of travelling & Christmas).
Reply 1 Like
C
Christian Scheuer @chrscheuer
2019-12-27 19:57:45.230Zreplies toKajMagnus⬆:
Sounds great :) Hope you're enjoying the holidays!
Reply
KajMagnus @KajMagnus
core-dev
support-team
2019-12-30 18:19:27.164Zreplies tochrscheuer⬆:
Thanks, yes nice Christmas holidays back in Gothenburg ... I was away for quite a while.

On Wednesday, most likely I'll upgrade your server to the next version, with upsertOptions.sendNotifications. POST this json:

{ upsertOptions: { sendNotifications: true }, pages: [pageToUpsert], }

Here's an end-to-end test with details: (I think the above info should be enough though)
https://github.com/debiki/talkyard/blob/177188d19a87c2d0abdd9f6e4e8a3915472cc020/tests/e2e/specs/api-upsert-page-notfs.2browsers.test.ts
Reply
C
Christian Scheuer @chrscheuer
2019-12-30 20:51:36.640Zreplies toKajMagnus⬆:
Wonderful, that looks great :) Thank you Magnus! I'll wait updating our code until you've launched the update just to be sure.
Reply
KajMagnus @KajMagnus
core-dev
support-team
2020-01-01 22:59:45.192Zreplies tochrscheuer⬆:
Now I've upgraded your server to the new version
Reply
C
Christian Scheuer @chrscheuer
2020-01-01 23:49:49.298Zreplies toKajMagnus⬆:
Awesome, thank you!
Reply

Reply (discussion)Add progress note