Webhooks – last piece of the puzzle for full service integration

Here's an example of how Stripe displays recorded events.

And for each event it will show us details about the contents of the event as well as the success/failure of attempts of sending that event to the webhooks registered in the account.

I wonder if we could get something off the ground if we skipped some of these more thorough implementation details for now and opted for a simpler approach to get started.

Yes: I'm thinking about implementing an /-/v0/events endpoint first. It's basically only the "Record an event object in the database" step (or not even that), plus an API endpoint. And, from what I've understood, it's needed in any case, as a complement to webhooks.

Stripe has such an endpoint: https://stripe.com/docs/api/events

A discussion at HackerNews: "Give me /events, not webhooks" https://news.ycombinator.com/item?id=27823109.
Quoting the blog post: "In general, you can't rely on webhooks alone to keep two systems consistent".
The first reply in the HackerNews discussion is from someone at Stripe.

/-/v0/events is via polling. You could poll as often as you want (e.g. every second); the load would be negligible (proportional to number of Ty sites, rather than Ty sites * members). Later, we could add webhooks as well.

(Thanks for the writeup about webhooks :- )   A good list of things to think about. — There's also versioning of the webhook payload format, maybe that could be a per site and webhook setting. It's mentioned in the HackerNews discussion b.t.w.)

a simple, no retrying, no event persistence, no event admin UI implementation as a first step.

That sounds like a good starting point for webhooks. Hmm, some persistence would actually be needed, since Ty .net is a SaaS — a pranks minded person could decide to send webhooks to the wrong server, causing unexpected weird things to happen. And then the Ty server would need to remember for example that "oops, that server doesn't want my webhooks".

Here's the JSON in a webhook request: (improvement ideas are welcome)

{
  "events": [{   // a list with exactly 1 event
    "id": 1234,  // event id — currently sequential ids, maybe will change
    "when": 1648625026090,   // Unix time, millis
    "eventType": "PostCreated",
    "eventData": {  ... },  // depends on the event type
  }],
  "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
}

Complete example — if someone posts a reply:

{
  "events": [
    {
      "id": 5,
      "when": 1648625026090,
      "eventType": "PostCreated",
      "eventData": {
        "post": {
          "id": 111,
          "nr": 4,
          "author": {
            "id": 102,
            "isGroup": false,
            "isGuest": false,
            "fullName": "Memah",
            "username": "memah",
            "tinyAvatarUrl": null
          },
          "pageId": "2",
          "urlPath": "/-2#post-4",
          "parentNr": 2,
          "pageTitle": "ideaTitle",
          "isPageBody": false,
          "isPageTitle": false,
          "approvedHtmlSanitized": "<p>memahsReplyTwo</p>\n"
        }
      }
    }
  ],
  "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
}

To construct the full URL to the new post, add the origin field to the urlPath field: http://e2e-test-cid-0-0-now-2498.localhost + /-2#post-4. (Or is having to do that, annoying?)

Another example: Someone created a new topic — a PageCreated event:

{
  "events": [
    {
      "id": 2,
      "when": 1648625007053,
      "eventData": {
        "page": {
          "id": "2",
          "author": {
            "id": 102,
            "isGroup": false,
            "isGuest": false,
            "fullName": "Memah",
            "username": "memah",
            "tinyAvatarUrl": null
          },
          "posts": [   // the Original Post aka page body
            {
              "id": 108,
              "nr": 1,
              // "author" — included above already, in  page.author.
              "parentNr": null,
              "isPageBody": true,  // then, author is page author
              "isPageTitle": false,
              "approvedHtmlSanitized": "<p>ideaTextOrig</p>\n"
            }
          ],
          "title": "ideaTitle",
          "excerpt": "ideaTextOrig",
          "urlPath": "/-2/ideatitle",
          "pageType": "Idea",
          "doingStatus": null,
          "answerPostId": null,
          "closedStatus": null,
          "deletedStatus": null,
          "categoriesMainFirst": [   // currently only includes the parent
                                     // category, not any grandparent category
            {
              "id": 2,
              "name": "CategoryA",
              "urlPath": "/latest/category-a",
              "categoryId": 2
            }
          ]
        }
      },
      "eventType": "PageCreated"
    }
  ],
  "origin": "http://e2e-test-cid-0-0-now-2498.localhost"
}

Some time later, the null fields will be excluded completely instead (if they don't have any values).

A page got closed — a PageUpdated event:

{
  "events": [
    {
      "id": 5,
      "when": 1648630983752,
      "eventData": {
        "page": {
          "id": "2",
          "title": "webhBasicQuTitle",
          "author": {
            "id": 110,
            "isGroup": false,
            "isGuest": false,
            "fullName": null,
            "username": "mei",
            "tinyAvatarUrl": null
          },
          "excerpt": "webhBasicQuBodyOrig",
          "urlPath": "/-2/webhbasicqutitle",
          "pageType": "Question",
          "doingStatus": null,
          "answerPostId": 109,
          "closedStatus": "Closed",    <———
          "deletedStatus": null,
          "categoriesMainFirst": [
            {
              "id": 2,
              "name": "CategoryA",
              "urlPath": "/latest/category-a",
              "categoryId": 2
            }
          ]
        }
      },
      "eventType": "PageUpdated"   <———
    }
  ],
  "origin": "http://e2e-test-cid-0-0-now-3095.localhost"
}

Security:

Signing requests with a HMAC: Not yet implemented.

In the next version, (in one or a few days), you'll be able to add custom headers
— then you can include a Basic Auth: username-&-password so your servers will accept the incoming webhook requests. Or you could add a X-Secret-Key: .... header or something.

But for now, you'll need to send the webhooks to an un-guessable URL, like /api/webhooks-in/2578892785076892415 where the last part is a long unguessable string.

Now the next version, with extId and ssoId, is running here at Ty .io — will upgrade Prod this evening or tomorrow morning.

The next version, with extId and ssoId, is in Prod now. And the PageUpdated event works now too (bug fixed).