No internet connection
  1. Home
  2. Ideas

API: Being able to fetch Topics based on answered state

By Julian Wolf @jwolf
    2021-02-04 11:46:05.522Z

    Hi

    I would really like this feature, I think it can be really useful to other people as well.

    Background to explain the use case:
    We have a slack workspace filled with students and would like to make a bot that can notify
    students each morning of unanswered questions so that they can start engaging with each other
    in helping each other with their questions, queries, etc.

    Perhaps we would need this to be based on categories as well... as an optional parameter, to make sure the right topics get sent out to them?

    Solved in post #14, click to view
    • 24 replies

    There are 24 replies. Estimated reading time: 20 minutes

    1. How nice this sounds :- )

      Yes categories seems like a good idea too. Maybe upvotes too (if many students have the same question).

      I'll have a look at this during the weekend and/or next week (together with some other API improvements).

      1. JJulian Wolf @jwolf
          2021-02-04 21:20:04.859Z

          Just thought of something as well.
          The topic Type, so if its a Question or problem?

          1. KajMagnus @KajMagnus2021-02-08 22:48:39.596Z2021-02-08 23:37:51.509Z

            Yes, ... And: (not now but later) tags too, and if assigned to anyone or not.

            Starting with this now b.t.w. (most of this is actually writing tests, not to do the actual change :- ))

        • In reply tojwolf:

          Hi Julian, Short message: Maybe the filter will look like: filter: { waiting: true, solved: false }.

          Because, details:

          I think maybe it can be good with, in addition to Solved = true/false,
          have question status Waiting = true/false,
          and status Pending = true/false, too.

          Consider this: A student asks a question. Then that question is solved == false, and the student is waiting (for help) == true. Then someone tries to help, but needs more info, and replies "To help you I need to know what's the fuzz bazz?".

          Thereafter, until the student has provided more info, it's less interesting for the ones trying to help, to see that topic listed among the other unsolved topics.

          The topic is still solved == false, but now the student is not waiting — waiting (for a reply) == false. Instead, the topic is pending more information from the student.

          Then, the student replies with more info. And thereafter, the student is waiting == true, and the topic is pending more info == false.

          ***

          A topic can be both waiting and pending, at the same time. Say, 1) someone replied "We need more info: ...", thereafter, pending == true. Then the student replies "What is that, can you explain?" — now the student is waiting == true whilst the topic remails pending == true. And the other students get to see it, among the unsolved & waiting topics, and can reply to the "please clarify" question.

          Or 2) a topic can be pending (something external to happen) == true, and the student asks for a status update — then, the student is waiting == true.

          ***

          So I think, for future compatibility, you can specify this topic filter: filter: { waiting: true, solved: false }.
          (I'll provide more details at the end of the week.)
          solved: false will start working in the upcoming Ty version
          waiting: true hasn't been implemented yet though.

          Some time later, Talkyard will get somewhat good at knowing who's supposed to reply / do-something next,
          and can include / exclude topics properly, based on if the question asker is waiting, or if it's his/her turn to reply.
          And if the filter already includes waiting: true this'll start working "automatically" some day without you having to edit & redeploy the Slack bot.

          1. JJulian Wolf @jwolf
              2021-02-09 10:32:46.598Z

              This is fantastic! Thank you so very much;
              Will this apply automatically to the instance of talkyard we are using?
              When it is done and implemented

              1. Yes. Initially, only solved: false will have any effect. And it'll probably take some month(s) until waiting: true has any effect. (So, initially, you'll see some topics where it's actually not your turn to reply — but the question asker's turn to reply)

            • In reply tojwolf:

              Hi Julian, status update: Now I just need to write e2e tests, code review & release a new version.

              I spent some days last week looking at how various other software have built their APIs (e.g. GraphQL, Dgraph, MongoDB, Clickup) — just so as to not do anything weird with Talkyard's API now when adding more features.
              And all seems fine ... So it took like 1 hour to write the actual code, and about 3 - 4 days to think about the API :- )
              not saying Ty's API is good / perfect, just that it's better than what it would otherwise have been o. O

              I guess this new filter feature will be available at the end of the week — there're [some other things also included in the new Ty version] that also need code review.

              1. JJulian Wolf @jwolf
                  2021-02-15 11:13:33.469Z

                  I completely understand, considering what I have to go through with the software I write.
                  I am having issues with the API key, will make another thread related to that at a later stage

                • In reply tojwolf:

                  Status update: Code review done, e2e tests done, building a new server today, to try out here at Ty .io for a day or two.

                  Will most likely deploy to Prod on Friday the latest. (Then you can use this.)

                  (Sorry this took a bit longer than what I thought, partly because of some private life things happening in between.)

                  1. JJulian Wolf @jwolf
                      2021-02-23 10:33:42.219Z

                      Not a problem at all!
                      Thank you soo much for this feature :)
                      Next is for me to tackle this and make it work

                    • In reply tojwolf:
                      KajMagnus @KajMagnus2021-02-26 09:18:21.510Z2021-02-26 09:33:50.899Z

                      @jwolf The new API is available (Ty v0.2021.07). Here is how you can try it out:

                      For a Bot:

                      Follow the For a Browser steps 1, 2 and 3 below to enable API requests and allow CORS (Cross-Origin Resource Sharing) from: http://localhost:8080 (or some other origin, maybe https://your-bot-name.example.com?).

                      Then you can try this cURL request: (note: edit the server addr on the last line. And remove xx_ if you want to search in a specific category only.)

                      curl  -X POST  -H "Origin: http://localhost:8080"  -H 'Content-Type: application/json'  -d '{
                          "listQuery":{
                            "listWhat": "Pages",
                            "xx_lookWhere": { "inCategories": ["extid:specificCatExtId"] },
                            "filter": { "isAuthorWaiting": true, "isOpen": true }
                          },
                          "pretty":true
                          }' \
                          https://your talkyard site/-/v0/list    # change this
                      

                      (The Talkyard server wants a CORS Origin HTTP header with an allowed origin, for bots too, not only for browsers.)

                      For a Browser:

                      First enable Cross Origin Resource Sharing (CORS):

                      1. Go here: /-/admin/settings/features.

                      2. Enable the API, and check Enable Cross-Origin Resource Sharing (CORS).

                      3. Type, on 2 separate lines:

                        http://localhost:8080
                        https://  your domain .example.com
                        

                      Thereafter:

                      1. Copy to your laptop this HTML page with CORS test helper scripts and cURL example:
                        https://raw.githubusercontent.com/debiki/talkyard/master/tests/e2e/utils/ext-cors-site.html
                        (it's this: https://github.com/debiki/talkyard/blob/master/tests/e2e/utils/ext-cors-site.html )

                      2. Start a server at 8080: ./node_modules/.bin/http-server -p8080 dir/with/that/html/page/, if you use Nodejs and have installed package http-server.

                      3. Go here: http://localhost:8080/ext-cors-site.html  (note: we enabled CORS from http://localhost:8080, above)

                      4. Open Dev Tools and type:

                        corsFetch({
                            url: 'https://  you  .talkyard.net/-/v0/list',
                            POST: {
                              listQuery: {
                                listWhat: 'Pages',
                                xx_lookWhere: { inCategories: ['extid:some_cat_id'] },   // optional, remove xx_
                                filter: {
                                  isAuthorWaiting: true,  // will work in the future
                                  isOpen: true            // works today
                                }
                              }
                            },
                            onDone: function(resp) { logToPageAndConsole(resp) }});
                        

                      You can also try the cURL examples (on ext-cors-site.html) and change: -H "Origin: http://localhost:8080" to -H "Origin: http://the.wrong.origin" to see what'll happen.

                      The API

                      Here's the API definition, in Typescript: pub-api.ts as of 2021-02-26 — the "List Query" stuff.

                      Here's an end-to-end test (the whole file) that uses this new API: api-list-query-for-topics-recent-etc-first.test.ts as of 2021-02-26

                      Reply1 LikeSolution
                      1. JJulian Wolf @jwolf
                          2021-02-26 09:52:59.012Z

                          AWESOME! Thank you soo much,

                        • In reply tojwolf:

                          @jwolf sorry I forgot, your site is access restricted. So you'll need an API secret in a Basic Auth HTTP header.

                          I'll write a bit about that shortly — until then, if you want to get started coding on the bot, you can use https://www.talkyard.io instead as API endpoint and http://localhost:8080 as origin:

                          curl  -X POST  -H "Origin: http://localhost:8080"  -H 'Content-Type: application/json'  -d '{
                              "listQuery":{
                                "listWhat": "Pages",
                                "xx_lookWhere": { "inCategories": ["extid:specificCatExtId"] },
                                "filter": { "isAuthorWaiting": true, "isOpen": true }
                              },
                              "pretty":true
                              }' \
                              https://www.talkyard.io/-/v0/list
                          
                          1. JJulian Wolf @jwolf
                              2021-02-26 09:53:49.800Z

                              I was just about to ask about the Auth :)

                              1. Hi again, here's how to authenticate the bot:

                                1. Enable the API, at: /-/admin/settings/features

                                2. Then go to the API tab,/-/admin/api which should now have appeared.

                                3. Generate an API secret; click Show.

                                4. Copy the curl --user tyid=2:1112223...abbbcccdd text from the cURL sample command.

                                5. Paste into the curl -X POST .... /-/v0/list request, like so:

                                  curl  --user tyid=2:111222333444555aaabbbcccdd  -X POST  -H "Origin: http://localhost:8080"  -H 'Content-ype: application/json'  -d '{
                                      "listQuery":{
                                        "listWhat": "Pages",
                                        "xx_lookWhere":{"inCategories":["extid:specificCatExtId"]},
                                        "filter": {"isAuthorWaiting": true, "isOpen": true}
                                      },
                                      "pretty":true
                                      }'   http://your talkyard site/-/v0/list
                                  

                                  That'll include an Authorization: Basic BBAASSEE64 header (with BBAASSEE64 being tyid=2:111222333444555aaabbbcccdd base 64 encoded).

                                Now you should be able to list access restricted things. (Let me know if it works for you?)

                                ***

                                (But what does tyid=2:1112223...bcccdd mean? tyid=2 means Talkyard user ID 2, which is Sysbot, a built-in admin user. And 1112223...bcccdd is Sysbot's API secret — the one you just generated.

                                1. JJulian Wolf @jwolf
                                    2021-02-26 11:17:06.850Z2021-02-26 11:33:19.101Z

                                    AWESOME! Is there perhaps an Authorization: Basic ... problem?
                                    It causes my curl to fail if I try it that way
                                    Oh wait scratch that, I saw now the base64 encoded part of the message, let me try that

                                    1. In reply toKajMagnus:
                                      JJulian Wolf @jwolf
                                        2021-02-26 11:36:54.878Z

                                        Ok, different issue on my side,
                                        I am trying "inCategories: ["extid:7"] Is this the correct way of doing it?

                                        1. KajMagnus @KajMagnus2021-02-26 13:26:08.285Z2021-02-26 20:07:31.375Z

                                          Oh I could have mentioned that:

                                          The external ID is something you choose — go to the category, click Edit category, and then in the bottom of the dialog, there's a setting External ID (optional): None, where you can type an ID of your choice.

                                          Then, use that ID like so: inCategories: ["extid:your_categorys_ext_id"].

                                          (Hmm maybe it could be called "Custom ID" instead)

                                          So, if you "accidentally" rename a category, the bot won't break — it cares only about the external ID.

                                          (Maybe it'd be nice if catslug:category-slug was supported too, if one was reasonably certain one would't rename the category slugs.)

                                          (There're Talkyard internal IDs too, not outwardly visible — just in case you wonder.)

                                          1. JJulian Wolf @jwolf
                                              2021-03-01 09:45:05.660Z

                                              Hi @KajMagnus

                                              I think I may experience some newbie problems regarding the api changes.
                                              I have it set up currently as

                                                  const payload = {
                                                      url: `${baseUrl}/-/v0/list`,
                                                      method: 'post',
                                                      data: {
                                                          listQuery: {
                                                              listWhat: "Pages",
                                                              lookWhere: { inCategories: [category] },
                                                              filter: { isOpen: true }
                                                          }
                                                      },
                                                      headers: {
                                                          'Content-Type': 'application/json',
                                                          Authorization: `Basic ${token}`
                                                      }
                                                  }
                                              

                                              The issue comes that isOpen: true does not seem to return anything :/ even though I have some posts that are not solved? I have perhaps misunderstood how the isOpen works?

                                              1. (Hi Julian, I'll have a look later today or tomorrow morning)

                                                1. @jwolf I wonder if there's a problem with the category ext id, i.e.: lookWhere: { inCategories: [category] }.

                                                  If you change lookWhere to zzlookWhere, then what happens? (Then, the field will have the wrong name, causing Talkyard to ignore it — and will fetch topics from all categories.)

                                                  What if I connect to your forum as super admin and try out the API request you included above?

                                                  1. JJulian Wolf @jwolf
                                                      2021-03-02 09:36:50.156Z

                                                      If I ignore the "lookWhere" property, I only get posts where no one has commented on it. It seems

                                                      1. Ok thanks for trying that out. So there was a bug, I've fixed it now on my localhost (see the private message), sorry for the temporary troubles

                                                        1. @jwolf — should work now (new Ty version deployed) if you include an ActiveFirst sort order:

                                                          {
                                                                      listQuery: {
                                                                          listWhat: "Pages",
                                                                          lookWhere: { inCategories: [category] },
                                                                          filter: { isOpen: true },
                                                                          sortOrder: 'ActiveFirst',   // <——— look
                                                                      }
                                                                  },
                                                          

                                                          (There's also PopularFirst and NewestFirst — all this corresponds to the buttons "(Recently) Active (topics) first", "Pouplar" and "New" in the forum topic list.)

                                      • Progress
                                        with doing this idea
                                      • @KajMagnus marked this topic as Planned 2021-02-04 15:24:02.293Z.
                                      • @KajMagnus marked this topic as Started 2021-02-08 22:48:44.057Z.