Feature request: Search API

@chrscheuer Now I upgraded the Ty .net server,

So now you can try out the search API / start using it. Here's an end-to-end test:
https://github.com/debiki/talkyard/blob/40ff70deb434d16f5d833ae8005158f873671637/tests/e2e/specs/api-search-full-text.test.ts

Here's the search API request:
https://github.com/debiki/talkyard/blob/40ff70deb434d16f5d833ae8005158f873671637/tests/e2e/utils/server.ts#L494

I'm getting a CORS error when trying to test this:

Access to fetch at 'https://forum.soundflow.org/-/v0/search' from origin 'http://localhost:8080' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

It's okay that I can't test from localhost, but would it be possible to white-list the soundflow.org domain for CORS requests?

What if I add a CORS domain whitelist config field in the admin area? Then the site admins can decide (i.e. you'd type soundflow.org or maybe *.soundflow.org in that field)

Yea that'd be great!

We have a very tight beta deadline again this round by the way :) Releasing a large new version on June 15, which means this coming week is the cutoff for features so we have enough time to beta test.
Do you think this and the other issue is possible to get looked at this week? If not that's just good to know, then we'll build in some workarounds for the features (search via our server etc.).

Do you think this and the other issue is possible to get looked at this week?

1) The other issue, yes. 2) This CORS issue: I think so but I'm not totally certain — looks more complicated to add per site CORS to the web framework, than what I thought.

I can post a status update tomorrow (that'd be fine? I mean, not too late)

That would be great - update tomorrow is fine. We can work around the CORS issue by sending through our own servers (even though it will make it slow for users) so would be great to get the markdown issue fixed and then see how far we can get with CORS.

I got the markdown issue fixed (not code reviewed yet).

I can add CORS headers via Nginx, I'll give this a try later today or tomorrow. (That's a better approach than using the app server for that, anyway, long term, I think.)

Cool - thanks for the update!

Hi @chrscheuer — I could repro the "Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header ... " problem and make it work by adding CORS headers (and some other changes — lots of CORS related things to read about).

These CORS requests are for /-/v0/search, right? Would you want people to do these search requests as "strangers", meaning, not logged in?

Or, if a user is logged in, would you want his/her session cookie to be included in the CORS request, so the response can include topics not publicly visible? (but visible to him/her if s/he is logged in)

(Or, not sure if / how this could work, but maybe in the distant future, using a Bearer token or Basic Auth password somehow. Seems tricky to distribute such secret things on a per user basis though, hmm)

Nice!
Yea these are for /-/v0/search. We log users in to their SF account when displaying the help panel which this is part of. But also roundtripping the forum SSO might be overkill for the search for now.
So basically yea it would be okay for the search to only display public results for now, and probably for any foreseeable future.

I don't think the setting allowing the soundflow.org domain to post requests has anything to do with this question though, right? The CORS header is about allowing Javascript hosted on the domain soundflow.org to send HTTP POST requests to the search endpoint - not about whether or not we should send session cookies - or am I misunderstanding something?

The CORS header is about allowing Javascript hosted on the domain soundflow.org to send HTTP POST requests to the search endpoint

Yes (and also GET, PUT, DELETE).

not about whether or not we should send session cookies

Those POST requests can optionally include one's session cookie. Then, there'd be CORS POST requests with the Soundflow user's Talkyard session cookie — then, Talkyard would look at the cookie and know who the user over at soundflow.org is and could include in the response access restricted topics s/he is allowed to see.

But by default CORS requests don't send cookies. I think that's a good start (i.e. to skip cookies), to see how things work out, with a bit less complexity now in the beginning, fewer things that can go wrong.

(Here's a bit about CORS and cookies — they call it "Requests with credentials": https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#Requests_with_credentials )

it would be okay for the search to only display public results for now, and probably for any foreseeable future

Ok that sounds good (simpler & safer for now :- ))

Great :) Yea makes perfect sense to take it in two separate steps. Both in terms of ease of implementation now, security aspects and the fact that we can get very far with just un-authenticated (public) search.
How do you think the timeline would look for the simple CORS case?

think the timeline would look for the simple CORS case?

I've made that work (I think) — I have in mind to do code review today, then I can deploy an early version only to this server Ty. io tomorrrow, allowing CORS requests from soundflow.org, and then you can try it out?
Maybe you / we will find something we didn't think about.

Sounds great - that would be perfect.
Our release is scheduled for June 15, which means we'll be doing videos and integration tests next week. So it's important we plan for whatever we make work now to not stop working before the release... Just so we get the admin option to add the domain added before rolling back the feature (if we get it working). Hope that makes sense.

I'm building the new server now, will upgrade early tomorrow morning it seems.

whatever we make work now to not stop working before the release

I think it's very unlikely that there's anything in this CORS stuff that needs to be rolled back

(Actually I don't completely understand this sentence: "Just so we get the admin option to add the domain added before rolling back the feature (if we get it working" maybe there's some cut and paste weirdness?)

Awesome! Thank you :)

Now the new server is running here on Ty .io, and it allows CORS from: http://localhost:8080 and https://soundflow.org.

You can do this:

1. Copy this HTML page with CORS test helper Javascript and cURL to your localhost:
   https://raw.githubusercontent.com/debiki/talkyard/master/tests/e2e/utils/ext-cors-site.html
   (it's this: https://github.com/debiki/talkyard/blob/master/tests/e2e/utils/ext-cors-site.html )

2. Start a server at 8080: ./node_modules/.bin/http-server -p8080 dir/with/that/html/page/

3. Go here: http://localhost:8080/ext-cors-site.html

4. Open dev-tools and type:

corsFetch({
    url: 'https://www.talkyard.io/-/v0/search',
    POST: { searchQuery: { freetext: 'pri' + 'sm' }},
    onDone: function(rsp) { logToPageAndConsole(rsp) }});

You can also try the cURL examples, and change: -H "Origin: http://localhost:8080" to -H "Origin: http://the.wrong.origin" to see what'll happen

It works!! Thank you so much for this quick fix. Let me know when this is ready on forum.soundflow.org :)
If there's any chance of it working tomorrow morning (we're doing a live demo with a press reporter) that would be amazing.

I just upgraded the server — you can go here: /-/admin/settings/features

and check Enable Cross-Origin Resource Sharing (CORS)

and then type, on 2 separate lines:

http://localhost:8080
https://soundflow.org

in the text box that then appears (but don't end with a slash, don't: https://soundflow.org/ )

AMAZING!! It works :)

Here it is in action in our app:

THANK YOU MAGNUS!!!

Ok :- ) Looks nice in the app I think, seems user friendly

Haha yea... One of these days when you get a Mac we have to get you on board ;)