No internet connection
  1. Home
  2. Support

URGENT: Talkyard production server search broken (!!)

By Christian Scheuer @chrscheuer
    2022-04-14 16:26:41.368Z

    Hi @KajMagnus,

    I'm seeing reports that searches on our forum has started to fail:

    Reply

    Linked from:

    1. Users seeing HTTP 500 failures with java.lang.NullPointerException
    • 11 replies
    1. C
      Christian Scheuer @chrscheuer
        2022-04-14 16:27:31.665Z

        This was somebody just searching on the forum in a browser (Safari), so not related to the API.

        Reply
        1. CChristian Scheuer @chrscheuer
            2022-04-14 16:28:22.063Z

            Video from our beta tester:

            Reply
            1. CChristian Scheuer @chrscheuer
                2022-04-14 17:56:09.565Z

                We're receiving very wide spread reports from users now - many users are seeing this issue now

                Reply
                1. CChristian Scheuer @chrscheuer
                    2022-04-14 18:07:55.935Z

                    @KajMagnus this is really getting critical. How do we get a hold of you when things stop working? I've tried sending an email.

                    If we can't get this fixed asap we'll have to start looking at self hosting this. We can't have a service down for several hours.

                    Reply
                    1. KajMagnus @KajMagnus2022-04-14 21:35:36.928Z

                      I'll have a look at the search problem.

                      Maybe there could be a Telegram chat

                      Reply
                      1. CChristian Scheuer @chrscheuer
                          2022-04-14 21:38:28.798Z

                          We had to implement a workaround so I used the HTML/CSS engine to insert a Google-provided search box instead now, so you might have difficulty testing this on forum.soundflow.org at the moment

                          Reply
                          1. KajMagnus @KajMagnus2022-04-14 21:45:03.494Z

                            Wow, that was a fast workaround. Sorry for the troubles.

                            The problem was, I think, caused by me increasing the search rate limits, which I did earlier today — and this I can test on localhost. (Didn't realize that that could cause this problem. If that's the problem)

                            Reply
                            1. CChristian Scheuer @chrscheuer
                                2022-04-14 21:47:28.951Z

                                5 hours is a long downtime for any service, so I had to get creative.

                                Yea my guess is also that it should be related to your recent rate limit changes.

                                Reply
                                1. KajMagnus @KajMagnus2022-04-14 22:21:51.762Z

                                  Ok. I tried on localhost, and could reproduce it (same error & stack trace) — and the problem disappeared, on localhost, when I removed the new rate limit setting.

                                  I've removed the setting in Prod now too, so I think search works again.
                                  Still, maybe better to leave things as is until tomorrow when we're rested?

                                  I'll reply to the other topic, I mean, topics, tomorrow ( = Firiday), & I'll fix the bug and release a new version ... should be in Prod tomorrow or the day after.

                                  Reply1 Like
                                  1. CChristian Scheuer @chrscheuer
                                      2022-04-14 22:23:31.166Z

                                      Thank you! Yes now that we have a workaround, waiting for the fix to be better tested is preferable.

                                      Reply
                    2. In reply tochrscheuer⬆:
                      KajMagnus @KajMagnus2022-04-18 12:01:10.385Z

                      Fixed now, marked as Done. ( & added auto tests)

                      Reply1 Like
                      1. Progress
                        with handling this problem
                      2. @KajMagnus marked this topic as Done 2022-04-18 02:24:09.639Z.
                      Reply (discussion)Add progress note