No internet connection
  1. Home
  2. Support

Export from hosted version to self-hosted version?

By Steve Mitchell @SteveM
    2019-10-28 16:14:28.619Z

    I'm considering installing and hosting Talkyard on my own server so I can have a more user-friendly email when people interact with Talkyard. I'm using the hosted version now, and would want to export all of my current data so I could move it to a locally hosted install, but I can't seem to find that in the admin section anywhere...

    Is there a way to do this?

    Reply
    • 21 replies

    There are 21 replies. Estimated reading time: 11 minutes

    1. KajMagnus @KajMagnus2019-10-28 23:20:11.348Z

      Almost:

      1) You can export JSON for your site here: https://comments-for-seabits-com.talkyard.net/-/export-site-json (if you're logged in as admin).

      And you can 2) import JSON (similar to what I did, when I imported the Disqus comments).

      However, I have not yet 3) made the export and import format identical — so right now you cannot import the JSON you export.

      I've been planning to fix this, for many months (so that one can import & re-import JSON to another server, e.g. self hosted). And you asking about this, gives me motivation to get started :- )

      (I was previously a bit short of time, and didn't keep the exp & imp format in sync. Now I have in mind to add an end-to-end test that exports JSON for a site, and imports the same JSON to a new site, and checks that all seems fine after the import — so all this will get tested automatically, thereafter.)

      Reply
      1. Progress
        with doing this idea
      2. @KajMagnus marked this topic as Started 2019-11-01 18:25:23.425Z.
      3. KajMagnus @KajMagnus2019-11-11 07:48:46.336Z

        Now I've made export-&-import mostly work — however, only textual contents for now; uploaded images not yet included in the export file.

        I think this'll cover like 99% of what one wants, for an embedded comments site, like yours. (Because then people typically reply with text comments, and don't post any images.)

        So, after I'm done with code review, and have released a new Talkyard version, you'll be able to export your site from Talkyard .net, and import your site to a self hosted server.

        Reply
        1. H
          @Happyfeet01
            2020-01-19 22:37:58.177Zreplies toKajMagnus⬆:

            How can I import?

            Reply
            1. KajMagnus @KajMagnus2020-01-19 23:06:52.859Zreplies toHappyfeet01⬆:

              Edit: I think you've done the first step already — when you went to /-/export-site-json. So you can do as below — at your new server, click Restore Backup. /Edit

              First, at your old Talyard server (the one you're migrating away from),
              go here: https://comments-for-your-blog.talkyard.net/-/admin/settings/features, and enable Experimental features:

              Experimental    [ ]   Enables complicated and less well tested features, like custom HTML pages.

              Then click Save. A new tab appears, with the title Backups. Go there and click Download Backup.

              At your new Talkyard server, do the same things, but click Restore Backup.

              Reply1 Like
              1. H
                @Happyfeet01
                  2020-01-20 05:04:56.744Z2020-01-20 05:14:23.042Zreplies toKajMagnus⬆:

                  Confused, Why is it wrong?

                  Invalid UserStats json at index 1 in the 'ppStats' list: Bad json for UserStats for user id -42: 'topicsNewSince' looks like a unix time in seconds, should be milliseconds [EsE7UMKW2], json: {"lastSeenAt":1576506196488,"numLikesGiven":1,"topicsNewSince":0,"firstChatMessageAt":null,"numSolutionsProvided":0,"numDaysVisited":0,"numChatTopicsRepliedIn":0,"notfsNewSinceId":0,"numLikesReceived":0,"firstSeenAt":1576506196186,"numChatMessagesPosted":0,"numDiscourseTopicsEntered":0,"numChatTopicsEntered":0,"numDiscourseTopicsCreated":0,"lastPostedAt":1576506196488,"tourTipsSeen":[],"firstNewTopicAt":null,"emailBounceSum":0,"numDiscourseTopicsRepliedIn":0,"nextSummaryEmailAt":null,"lastSummaryEmailAt":null,"numSecondsReading":0,"numDiscourseRepliesRead":0,"userId":-42,"firstDiscourseReplyAt":1576506196488,"lastEmailedAt":1576513196048,"numChatTopicsCreated":0,"numChatMessagesRead":0,"numDiscourseRepliesPosted":1} [TyE76K0RKD2]

                  Reply
                  1. KajMagnus @KajMagnus2020-01-20 05:56:41.786Zreplies toHappyfeet01⬆:

                    It's a bug — a runtime assertion I've added, and it's too restrictive. Shall I send you a private message with a JSON dump that doesn't have that problem? (because probably I won't fix the bug and deploy a new version, until later this week or next week)

                    Reply
                    1. H
                      @Happyfeet01
                        2020-01-20 05:58:53.837Zreplies toKajMagnus⬆:

                        yes sure..

                        Reply
                        1. KajMagnus @KajMagnus2020-01-20 07:15:05.744Zreplies toHappyfeet01⬆:

                          Ok, I'll do. Right now I'm just adding a bit more end-to-end test checks, so I'll be more certain that there won't be other errors.

                          (Previously I didn't include Not-logged-in / Guest users, when end-to-end test importing a site dump, and therefore I didn't find this bug.)

                          Reply
                          1. KajMagnus @KajMagnus2020-01-20 09:29:19.118Zreplies toHappyfeet01⬆:

                            Sorry this takes a bit longer than what I thought. I'll reply later today (or tomorrow). (I'm in Europe, CET, ... what about you? Oh, CET you too obiously, Germany?)

                            The address to your server is https://comments.dasnetzundich.de/ right? (your new Hetzner server)

                            Reply
                            1. H
                              @Happyfeet01
                                2020-01-20 09:43:28.375Zreplies toKajMagnus⬆:

                                Yes and Moin, Moin

                                i also located in Germany.
                                Yes https://comments.dasnetzundich.de is the Talkyard Server

                                Reply
                                1. KajMagnus @KajMagnus2020-01-21 11:24:45.514Zreplies toHappyfeet01⬆:

                                  Ok, Moin Moin you too

                                  I discovered some things I haven't yet finished. I'll release a new server version this weekend, and then your server should auto upgrade and thereafter you can import the comments.

                                  (I've mostly fixed those unfinished things now, and ... Code review and testing will take some time, plus some more things I'd like to include too in the new version.)

                                  Sorry for the delay / extra time this takes

                                  Reply
                                  1. H
                                    @Happyfeet01
                                      2020-01-21 11:33:38.847Z2020-01-21 12:16:45.469Zreplies toKajMagnus⬆:

                                      Oh that’s nice 😊 I wait for it

                                      Reply
                                      1. KajMagnus @KajMagnus2020-01-26 09:08:07.050Zreplies toHappyfeet01⬆:

                                        Hi @Happyfeet01 , this is getting even more delayed,

                                        ... Now I'm done with all coding, ... and instead fixing a bug I accidentally added, and then there's code review, and then I'll want to have the server running here at Talkyard .io for some day before releasing more widely.

                                        Sorry about the even-more-delay.

                                        (This took longer e.g. because it turned out I had to add an UX config option (unrelated to this topic), and do some changes to the UX.)

                                        Reply1 Like
                                        1. H
                                          @Happyfeet01
                                            2020-01-26 09:24:14.098Zreplies toKajMagnus⬆:

                                            Okay. I can wait. Is still hostet von talkyard.io

                                            Reply
                                            1. KajMagnus @KajMagnus2020-01-26 10:00:01.764Zreplies toHappyfeet01⬆:

                                              Ok :- )

                                              Reply
                                              1. KajMagnus @KajMagnus2020-01-31 17:57:49.657Zreplies toHappyfeet01⬆:

                                                Status update: Probably a new server will be available for you at the end of next week.

                                                (I should have fixed the problems with importing now, and ... there were some a bit urgent othe things i need to include in the same release, ... and code reviewing & fixing them took loner than what I thought)

                                                Reply1 Like
                                                1. KajMagnus @KajMagnus2020-02-08 10:43:43.144Zreplies toHappyfeet01⬆:

                                                  Still not ready: I just tested importing to a self hosted server, and found 2 more bugs to fix ...

                                                  .. Now I've fixed them (earlier today), and now importing to self-hosted works — the fixes will be in the upcoming version, not yet released.

                                                  Reply1 Like
                                                  1. KajMagnus @KajMagnus2020-02-16 18:05:25.620Zreplies toHappyfeet01⬆:

                                                    Sorry I didn't bump the open source version yet. Turns out I added a bit annoying blog comments UX bugs (related to drafts & previews) in the latest version (which is only here at Ty .io and Ty .net).

                                                    Bugs now fixed (but auto tests & code review left to do), I think I can release a version that works for you in the 2nd half of next week.

                                                    Reply1 Like
                                                    1. H
                                                      @Happyfeet01
                                                        2020-03-01 17:19:35.987Zreplies toKajMagnus⬆:

                                                        Any news about this?

                                                        Reply
                                                        1. KajMagnus @KajMagnus2020-03-01 18:59:32.810Zreplies toHappyfeet01⬆:

                                                          Yes, I released a new version just recently. Your server should auto upgrade itself, this night at about 04:10 (your server's local time I think).

                                                          (Thanks for asking — somewhat likely, I would have forgotten to do this today otherwise.)

                                                          There're new and better backup scripts — they backup also the Docker-Compose config file, Nginx config and LetsEncrypt certs, and use less disk, for backing up images and uploads.
                                                          Maybe you'd like to use these new backup scripts?

                                                          This should work, for upgrading to the new scripts:

                                                          sudo -i
cd /opt/talkyard
git fetch origin
git merge --ff-only origin/master

                                                          However (!) this might cause Git edit conflicts. Your blog, https://dasnetzundich.de/, looks like a technical blog? How do you feel about resolving possible Git edit conflicts? An alternative is to just copy-paste the new backup scripts one at a time; they're just 3 files.

                                                          ***

                                                          Anyway, once your server has upgraded itself (or if you upgrade it now directly: cd /opt/talkyard ; scripts/upgrade-if-needed.sh),
                                                          you can go to https://comments-for-dasnetzundich-de.talkyard.net/-/admin/backup and click the Download Backup button,
                                                          and then click Import, on the self hosted server.

                                                          Reply
                                                          1. H
                                                            @Happyfeet01
                                                              2020-03-01 19:58:17.353Zreplies toKajMagnus⬆:

                                                              Yes my this is an technical blog.

                                                              I try to update now and write if there are any conflicts.

                                                              Reply
                                                              Reply (discussion)Add progress note