How to manage the posted attachments in my own talkyard server?
I have a talkyard environment in my own server.
Now, the Disk usage is too high , so i need to delete some attachments to free the disk space, but I dont konw how to do that.
Anyone can help me ?
- KajMagnus @KajMagnus2021-08-02 04:24:28.816Z2021-08-02 04:31:17.269Z
How does the OS file system look? Is there one single partition, or is maybe
/opt/
or/var/
on separate partitions?The attachments, called "uploaded files" or "uploads" in Talkyard, are stored in the file system.
- The ones in use, are at
/opt/talkyard/data/uploads/
- Backups of uploads can take lots of space. They are here:
/opt/talkyard-backups/archives/*-uploads-up-to-incl-*
, you can see the upload backup directories if you do this:
ls -1d /opt/talkyard-backups/archives/*-uploads-up-to-incl-*
You can check the sizes of the uploads in use, and the backups, like so:
# view upload dir sizes: du -hs /opt/talkyard/data/uploads/ # the ones in use du -hs /opt/talkyard-backups/archives/*-uploads-up-to-incl-* # backups
Here's a nice way to find out more exactly which files are large:
apt install ncdu ncdu /opt/talkyard/data/uploads/public/
You can click the arrows Up and Down on the keyboard, and Enter, to navigate between the directories. The one at the top with the most
#####
is the directory with the largest size.The directories are numbers from 0 to 9 — and the smallest files are stored in 0, and the larger the file, the larger the directory number. If you want to delete all really large uploads, you could simply delete, say, directories 6, 7, 8, 9. But first you might want to have a look and see what's inside.
- The ones in use, are at
- In reply toliliming1976⬆:KajMagnus @KajMagnus2021-08-02 04:37:22.082Z
Are you sure it's the uploads that use lots of disk space? Can I ask, what types of files do you upload? (maybe large images or videos?)
In any case, what if you copy-paste the output from this:
du -hs /opt/talkyard/data/* du -hs /var/log/ du -hs /var/log/nginx/* du -hs /var/log/talkyard/* du -hs /opt/talkyard-backups/archives/*-uploads-up-to-incl-* du -hs /opt/talkyard-backups/archives/*-daily-* | tail -n15
You could send it in a private message if you want, or maybe edit the output and remove the hostname from the backup file names (if you want to).
- Lliliming @liliming1976
Thanks for ur helps.
My OS is Centos7.0 , and Talkyard server installed by docker container.
Actually, I just want to know if there is a web administration page for upload files?
Or, I can only manage the upload files by command line?- KajMagnus @KajMagnus2021-08-02 07:29:00.689Z
For now, you'll need to use the command line. The sample commands help you find the large files, although their names are in hex
- In reply toliliming1976⬆:KajMagnus @KajMagnus2021-08-02 05:00:46.708Z
Do you think that after a while, people will have uploaded lots of files again, so that the disk usage is too high again?
Then, if it's not desirable to increase the disk space, maybe it could be helpful if Talkyard could automatically delete uploaded files larger than, say, 10 MB, after two weeks, or something like that. And to exclude such files from backups.
And if there was an info message, if uploading such large files, telling the user that they're "too large" and will get auto deleted after X days to free disk space.Also, I suppose it'd be helpful with a list of uploaded files, in the Admin Area, where you could sort by file size, and see the original file names, and delete files.
- Lliliming @liliming1976
I could not find any page about uploaded files list in Admin Area.
- KajMagnus @KajMagnus2021-08-02 07:27:12.144Z
Right now there is none (I was thinking about creating one)
- Lliliming @liliming1976
Got, thank you very much.
Look forward to see this function.- KajMagnus @KajMagnus2021-08-11 13:35:51.477Z
Ok, how short of disk are you? Using the Linux terminal, until I've implemented this, how complicated / frustrating would that be on a scale from 0 to 10?
- Lliliming @liliming1976
Problem resolved. Now the disk space is enough, you can do it in your plan.
Thanks.- KajMagnus @KajMagnus2021-08-13 13:14:15.903Z
Ok, thanks for the update
- In reply toKajMagnus⬆:Lliliming @liliming1976
Sadly,I know nothing about coding in Talkyard.
I just need an Existing management page to help me do that. - In reply toKajMagnus⬆:@Mr.Nobody
it could be helpful if Talkyard could automatically delete uploaded files larger than, say, 10 MB, after two weeks, or something like that. And to exclude such files from backups.
I really like this idea! Also would be nice to auto-delete attachments that are not being used in any post anymore (e.G. if someone uploads, then delete the topic, is the attachment deleted with it?)
Also, I suppose it'd be helpful with a list of uploaded files, in the Admin Area, where you could sort by file size, and see the original file names, and delete files.
If this is ever added, please add the topic/comment's age to that file browser so that it is easier to delete attachments from:
- content older than X days;
- content without activity (replies and/or likes) in more than X days;
I would not want to delete attachments from content that is being used in important content.
- KajMagnus @KajMagnus2021-08-19 14:37:25.731Z
auto-delete attachments that are not being used in any post anymore
Yes, this is how it's going to work (some time later). There'll be some "timeout" — e.g. if one deletes a page with an image, then, if one doesn't undelete the page within, say, 1 month, the image gets deleted. (This "timeout" could be configurable, per site.)
And one will be able to delete uploaded files from disk, and Talkyard will notice (after a while).
please add the topic/comment's age
Good point. I had in mind to include the original name of the uploaded file, but it didn't occur to me that the page / reply where it occurs, is helpful as well. So, the page title, and the first words / first sentence in the post (Orig Post or reply) where it appears, too.
And number of likes in total, and the last X days, that's a good idea too. And number of page views maybe
I would not want to delete attachments from content that is being used
I think Imgur deletes images that no one looked at the last 6 months — there could be some admin-toggle-on-off functionality like that in Talkyard.
- Progresswith handling this problem