Please find logs of CLI but unfortunately I did a mistake and only informational messages only not protocol communication. Maybe it will help still but if needed I can rerun the Compact again tomorrow.
Unfortunatelly the file do not contain any CLI log lines (but only JOBLOG ones). The CLI log lines have CLI:<session_id> and in order to have all details you have to set the CLI service log level to Protocol Communication before connecting to CLI for executing the compact operation.
Sorry for this.
BR,
Ioan
PS: just to let you know that the latest 10.2.2 patch is 10.2.3.8 (details available here)
Also for the new update, Yes, I understand that there is new update for some fixes but we keep this one on hold as we expect last quarter last year that the new X4 will come out and we will shift this server to a new servers / machines.
Hope this log will be helpful to trouble shoot my issue, I cannot run this server properly, under stress and monitoring as my free disk space is under 2%.
I’ve checked the CLI logs and nothing strange found there.
Could you please also share the output from the following CLI command:
<domain#> show disposablemetadatainformation
On the other side I failed to understand is why you have configured more than 1TB for your messages (10 message storages x 128 files x 1GB) if you do not have enough space available.
For the question why more than 1TB containers.
This is due to the troubleshooting method I did, trying to see what is going on. And trying to force axigen to expand and create new containers hoping when It makes compact it will recreate some hsf to other container.
My original containers are 7 only. Created 8 and 9 later on when I encounter this issue.
Me neither cannot find issue why Axigen acted like this.
A few years ago (2018) I also had this problem, No recommendations helped (e.g. commands FINDINVALIDMSG purge, SHOW DisposableMetadataInformation, COMPACT All forced, etc.),
I described this problem in an earlier version of this forum, but now I can’t find it on the community Axigen Community Forum).
Generally, it helped me to perform the restore procedure. The results of this procedure was to reduce storage space to the total size of all accounts, groups, publicFolder, etc. in this domain. The results are as expected.
At the beginning, the all message storage size were 84,3 GB.
FTP Backup Size were 18,7 GB
Total storage space after recovery are 18,6 GB.
The total number of messages, accounts, filters, group, etc. was exactly the same as before RECOVERY procedure.
My RESTORE procedure what I was using was this:
Stop/block all traffic (sending, receiving, etc.) in domain8.
Thanks btw, Yes this will work but for me this is not a solution but only a work around. In which this needs to be address by Axigen. And if you encounter similar issue from before then this is not a unique issue.
I need a permanent solution incase this will happen again. I do have much bigger axigen deployment and we cannot make a schedule downtime for this scenarios.
Reviewing this thread I didn’t find a refference if you already executed the CLI SCAN ALL command (details here) and my advice is to start a screen session from which you are executing SCAN ALL softPurge after you have entered into the coresponding domain context.
If possible please share the CLI logs coresponding to that command if you didn’t recover back some space.
Just as a temporary solution: are you able to add a new disk and move there some message locations (like messages[7-9]) so you will not be pressed by having not enough space?
Good to know that you managed to solve your problem. In order to get the current situation you have to run first SCAN ALL clearCache and after this a new SCAN ALL.
There is nothing that I could suggest to do in the future to prevent this things as these are not usual situations (maybe configure a weekly cron job based on SCAN ALL CLEARCACHE + SCAN ALL CLI commands that may allert you in case there are some space consumed by leaks so you may apply the purge command).
What you have to be sure is to not configure storages that may consume / overpass your entire disk space.
I really hope you have carefully made mentioned changes as usually readjusting the storages to lower values is usually not supported.
In case you have double checked that the maxFileSize value configured into the storages are bigger than the size of any storage files from that storage unit after the compact was executed than you may have dodged the bullet.
Anyway you have earned the new bravery badge that I have just created.