I have strange behavior from 1 Axigen Deployment in which suddenly I lost disk space. Upon investigation / analyzing found out that Axigen Storage became bloated.
In this table you can see the difference.
Left side will be the actual sizes per account in WebAdmin.
Rigth side is the actual sizes per container in the domain.
Since I post this topic till today the storage bulk up another 30GB.
I know the amount of emails / traffic that this server serve and it will not reach nor consume that space.
Please find logs of CLI but unfortunately I did a mistake and only informational messages only not protocol communication. Maybe it will help still but if needed I can rerun the Compact again tomorrow.
Unfortunatelly the file do not contain any CLI log lines (but only JOBLOG ones). The CLI log lines have CLI:<session_id> and in order to have all details you have to set the CLI service log level to Protocol Communication before connecting to CLI for executing the compact operation.
Sorry for this.
BR,
Ioan
PS: just to let you know that the latest 10.2.2 patch is 10.2.3.8 (details available here)
Also for the new update, Yes, I understand that there is new update for some fixes but we keep this one on hold as we expect last quarter last year that the new X4 will come out and we will shift this server to a new servers / machines.
Hope this log will be helpful to trouble shoot my issue, I cannot run this server properly, under stress and monitoring as my free disk space is under 2%.
I’ve checked the CLI logs and nothing strange found there.
Could you please also share the output from the following CLI command:
<domain#> show disposablemetadatainformation
On the other side I failed to understand is why you have configured more than 1TB for your messages (10 message storages x 128 files x 1GB) if you do not have enough space available.
For the question why more than 1TB containers.
This is due to the troubleshooting method I did, trying to see what is going on. And trying to force axigen to expand and create new containers hoping when It makes compact it will recreate some hsf to other container.
My original containers are 7 only. Created 8 and 9 later on when I encounter this issue.
Me neither cannot find issue why Axigen acted like this.
A few years ago (2018) I also had this problem, No recommendations helped (e.g. commands FINDINVALIDMSG purge, SHOW DisposableMetadataInformation, COMPACT All forced, etc.),
I described this problem in an earlier version of this forum, but now I can’t find it on the community Axigen Community Forum).
Generally, it helped me to perform the restore procedure. The results of this procedure was to reduce storage space to the total size of all accounts, groups, publicFolder, etc. in this domain. The results are as expected.
At the beginning, the all message storage size were 84,3 GB.
FTP Backup Size were 18,7 GB
Total storage space after recovery are 18,6 GB.
The total number of messages, accounts, filters, group, etc. was exactly the same as before RECOVERY procedure.
My RESTORE procedure what I was using was this:
Stop/block all traffic (sending, receiving, etc.) in domain8.
Thanks btw, Yes this will work but for me this is not a solution but only a work around. In which this needs to be address by Axigen. And if you encounter similar issue from before then this is not a unique issue.
I need a permanent solution incase this will happen again. I do have much bigger axigen deployment and we cannot make a schedule downtime for this scenarios.
Reviewing this thread I didn’t find a refference if you already executed the CLI SCAN ALL command (details here) and my advice is to start a screen session from which you are executing SCAN ALL softPurge after you have entered into the coresponding domain context.
If possible please share the CLI logs coresponding to that command if you didn’t recover back some space.
Just as a temporary solution: are you able to add a new disk and move there some message locations (like messages[7-9]) so you will not be pressed by having not enough space?