Domain Storage unusual Size in File Level

Good day,

I have strange behavior from 1 Axigen Deployment in which suddenly I lost disk space. Upon investigation / analyzing found out that Axigen Storage became bloated.

In this table you can see the difference.
Left side will be the actual sizes per account in WebAdmin.
Rigth side is the actual sizes per container in the domain.

Bloated size is 451GB.

Accounts GB
1 22 84G ./messages5
2 11 86G ./messages7
3 2 82G ./messages6
4 1 87G ./messages2
5 1 84G ./messages4
6 1 43M ./objects
7 71 4.3G ./messages9
8 1 83G ./messages3
9 1 78G ./messages8
10 6 80G ./messages
11 15 665G
12 6
13 14
14 1 -451
15 1
16 13
17 46
18 1
214
![image 322x404](upload://9Q20HkdGz0jwTyca7ysbHeiPOvQ.png)

Hoping to find answer and suggestions.

Regards,
Jay

1 Like

Any suggestions / answers?

Hello Jay,

Did you try to run a compact job for this domain ?

Also send me the output of below CLI commands:

<#> update domain YOUR_DOMAIN
<domain#> SHOW StorageInformation details

Regards,
Florin

Hi Floria,

Thanks for the reply.

Yes, I run compact on this domain to see if it resolve the issue. But no luck.

Please see attached output (few missing lines on the first container)
storage.txt (150.6 KB)

Regards,
Jay

@florin.burada do we have any answer here?

I am still waiting for any feedback that can help on my case.

Regards,
Jay

Good day,

Any support here that can give me a help?

Since I post this topic till today the storage bulk up another 30GB.
I know the amount of emails / traffic that this server serve and it will not reach nor consume that space.

Please see the latest details of the containers

87G ./messages5
87G ./messages7
88G ./messages6
88G ./messages2
87G ./messages4
48M ./objects
9.7G ./messages9
86G ./messages3
83G ./messages8
84G ./messages
697G .

Please give me some advice or answer.

Currently I am running out of space on this server I have 1% left and I cannot keep up freeing some space anymore.

Regards,
Jay

Hello Jay,

Could you please trigger a force compact for your message storages?

Like running the following CLI commands:

<#> update domain your.domain
<domain#> compact all forced

Looking forward for your feedback.

BR,
Ioan

@indreias

No luck at all.
I did everything as per axigen documentations.
did FINDINVALIDMSG * PURGE
did COMPACT ALL FORCED

I am in similar state right now and I need solution for this.
The axigen runs without issue except for this unexplained occupation of storage.

Regards,
Jay

Hello Jay,

In order to understand better your particular issue please provide the CLI protocol log lines from COMPACT ALL FORCED command.

Also please let us know the exact version of your Axigen installation (and OS details).

Thx,
Ioan

@indreias

In a while I will give you the CLI protocol logs from the new COMPACT ALL FORCED command and all the details needed.

Thanks,
Jay

@indreias

Please find logs of CLI but unfortunately I did a mistake and only informational messages only not protocol communication. Maybe it will help still but if needed I can rerun the Compact again tomorrow.

CLI.txt (144.5 KB)

Operating system: Linux (Centos 6.10) / x64
Current Server version: 10.2.2.96

Regards,
Jay

Hello Jay,

Unfortunatelly the file do not contain any CLI log lines (but only JOBLOG ones). The CLI log lines have CLI:<session_id> and in order to have all details you have to set the CLI service log level to Protocol Communication before connecting to CLI for executing the compact operation.

Sorry for this.

BR,
Ioan

PS: just to let you know that the latest 10.2.2 patch is 10.2.3.8 (details available here)

@indreias

Please find correct CLI Protocol Logs for your reference.

CLI.txt (2.6 MB)

Also for the new update, Yes, I understand that there is new update for some fixes but we keep this one on hold as we expect last quarter last year that the new X4 will come out and we will shift this server to a new servers / machines.

Hope this log will be helpful to trouble shoot my issue, I cannot run this server properly, under stress and monitoring as my free disk space is under 2%.

Regards,
Jay

Hello Jay,

I’ve checked the CLI logs and nothing strange found there.

Could you please also share the output from the following CLI command:

<domain#>  show disposablemetadatainformation

On the other side I failed to understand is why you have configured more than 1TB for your messages (10 message storages x 128 files x 1GB) if you do not have enough space available.

Best regards,
Ioan

@indreias

Please see below details:

<domain#> show disposablemetadatainformation

  • Current size: 8848454 Kb
  • Maximum available size: 301989888 Kb
  • Percent from available size: 2 %
  • Maximum storage capacity: 1207959552 Kb
  • Percent from storage capacity: 0 %

For the question why more than 1TB containers.
This is due to the troubleshooting method I did, trying to see what is going on. And trying to force axigen to expand and create new containers hoping when It makes compact it will recreate some hsf to other container.

My original containers are 7 only. Created 8 and 9 later on when I encounter this issue.

Me neither cannot find issue why Axigen acted like this.

Regards,
Jay

@indreias

Hi there

Do you have any idea / suggestion how to fix this issue?

I am still waiting for some help.

Regards,
Jay

Hello everyone,

Any suggestion will be appreciated. All help will do.

Regards,
Jay

Hi,

A few years ago (2018) I also had this problem, No recommendations helped (e.g. commands FINDINVALIDMSG purge, SHOW DisposableMetadataInformation, COMPACT All forced, etc.),
I described this problem in an earlier version of this forum, but now I can’t find it on the community Axigen Community Forum).
Generally, it helped me to perform the restore procedure. The results of this procedure was to reduce storage space to the total size of all accounts, groups, publicFolder, etc. in this domain. The results are as expected.
At the beginning, the all message storage size were 84,3 GB.
FTP Backup Size were 18,7 GB
Total storage space after recovery are 18,6 GB.
The total number of messages, accounts, filters, group, etc. was exactly the same as before RECOVERY procedure.

My RESTORE procedure what I was using was this:

  1. Stop/block all traffic (sending, receiving, etc.) in domain8.
  2. Make a backup domain8 (FTP Backup).
  3. Delete domain8 (remove all space on FS).
  4. Create Domain8 (option Add a new domain).
  5. Performing a recovery (from the copy made in p2.), details in (Axigen Mail Server - How to restore a domain using FileZilla).
  6. Check new Internal IDs for all accounts.
  7. Recreate user filters by changing file name /var/opt/axigen/filters/DomainStorageId-Internal ID.wmfilter
  8. Start all traffic (sending, receiving, etc.) in domain8.

Our Axigen Mail-Server version was 8.0.2 (linux/x86),

Best regards,
bcteam

@bcteam

Thanks btw, Yes this will work but for me this is not a solution but only a work around. In which this needs to be address by Axigen. And if you encounter similar issue from before then this is not a unique issue.

I need a permanent solution incase this will happen again. I do have much bigger axigen deployment and we cannot make a schedule downtime for this scenarios.

Regards,
Jay

Hello @Jay ,

Reviewing this thread I didn’t find a refference if you already executed the CLI SCAN ALL command (details here) and my advice is to start a screen session from which you are executing SCAN ALL softPurge after you have entered into the coresponding domain context.

If possible please share the CLI logs coresponding to that command if you didn’t recover back some space.

Just as a temporary solution: are you able to add a new disk and move there some message locations (like messages[7-9]) so you will not be pressed by having not enough space?

HTH,
Ioan