How to monitor the automatic compacting of the storage files

This article describes how the automatic storage compacting works, as well as how to get information about this automatic process by using the log files.

Resolution

Axigen X1 introduces a new feature which was required by many of our customers — the auto-compacting of Axigen storage files. The operation is triggered automatically for files that are detected to have a "compaction" level of less than 40%*. The first detection attempt takes place 15 minutes after the service start and, in case no such files are detected, it is re-scheduled one hour later. The process goes on until the service is stopped.

*) The selection will also check if the file creation date / last compact time was at least 1 day before.


The auto-compact activity can be monitored by checking the PROCESSING log file for the following lines:

2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: found 8 HSF2 storage units registered; search a fragmented container file (SAPI load: 1.0000; SAPI waitLoad: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: scan storage unit 00-00-C172-4D08CC89-F65776D0 (storageLoad: 0.8425; waitLoad: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: skip compacted container 00@00-00-C172-4D08CC89-F65776D0 (compaction level: 99.50%; load: 0.8425)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: skip compacted container 01@00-00-C172-4D08CC89-F65776D0 (compaction level: 99.50%; load: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: scan storage unit 01-00-7468-4D08CC89-B6536D2E (storageLoad: 0.0000; waitLoad: 0.0000)
...
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: skip compacted container 08@01-00-7468-4D08CC89-B6536D2E (compaction level: 99.50%; load: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: scan storage unit 02-00-2536-4D08CC89-86E50079 (storageLoad: 0.0666; waitLoad: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: scan storage unit 02-00-2536-4D08CC89-86E50079 (storageLoad: 0.0666; waitLoad: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: skip compacted container 00@02-00-2536-4D08CC89-86E50079 (compaction level: 46.45%; load: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: enqueue container 01@02-00-2536-4D08CC89-86E50079 (compaction level: 36.91%; priority: 6443; load: 0.0020; storageLoad: 0.0666; waitLoad: 0.0000)
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Autocompact: enqueue container 02@02-00-2536-4D08CC89-86E50079 (compaction level: 0.00%; priority: 9971; load: 0.0000; storageLoad: 0.0666; waitLoad: 0.0000)
...
2017-02-22 17:21:18 +0200 08 host62 PROCESSING:51000000: Start autocompact for container 02@02-00-2536-4D08CC89-86E50079
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Start compacting storage usid '02-00-2536-4D08CC89-86E50079'
2017-02-22 17:21:18 +0200 16 host62 PROCESSING:51000000: >> Start compacting container file '/var/opt/axigen/domains/axigen.com/messages/02.hsf' (initial size: 703736 KB)
2017-02-22 17:21:19 +0200 16 host62 PROCESSING:51000000: >> 1156 records found
...
2017-02-22 17:21:19 +0200 16 host62 PROCESSING:51000000: >> 1150 of 1156 records created
2017-02-22 17:21:20 +0200 16 host62 PROCESSING:51000000: >> All records created, start copying them to compacted container
2017-02-22 17:21:20 +0200 16 host62 PROCESSING:51000000: >> 115 of 1156 records copied
...
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >> 1035 of 1156 records copied
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >> 1150 of 1156 records copied
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >> All records copied, reinitializing the compacted container
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >> Compact finished successfully (final size: 34200 KB; 669536 KB gained)
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >> Finished compacting storage usid '02-00-2536-4D08CC89-86E50079', 1 of 1 container files compacted
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >>     Initial storage size: 703736 KB
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >>     Final storage size  : 34200 KB
2017-02-22 17:21:21 +0200 16 host62 PROCESSING:51000000: >>     Storage space gained: 669536 KB
2017-02-22 17:21:21 +0200 08 host62 PROCESSING:51000000: Autocompacted container 02@02-00-2536-4D08CC89-86E50079; storage space gained: 669536 KB

...

2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: found 8 HSF2 storage units registered; search a fragmented container file (SAPI load: 0.0000; SAPI waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 00-00-C172-4D08CC89-F65776D0 (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 01-00-7468-4D08CC89-B6536D2E (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 02-00-2536-4D08CC89-86E50079 (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 02-00-3492-4D08CCFA-251B15CD (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 02-00-6FAD-4D08CCFA-4FC1DDD5 (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 02-00-99E6-4D08CCFA-219910DF (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 02-00-EA82-4D08CCFA-0BC86EC6 (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: scan storage unit 03-00-6835-57FE1F77-BAD65955 (storageLoad: 0.0000; waitLoad: 0.0000)
2017-02-22 19:59:07 +0200 16 host62 PROCESSING:5100008E: >> Autocompact: no fragmented container file found

...

2017-02-24 07:29:13 +0200 16 host62 PROCESSING:51000132: >> Autocompact: skip fresh container 6E@02-00-EA82-4D08CCFA-0BC86EC6 (remaining time: 01:47:43; compaction level: 19.64%; load: 0.0000)
2017-02-24 07:29:13 +0200 16 host62 PROCESSING:51000132: >> Autocompact: skip compacted container 6F@02-00-EA82-4D08CCFA-0BC86EC6 (compaction level: 100.00%; load: 0.0000)