Sort indexes

Advanced Configuration of Axigen

This article is about Axigen's new sort indexes, available starting with Axigen X3 Update 2 (10.3.2)

The new sort indexes used by Axigen starting with version 10.3.x are implemented using a set of sqlite databases that are stored in the filesystem the Axigen domain is hosted on, but outside the Axigen domain storage managed from CLI / WebAdmin. The root folder for the new sort databases is a folder named following the convention <Axigen working directory>/domains/<domain_name>/sort.idx.

The location of <Axigen working directory> depends on your operating system and configuration. By default, the sort indexes paths would be: 

  • On Linux: /var/opt/axigen/domains/<domain_name>/sort.idx
  • On Windows: C:\Program Files\Axigen Mail Server\domains\<domain_name>\sort.idx

Database location

<Axigen working directory>/domains/<domain_name>/sort.idx/<bucket>/<account_id>/<folder_id>/simple (or headers_stream, etc)

For shared or public folders read / unread status:

<Axigen working directory>/domains/<domain_name>/sort.idx/<bucket>/<account_id>/private/<owner_id>/<folder_id>/simple 

Find the database for a given folder

Database location is computed in the same manner for SortIndexes and SearchIndexes (but sort indexes are stored in sort.idx whereas search indexes are stored in email.idx)

Given a database, find which folder it indexes

Precomputing sort indexes

Since sort indexes are required for WebMail to function and old sort indexes are lost at upgrade time, as soon as the new sort index implementation is started it spawns a number of sort index jobs that precompute sort indexes for a number of folders (INBOX, Sent, Archive) for all the accounts in the domain.

What the index contains

The new sort indexes contain use a set of tables that define sort orders for email headers.

To allow sorting by date, subject, sender, recipient, etc., the email headers that define all the sort criteria are stored in databases written as files in the filesystem that are readable (in terms of Operating System permissions) by the axigen user and group.

What to expect when the index is built for the first time

Because sort indexes will be recomputed from scratch the first time a sort index and rebuilding a sort index requires parsing all the emails from a given folder some delays / temporary hangs are to be expected until the majority of sort indexes have been rebuilt. While sort indexes are being recomputed Axigen will perform a large amount of I/O calls.

Approximate index size

The sort index dimension heavily depends on the actual email headers that are stored within a given mailbox.

Some examples of sort index sizes are listed below:

Mailbox information (as shown by CLI registryinformation)

Sort Index size (size of root folder as shown by du -hs)

Mbox size = 4048289 Kb (3.8 GB)
Mbox message count = 30200 messages
Mbox folder count = 70 folders
22 MB

Mbox size = 12454895 Kb (11.8 GB)
Mbox message count = 192425 messages
Mbox folder count = 204 folders

46 MB

Mbox size = 7603021 Kb (7.2 GB)
Mbox message count = 62133 messages
Mbox folder count = 58 folders

53 MB

Collation

Collation as described here: http://www.unicode.org/reports/tr35/tr35-collation.html#Root_Collation

Where applicable Axigen performs sorting based on a collation algorithm that is UTF-8 aware and that makes sense for most languages.

That means ordering for text fields (such as subject, mail from) is aware of diacritical marks so that "u < ü < U".

CLI Commands

Commands that display information related to the new type of sort indexes are listed below.

A number of sort indexes (e.g. VCARD, CardDAV HREF, etc.) have not changed — these are managed using the CLI commands that predate the release of 10.3.2 (i.e. SHOW SortIndexInformation works on legacy sort indexes).

Status of indexes for a given account

In the CLI account context (the context that is opened by the update account command), a number of CLI calls show information about sort indexes:

CLI command

Description

<domain-account#> SHOW NewSortIndexInformation
The sort-index information:
Index type: body_snippet
Index count: 12
Total key count: 73803
Total size: 11172 KB

Index type: header
Index count: 10
Total key count: 0
Total size: 42900 KB

The command displays, for each of the new sort index types:

  1. the index type
  2. the total number of indexes
  3. the total number of keys that are part of this index type
  4. the size taken by the index


<domain-account#> SHOW NewSortIndexInformation folder INBOX
The sort-index information:
Folder name: INBOX
Index type: body_snippet
Index key count: 1264
Index size: 196 KB

Index type: headers_stream
Index key count: 1264
Index size: 332 KB

Index type: message_data
Index key count: 1264
Index size: 72 KB

Index type: simple
Index key count: 0
Index size: 128 KB

+OK: command successful
When using the "folder" argument the command can be used to display the status for an individual folder.
<domain-account#> SHOW NewSortIndexInformation allfolders
The sort-index information:
Folder name: ana
Index type: body_snippet
Index key count: 0
Index size: 16 KB

Index type: header
Index key count: 0
Index size: 36 KB

Index type: headers_stream
Index key count: 0
Index size: 16 KB

Index type: message_data
Index key count: 0
Index size: 16 KB

Index type: simple
Index key count: 0
Index size: 36 KB

Folder name: Drafts

...
When using the "allfolders" argument the command displays sort index status for all the indexed folders.

Status of indexing jobs at server level

In the processing context (accessible by using config server followed by config processing), a command provides information about the sort index jobs currently running on the server:

CLI command

Description

<server-processing#> SHOW NewSortIndexJobs

No running new sort index jobs


+OK: command successful

Shows the status of the running mail sort index jobs.