NMHListAllFoldersNoticingSpeedup

By admin

One
CARDINAL

thing that mail clients almost universally want is a full list of all of your mail folders. Often getting this list is considered expensive, so the mail client will cache this information somehow once it asks for it. (

N)MH
ORG

and the frontends for it are no exception to this, with (

N)MH
ORG

providing the basic commands to list all folders on demand (such as folder ).

As an (

N)MH
ORG

frontend, exmh (also) is no exception to this. In fact, when exmh was written it was felt that listing all

MH
ORG

folders was sufficiently costly that exmh should not merely cache this information in memory, but record it in a persistent cache file, normally ~/Mail/.folders (which then became a convenient thing to use from the command line, for example to grep to see if you had a particular folder already and where exactly in your folder hierarchy you’d put it).

Exmh
PERSON

updated this .folders cache if you created or renamed folders inside it, but if you did that outside of the command line, you would get to click on the ‘Find all folders’ menu item. Forgetful people might well invoke this every so often just in case.

(

Exmh
PERSON

is

about thirty years old
DATE

, and

thirty years ago
DATE

having a persistent cache of (

N)MH
ORG

folder names was definitely a good idea, even if it had to be manually refreshed every so often.)

When I

first
ORDINAL

started using exmh, ‘Find all folders’ was the kind of thing that you took a break for.

Exmh
PERSON

would pop up a message telling you it was working, you’d iconify the whole thing, and then

some minutes
TIME

later you’d notice that the message window’s icon had gone away and everything was done. Over

the years
DATE

this stayed more or less the same, both when I was working on local disks and when I was using exmh (and

NMH
PERSON

) on a NFS mounted filesystem from, say, our original

Solaris
LOC

fileservers. At most I vaguely noticed that the message window wasn’t staying up for quite as long. But I didn’t really pay attention to this, even when we moved to all-SSD storage in our current fileservers in

2018
DATE

.

As I’ve mentioned earlier, I’m now (also) using a

second
ORDINAL

NMH frontend, MH-E in

GNU Emacs
PRODUCT

.

MH-E
ORG

also maintains a cache of

NMH
ORG

folder information, but unlike exmh it’s not a persistent cache;

MH-E
ORG

builds the cache in the background on startup and then maintains it in memory for as long as you’re running Emacs. For reasons outside the scope of this entry I was recently dealing with a situation where I could have to completely rebuild this cache and wait for the rebuild to be complete.

When I started writing the code that waited for this complete folder cache rebuild, I expected to have to sit there drumming my fingers impatiently for

at least a few seconds or
CARDINAL


tens of seconds
TIME

. Instead, when I actually tested my code it was a ‘blink and I’ll miss it’ experience. Testing from the command line shows that

today
DATE

, even on a NFS mounted filesystem ‘ folders -recurse ‘ usually takes

under half a second
CARDINAL

; even a bad case (with cold caches on the

NFS
ORG

client) seems to be

only a couple of seconds
CARDINAL

. Going back and testing in exmh shows the same timing, more or less; exmh pops up the message window telling me it’s going to be a while finding all of my folders, and then the message window almost immediately disappears again.

It’s not unexpected that the march of time improves the performance of my environment. But it’s still nice to see and experience, even if it took me a while to notice.

(Looking back, this increased speed in what boils down to directory scanning is probably mostly due to moving to all SSD storage in

2018
DATE

, and I probably didn’t notice at the time because this isn’t something I do often or pay much attention to.)


PS
ORG

: MH-E’s folder cache implementation is actually reasonably clever. It’s allowed to be incomplete (and MH-E will deliberately invalidate portions of it under some circumstances), and

MH-E’s
ORG

normal folder completion will fill in sections of the cache on demand if necessary. Also,

MH-E
ORG

usually doesn’t need folder information right away, because you usually start MH-E and then look at your inbox for a bit before refiling messages or changing folders. However, the file-like style of completion this requires can be inconvenient sometimes, so I wrote an alternate version that needs total knowledge of all folders.