NMHListAllFoldersNoticingSpeedup

Created on November 12, 2023 at 11:39 am

One CARDINAL thing that mail clients almost universally want is a full list of all of your mail folders. Often getting this list is considered expensive, so the mail client will cache this information somehow once it asks for it. ( N)MH ORG and the frontends for it are no exception to this, with ( N)MH ORG providing the basic commands to list all folders on demand (such as folder ).

As an ( N)MH ORG frontend, exmh (also) is no exception to this. In fact, when exmh was written it was felt that listing all MH ORG folders was sufficiently costly that exmh should not merely cache this information in memory, but record it in a persistent cache file, normally ~/Mail/.folders (which then became a convenient thing to use from the command line, for example to grep to see if you had a particular folder already and where exactly in your folder hierarchy you’d put it). Exmh PERSON updated this .folders cache if you created or renamed folders inside it, but if you did that outside of the command line, you would get to click on the ‘Find all folders’ menu item. Forgetful people might well invoke this every so often just in case.

( Exmh PERSON is about thirty years old DATE , and thirty years ago DATE having a persistent cache of ( N)MH ORG folder names was definitely a good idea, even if it had to be manually refreshed every so often.)

When I first ORDINAL started using exmh, ‘Find all folders’ was the kind of thing that you took a break for. Exmh PERSON would pop up a message telling you it was working, you’d iconify the whole thing, and then some minutes TIME later you’d notice that the message window’s icon had gone away and everything was done. Over the years DATE this stayed more or less the same, both when I was working on local disks and when I was using exmh (and NMH PERSON ) on a NFS mounted filesystem from, say, our original Solaris LOC fileservers. At most I vaguely noticed that the message window wasn’t staying up for quite as long. But I didn’t really pay attention to this, even when we moved to all-SSD storage in our current fileservers in 2018 DATE .

As I’ve mentioned earlier, I’m now (also) using a second ORDINAL NMH frontend, MH-E in GNU Emacs PRODUCT . MH-E ORG also maintains a cache of NMH ORG folder information, but unlike exmh it’s not a persistent cache; MH-E ORG builds the cache in the background on startup and then maintains it in memory for as long as you’re running Emacs. For reasons outside the scope of this entry I was recently dealing with a situation where I could have to completely rebuild this cache and wait for the rebuild to be complete.

When I started writing the code that waited for this complete folder cache rebuild, I expected to have to sit there drumming my fingers impatiently for at least a few seconds or CARDINAL

tens of seconds TIME . Instead, when I actually tested my code it was a ‘blink and I’ll miss it’ experience. Testing from the command line shows that today DATE , even on a NFS mounted filesystem ‘ folders -recurse ‘ usually takes under half a second CARDINAL ; even a bad case (with cold caches on the NFS ORG client) seems to be only a couple of seconds CARDINAL . Going back and testing in exmh shows the same timing, more or less; exmh pops up the message window telling me it’s going to be a while finding all of my folders, and then the message window almost immediately disappears again.

It’s not unexpected that the march of time improves the performance of my environment. But it’s still nice to see and experience, even if it took me a while to notice.

(Looking back, this increased speed in what boils down to directory scanning is probably mostly due to moving to all SSD storage in 2018 DATE , and I probably didn’t notice at the time because this isn’t something I do often or pay much attention to.)

PS ORG : MH-E’s folder cache implementation is actually reasonably clever. It’s allowed to be incomplete (and MH-E will deliberately invalidate portions of it under some circumstances), and MH-E’s ORG normal folder completion will fill in sections of the cache on demand if necessary. Also, MH-E ORG usually doesn’t need folder information right away, because you usually start MH-E and then look at your inbox for a bit before refiling messages or changing folders. However, the file-like style of completion this requires can be inconvenient sometimes, so I wrote an alternate version that needs total knowledge of all folders.

Connecting to blog.lzomedia.com... Connected... Page load complete