Currently (apparently) the filtering back-end writes to the Inbox before doing a delete, move whatever. A rewrite to process messages in memory would: - improve speed - reduce the chance of running out of disk space (especially considering messages don't disappear until compact) - reduce the frequency of Inbox compacts when set up to trigger - reduce the chance of a message ending up in the wrong folder due to a crash when downloading messages. The lucky assignee of this bug report should work out what needs to be done to make the filtering service operable on an in-memory message (for incoming and outgoing filters) as well as on-disk messages (for after the fact filters). Hence this might be done at the same time as the back end of bug #9417, comprehensive filtering.

Darn! Missed it by one bug!

Bug 9417 seems like it's just "auto filtering after the fact" (which might be a better summary than "comprehensive mail filters" :-), with a couple of new filter actions being requested. Seems pretty orthogonal to this issue. Also, I cc'd David Bienvenu, who knows everything there is to know about the way we filter incoming messages.

I guess you could describe it as auto filtering, but bear in mind it was around and the master bug for various types of filtering before all those other bugs existed. The reason I mentioned it was that someone not considering it might be inclined to convert it to a filtering system that's purely in memory. On second thoughts this would be OK as you could just load the message when it is on disk. But basically writing the back-end actions for filtering mail (ie delete an existing is different than a delete a new under this scheme) should probably be done at the same time (to test the design), but that it is only a part of the entire back-end implementation of other types of filtering. Hence it would be useful for this architecture change to be done before the advanced filters get embarked upon.

while I agree that it would be nice to not write to the inbox when filtering, I disagree with some of what you say. Filtered messages do not remain in the inbox; the inbox gets truncated at the start of the filtered message, so there is no effect on compaction, and there is not much of a window where messages can end up in the wrong folder - basically, we'd have to crash after finishing copying the message to the destination folder and before calling truncate. It's basically a speed issue, and the increased complexity of keeping the message in memory when filtering has never quite seemed worth it. If the message gets big, we'd need to spill it to a file anyway, and that file might as well be the inbox. If for some reason, the truncate is not happening, that would be a bug. Do you have evidence of that?

No, you're probably right with respect to the truncation. With respect to crashing behaviour, I know from experience that messages ending up in a totally different folder is a common occurrence for me in 4.x. I believe the 5.x code is basically taken from there. I'll have to wait and see whether the problem persists in Moz. Point taken with regards to spilling, could an intelligent implementation just filter on the message proper and ignore attachments? I doubt anyone's going to receive a message where the main section is bigger than the virtual memory size.

Turns out that 5.0 was NOT truncating the inbox after filtering. I fixed that (thanks for the tip!). The problem with just filtering on part of a message is that we have body filters, and it's not easy to know in advance that we're NOT going to need the body, at least not in 4.x because we had js filters. In 5.0, we could look at all the filters and see if any of them were body filters but it just makes the logic that much more complicated.

By js body filters do you mean on the attachments themselves?

What's the status of this? WONTFIX?

very unlikely that we would change this, at this point.

*** Bug 32888 has been marked as a duplicate of this bug. ***

sorry for the spam. making bugzilla reflect reality as I'm not working on these bugs. filter on FOOBARCHEESE to remove these in bulk.

->WONTFIX per comment 8.