** Observed with 3/16/99 Mozilla Win32 build ** Send yourself 2 msgs: 1. HTML and Plain text 2. Both should contain a Japanese Subject line: "Kore ha Nihongo no Meeru desu." 3. The same wording as the subject line should be in the body text. 4. Send yourself these 2 msgs. Receive them and display them. Note that both have problems of character corruption right after "Me" in "Meeru". Apparently the long vowel symbol is causing a problem. Should check if this is a general conversion failure from JIS to Unicode in this area.

momoi, please clearify the sending is done on 5.0 or 4.5 ? Is this is a sending problem or a receiving problem ? If the problem is receiving, then the target should be M3, if the problem is sending, then mark it M4. I mark it as M3 for now, please change the target ASAP. Thanks.

The original msgs were sent by Comm4.51 and are correctly displayed by the same client.This seems to be either a display/conversion problem or parsing problem by 5.0, i.e. mail headers and body may not be parsed correctly when non-ASCII characters are there.TM should be M3m therefore.

I propose to test the exact same text by creating a ISO-2022-JP meta-tagged HTML. This way, we can verify if this is a converter bug or somewhere in the messenger.

This does not seem to be a straight conversion failure. I see other kinds of corruption when messages get longer. By the way, I see that the problem string I described above gets displayed OK in some msgs (typically a 1-line msg) but get corrupted in longer ones. 1-line display test for the layout is at the above URL. The browser has no problem with it.

The problem does not happen to all the japanese characters. Looks like it happens for some specific characters (of iso-2022-jp) which confilicts with html (e.g. 0x213C where 3C is '<'). Since we have already decided to change libmime to use unicode, the problem will be solved when the change is available. Even in the new code when to convert to unicode is important, we should consider this to avoid the problem like this bug. Rich, can this be ready be M4?

Adding myself to cc-list.

My plate is really full for M4 already. I am doing some major reworking of the output routines for libmime and I am not sure I will be able to address this issue without some assistance. - rhp

We need to review where to put the unicode converter in the new code. I18n eng. can do that help, also possible to do a simple test (by using the characters which had problems in M3) when the code is checked in.

Just updating this bug. I probably won't be able to do any major investigation in this area before M4. Naoki, since we are not generating XML for headers and HTML for the body, we need to investigate the header issue to see the behavior now. (Also, if the emitters need to be exporting charset information, I could use a little help on what we should output.) - rhp

I saw this is working in my local build with Rich's change. There is a bug in UTF-8 encoder (#4968) so the text is truncated.

Rich, would you change the status to 'FIXED'?

** Checked with 4/16/99 Win32 build ** Now that we can see non-ASCII body, I'm able to check on this problem. I sent the same test string which failed originally (see the URL above), and the result indicates that there is still the same problem remaining in processing isio-2022-jp charcters containg 0x3C in the first or the 2nd byte. As far as I can see this is not a truncation problem reported elsewhere. Reopening it ...

The display of the test string works on headers. So maybe I'm just seeing the efects of the truncation bug in the body, but this is hard to tell without seeing the rest of the string. There is some indication that I'm seeing a messed-up HTML structure. I'm going to leave the status as is but we need to look into this further.

I tried some of the test cases sent from momoi by using my local build. My build has a hack to work around the UTF-8 truncation (#4968). I saw the message body without truncation and looks fine execpt some character display as boxes (but this can be seen in the thread pane). So I think the new code is working fine. But the verification is not possible because of the blocking bug (#4968).

I checked in an error handling code which will reallocate the buffer in case the converter's estimate length was incorrect thus avoid the truncation. It will be included in the next build. Please verify this bug again with that build.

4968 was FIXED 4/19.

** Checked with 4/20/99 Win32 build ** Now that theh truncation bug has been fixed, I can now see all the text in the mail body part. The parsing failure does not occur with the original problem string I used to file this bug. There are other similar strings in othe mail messages in my mailbox and none show this type of parsing failures. Marking the fix verified.