Page 1 of 1

Proposal on how to fix subject encoding problem

Posted: Tue Aug 19, 2008 11:17 pm
by jojobear99
Apologies for double posting this...I didn't want it to get lost because it was buried in the comments to a post where Renier suggested the problem with subject encoding is in the indy components...

I have taken a look at the source code for PopTray and the RFC spec on email headers, and I believe the subject encoding problem is one that could easily be resolved without waiting for Indy components to do character set conversion. (I don't actually have a Delphi compiler set up to write and test a patch, but here's an outline of the proposal for a solution I've come up with so far:

In uMain.pas, there is a procedure ShowMailMessage(...). In particular, on line 1295, there is a line:
I believe at this point you need to "translate" the subject. The subject is interpreted by the Indy components as Ansi, which is correct based on the email header RFC. However, there's some stuff that was added as an after-thought for how to accomodate international subjects in headers that only ascii is allowed in. So we can "undo" or fix the subject line at this point.
Maybe that line becomes:
and you add a new procedure FixEncoding(subject:string) : string
that will "fix" the subject encoding into something more human readable.

According to the RFC, the format for specifying a non-ansi subject is:
"=?" charset "?" encoding "?" encoded-text "?="
This should be simple enough format to tokenize using standard string tokenizing libraries. if the string does not follow this pattern, return the string as is. Otherwise, we have to process it and fix its encoding.

Encoding is either "Q" for Quoted-Printable or "B" for Base64. Those are the only two options, so not too complicated there, other than locating or creating a library function to decode those two encodings. Base64 allows for longer subjects with foriegn characters, but its not human readable, whereas quoted printable only changes spaces and non-ascii characters using escape codes. I haven't researched this yet, but I would imagine a library or example code for how to decode Base64 and Quoted-Printable back to "normal text" is not difficult to find.

Charset slightly more complicated, in that there's a larger list of choices that are possible. I don't know how good Delphi's built in support for doing charset conversion is, I've only done them in C++ and Java. In C++ everything had to be converted to Unicode instead of UTF-8 and displayed as "wide strings" which is a little weird at first, but doable, though it required a few format strings with %S conversions to change from ansi-unicode and vice versa. In Java its as simple as passing the string and a string with its encoding to a special library function and it auto-magically swaps the encoding. But if its particularly complicated, even just adding support for the most common one, UTF-8, would very much benefit users of pop-tray. UTF-8 to Unicode is a relatively simple conversion even if you have to code it manully. Getting the list component to display wide strings rather than ansi strings may be automatic, or might require a minor change somewhere, not sure.

Anyway, I hope that is helpful! I'd be happy to discuss implementing this feature farther or do some more research if one of these steps turns out to be a hang-up...its a feature I'd very much like to see!

Re: Proposal on how to fix subject encoding problem

Posted: Wed Aug 27, 2008 12:43 pm
by Tahtu
jojobear99 wrote:Renier,
I have taken a look at the source code for PopTray and the RFC spec on email headers, and I believe the subject encoding problem is one that could easily be resolved without waiting for Indy components to do character set conversion.
Yes, you are right.

I used the source code of PopTray to port it to full Unicode Support - enclosed the encoding of the headers you mentioned. After that I sent the whole changed source code and a compiled version inclusive a setup program to Renier.

All is ready to publish - but Renier is not interessed about it. That was some month before.

Maybe you are more successfull to please Renier than I... :roll:

Additional I added HTML preview and RSS support. All of them works fine on my end since more than 4 month...

Posted: Wed Aug 27, 2008 5:22 pm
by jojobear99
Tahtu, even if Ranier has been too busy to review your changes and post them to the website, if you are willing, I would be most interested in obtaining a copy of your modified unicode support version from you for my personal'd certainly be a lot less work than re-inventing the wheel and implementing unicode support myself :) I will private message you my email

Posted: Wed Aug 27, 2008 5:47 pm
by Tahtu
I respect this project is Reniers one - so I think it's up to him to decide if this project should get further developed or not - and if yes how. He is allowed to publish my developement or let it hidden. He know if he invite me to develope this project under this name on this domain I will do my best - but only under that circumces.

Posted: Wed Aug 27, 2008 7:34 pm
by jojobear99
Tahtu, I'm not fully sure I understand your reasons for concern about releasing your modifications with or without Renier's express endorsement.

it is my belief that Renier has made his intentions clear when he released Poptray under the GNU GPL license, which as you are probably aware is a license permitting others to make derivative works so long as the reputation of the original author is protected by you putting your own name into the contributors in the about part of the software, and that if you release it publically you do so under the same license (ie: with source code available and not as closed source for profit). That license specifically does not require involving the original author to release modified versions into the wild as many authors discontinue supporting further development of their software whether that be for time, lack of interest, or other reasons--but the open source movement allows development of great products to continue past the lifetime of interest and availability of the original author.

I can understand if you do not wish to publically release a product to compete with Renier's edition of pop-tray, which is why I asked if you would be willing to release your modified executable privately with or without source code. I'd just like to be able to preview my email without it showing gibberish every second or third message. I have looked at competing products as a solution to the lack of unicode support, however, none of the alternatives are anywhere near as good as pop-tray in other respects.

It would be a big blessing to me to have these enhancements to my copy of poptray, and even larger blessing to be able to have them without having to purchase a delphi compiler and spend many hours learning the libraries necessary to add this feature support outside of my full time job (I'm really a C++ programmer, not a delphi programmer).

And although I know you are under absolutely no obligation to release your modifications, I do hope you would reconsider how much benefit and blessing you have to offer others by sharing. I have seen many stickies on this forum of plugins, even to those located on sites like geocities, to enhance the functionality of pop-tray. As such, it is quite clear the plugin I installed was not authored by Renier, and I could not blame Renier when the plugin initially wiped out one of my saved accounts. However, I was still happy that the plugin was linked none the less because its feaures were very useful, even if it did not have some of throughness of testing and stability under all circumstances that I'd expect from something released as Ranier's code.

Would Ranier's express blessing for you to release your enhanced code from an external location (sourceforge, geocities, rapidshare, etc.) be sufficient? Asking for an original author to bundle your modifications under their name is sometimes asking a lot, and I hope you would not have too much pride to withhold from giving back to the community of pop-tray devotees simply because someone simply didn't have time to review your patches thoroughly or had some constructive criticism you disagreed with or whatever the reason its not part of the main trunk.

Posted: Thu Aug 28, 2008 4:30 am
by Rdsok
I know Renier is busy with his other projects and hasn't had much time to release another version in some time... I also don't believe that he has stopped development... he's just busy and doesn't have much time to work on it as much. He still posts here also... so he doesn't ignore the forums ( he even corrected an index issue to help fix the forum search after its move to another server )...

I also don't believe he is concerned about another branch being developed otherwise he wouldn't have released it as a GNU license.

I would say that the exact name shouldn't be used so a user can distigish between the "official" and the derivitive release. Say something like "PopTray Unicode", "UniPopTray"... "UniPopT" ( got to also have a rap-like named example eh ) LOL

Posted: Thu Aug 28, 2008 8:11 am
by Tahtu
jojobear99 wrote:Tahtu, I'm not fully sure I understand your reasons for concern about releasing your modifications with or without Renier's express endorsement.
Ok, I will try to explain it again - from a different point of view.

Imho there is no sense to split a project (summary of developing, publishing and using it). The world has much to many similary projects (not only software related) - adding a further project means not all existing engergies will be focused on it. So I have absolutely no interessed to initate a splitting of the development of PopTray.

Since Renier is the leader of the PopTray project it's up to him to decide any to implement my work into it or not. Not to deciding something is also a decision. So he decided against my work and me. I respect his decision.

Maybe you can respect my decision not to publish my work on any other way than via Renier?

My changes are very essential and a commercial toolbox is needed to compile. So I would understand that Renier is not satisfied with all this changes - it's not longer "his baby".

This kind of view is the same point of view the humans done in the last thousands year: Egoism. While everybody looks only to hisself and his rights the most people do not have an advantage of it. Imho we humans have to learn working together.

I do not want to make the same mistake - because of that I will not do anything against Reniere or without him while he do not please me to do it.

Is my position more clearly right now?

Posted: Thu Aug 28, 2008 4:13 pm
by Renier
OK, I'll answer here.

I've been considering Tahtu's suggestion for a while now. I know I haven't been able to work on PopTray for a long time now.

The problem with Tahtu's version though is that it uses a commercial library (which basically means it isn't licensable under GPL anymore, and people won't be able to compile it without buying that library).

There are also a few features from normal PopTray removed (customization etc).