tex2lyx swallows non-latin1 characters (missing unicode support)

[Georg Baum]

Reported by Jacek M. Holeczek on the devel list:

Hi,

1. the "revtex4 home page" that I use is:

        ​http://authors.aps.org/revtex4/

2. the summary of revtex4 commands and options is here:

        ​http://authors.aps.org/revtex4/summary.ps
        ​http://authors.aps.org/revtex4/summary.tex
   according to this summary the following commands/options can have
   "[optional text]": "\email", "\homepage", "\altaffiliation",
   "\bibitem" (have a look at "\bibinfo" and "\footnotetext", too).

3. the revtex4 author's guide:

        ​http://authors.aps.org/revtex4/auguide.ps
        ​http://authors.aps.org/revtex4/auguide.tex
   contains some more elaborate descriptions of the "[optional text]".

4. please find attached a simple "tex2lyx_import_test.tex" file, which

   demonstrates the "tex2lyx import problems" (this is for sure not an
   comprehensive set of characters that are badly imported, just the
   result of my quick scan)

5. last, but not least, I did consider adding my problems into bugzilla,

   but it required me to "create an account" - that was too much for me.
   Sending an email to lyx related lists was much easier - my email
   address gets "confirmed" just the first time I send to a particular
   list. (I'm pretty sure that the bugzilla would require me to remember
   yet another "user" and "password" pair - that sucks, I've got way too
   many of them already).
Hope it helps,
Best regards,
Jacek.

[Georg Baum]

test file

[lasgouttes]

The testcase that georg added is not really related to revtex. I think the
revtex problems mentioned below are fixed now.

I do not know however what we should do about the accents. Georg?

[Georg Baum]

Somebody needs to go through all of them and add support if needed. This will
be a lot easier after tex2lyx is updated to output unicode, since tex2lyx
currently does not know anything about encodings.
Since this will be wasted effort if José implements his python version I am not
going to do anything on the accents.

[Georg Baum]

* #4299 has been marked as a duplicate of this bug. *

[Juergen Spitzmueller]

* #4714 has been marked as a duplicate of this bug. *

[Uwe Stöhr]

I'll bring Unicode support for tex2lyx on the agenda at our next LyX developer
meeting.
Postponing.

[Juergen Spitzmueller]

Jean-Marc, whenever you feel that this is ready, please set the target of this
to the then forthcoming minor release. I guess we just need to copy the tex2lyx
directory from trunk once things have settled, right?

[lasgouttes]

I had given up on the idea of porting this to branch, but I can do it if you
want me to. Most of the code is in tex2lyx, but there are some bits in docstream
too.

There a still a few easy things that would be nice like

• a -e option to set the encoding
• passing current encoding to tex2lyx for child documents.

[Juergen Spitzmueller]

I haven't followed the changes in detail, but if it is not too dangerous, I
would very much opt for fixing this problem in branch and for subsequently
moving tex2lyx towards the current file format. The latter (tex2lyx support for
newer features) is a primary goal, in my opinion.

I guess you know best if this goal can be reached within the current cycle.

[Uwe Stöhr]

Subject: Re: tex2lyx swallows non-latin1 characters (missing unicode

support)

I had given up on the idea of porting this to branch, but I can do it if you
want me to.

I'm not Jürgen but want you to do this ;-) When you need help for testing etc., please ask. I'll try
to work again on tex2lyx when I have more spare time than in the moment.

There a still a few easy things that would be nice like

• a -e option to set the encoding
• passing current encoding to tex2lyx for child documents.

The best would be to open enhancement bug reports that this won't be forgotten.

[lasgouttes]

The remaining problems are fixed in trunk now.

URL: ​http://www.lyx.org/trac/changeset/28294
Log:
now that tex2lyx reads the files as utf8, file names should be correct

URL: ​http://www.lyx.org/trac/changeset/28295
Log:
Add a new command line parameter -e that sets the default (latex) encoding.
This is overridden by any subsequent inputenc command.

The current encoding is also passed down to \input or \included files.

[lasgouttes]

* #1961 has been marked as a duplicate of this bug. *

[lasgouttes]

* #2130 has been marked as a duplicate of this bug. *

[lasgouttes]

Fixed in branch at ​http://www.lyx.org/trac/changeset/28858

[Uwe Stöhr]

LyX 1.6.3 is ready