Unicode characters in math insets breaks pdf/latex generation

[anek@…]

I get this on Mac OSX 10.4.10, but I assume it to be a general problem. To replicate:

Create a new document.
Insert a greater than or equal sign (≥) using the keyboard.
Select View -> PDF (pdflatex) and you get a pdf.

Now delete this text and insert a math inline formula
Again enter ≥ using the keyboard
Select View -> PDF (pdflatex) and you get an error (changing encoding to utf8 as suggested in the
error dialogue does not seem to help)

Since you can type ≥ an end-user would expect it to come out in print. So as I see it typing should
either be prohibited or the character should be converted to a command in the process when it is
converted to a pdf.
A work around is to convert the document to LyX1.4 (which does convert the character to a command)
and then reopen it. This is discussed under #3313.

This bug relates somewhat to #3511, #3781, and #3765.

[Georg Baum]

Yep. mathed is not prepared for unicode at all. Some characters may work by
chance for certain encodings, but in general mathed does not take care of
unicode characters.

[Uwe Stöhr]

This is a general problem we can solve by creating a new file similar to
unicodesymbols that contains the translations for math - something like that:

0x2264 "
le"

This translation is only called when the character is inserted to mathed, for
text the existing unicodesymbols file is used.

I put in the agenda for LyX 1.6.0.

[Georg Baum]

I guess that an additional column in the existing file would be easier to
maintain. This is something I had in mind from the beginning, but deliberately
ignored because not getting the text export right for the special characters
that already work in 1.4.x would be a regression, and the math stuff is clearly
a new feature.

[anek@…]

As mentioned, a workaround is to export to 1.4-format and then re-import. Not a good workaround, but
shows that the code in lyx2lyx (specifically the revert_unicode part, and even more specifically the check
if you are in a math inset, I believe) can solve this problem. AFAIU, this means that if a similar construct
could be used in the latex-preparation, the bug could be solved.

[younes]

* #4891 has been marked as a duplicate of this bug. *

[forenr@…]

Fixed in trunk:
​http://www.lyx.org/trac/changeset/25028

[Georg Baum]

This is an interesting solution, and certainly good enough for 1.6.0, but it
has still some minor problems:

• It breaks tex2lyx, because tex2lyx assumes that in math LyX and LaTeX formats

are identical

• many symbols have native commands in math mode, and it would be better to use

those instead of \mathsym, which is only a workaround

• the name \mathsym is dangerous, \lyx_mathsym would be better

[forenr@…]

• I don't see any way to not break tex2lyx (if we want to support unicode

symbols in math) other than allowing utf8 encoding only and using something like
\DeclareUnicodeCharacter{XXXX}{\ifmmode<code for math>\else<code for text>\fi}
in the unicodesymbols file. Then, tex2lyx is already broken as regards unicode,
and it was supposed to be replaced by something else based on python.

• When it is clear that some symbol has a native math mode command, it easy to

account for it and avoid \mathsym. I see that \ensuremath{} is currently used
in this case, so I am going to commit a patch addressing this.

• Yes, I agree that \mathsym is too generic to hope no clash will occur.

I am going to fix that, too.

[forenr@…]

Hmm, on second thought, I think that comment 3 is the way to go for avoiding
\mathsym when we already have a native math mode command.

[Georg Baum]

implementation of comment 3

[forenr@…]

You patch works, so I committed it and also changed the most obvious symbols
in the unicodesymbols file:
​http://www.lyx.org/trac/changeset/25061

[younes]

Thanks a lot to you Georg! And to you of course Enrico :-)

[Juergen Spitzmueller]

* #4845 has been marked as a duplicate of this bug. *

[Uwe Stöhr]

Your recent mathed Unicode work introduces two new regression bugs that make the
UserGuide uncompilable, see #4912.

[forenr@…]

Fixed:
​http://www.lyx.org/trac/changeset/25088
​http://www.lyx.org/trac/changeset/25090
​http://www.lyx.org/trac/changeset/25109

[Juergen Spitzmueller]

* #5120 has been marked as a duplicate of this bug. *

[Juergen Spitzmueller]

* #5171 has been marked as a duplicate of this bug. *

[Juergen Spitzmueller]

* #5202 has been marked as a duplicate of this bug. *

[Juergen Spitzmueller]

* #5370 has been marked as a duplicate of this bug. *

[Juergen Spitzmueller]

LyX 1.6.0 is released.