ToString CharacterEncoding

[mmatera]

In the next round of improving format compatibility with WMA, we should try to reproduce the following behavior of ToString:

In WMA, by default, ToString produces a string with OutputForm in $CharacterEncoding encoding ("Unicode" ==="UTF-8" by default):

In[1]:= ToString[x^2>0 ->  x \[Element] Reals]                                                                                                                                

Out[1]=  2
        x  > 0 -> x ∈ Reals

In[2]:= ToString[x^2>0 ->  x \[Element] Reals, OutputForm, CharacterEncoding->"Unicode"]                                                                                      

Out[2]=  2
        x  > 0 -> x ∈ Reals

Notice that power is presented in "2D form".

If ASCII encoding is request,

In[3]:= ToString[x^2>0 ->  x \[Element] Reals, OutputForm, CharacterEncoding->"ASCII"]                                                                                        

Out[3]=  2
        x  > 0 -> x \[Element] Reals

Now, Unicode characters are just replaced by their NameCharacter form.

In InputForm, the Unicode and ASCII seem equivalent

In[4]:= ToString[x^2>0 ->  x \[Element] Reals, InputForm, CharacterEncoding->"Unicode"]                                                                                       

Out[4]= x^2 > 0 -> Element[x, Reals]

In[5]:= ToString[x^2>0 ->  x \[Element] Reals, InputForm, CharacterEncoding->"ASCII"]                                                                                         

Out[5]= x^2 > 0 -> Element[x, Reals]

But if we include a String with Unicode characters, we also have a replacement to the NameCharacter form:

In[6]:= ToString["x^2>0 ->  x \[Element] Reals", InputForm, CharacterEncoding->"ASCII"]                                                                                       

Out[6]= "x^2>0 ->  x \[Element] Reals"

Forms in $BoxForms, on the other hand, produce Box Expressions encoded as strings:

'''
In[7]:= ToString[x^2>0 -> x [Element] Reals, StandardForm, CharacterEncoding->"Unicode"] //InputForm

Out[7]//InputForm= "!(*RowBox[{RowBox[{SuperscriptBox["x", "2"], ">", "0"}], "", RowBox[{"x", "∈", TemplateBox[List[], "Reals"]}]}])"

Notice that here I have to wrap the output in `InputForm` to see the string, because otherwise the front end tries to interpret the content of the string.

If we ask for `ASCII` encoding, the Unicode characters are replaced by its NameCharacter form:

In[8]:= ToString[x^2>0 -> x [Element] Reals, StandardForm, CharacterEncoding->"ASCII"] //InputForm

Out[8]//InputForm=
"\!\(\*RowBox[{RowBox[{SuperscriptBox["x", "2"], ">", "0"}], "\[Rule]", RowBox[{"x", "\[Element]", TemplateBox[List[], "Reals"]}]}]\)"
``

Finally, in WMA, FullForm is not a valid second argument:

In[9]:=  ToString[x^2>0 ->  x \[Element] Reals, FullForm]                                                                                                                    

ToString::fmtval: FullForm is not a valid format type.

                   2
Out[9]= ToString[x  > 0 -> x ∈ Reals, FullForm]

[rocky]

Thanks for the investigation and information. Anything with two-dimensional output should be done only after we master DisplayForm. That's a bit down the line.

[mmatera]

Thanks for the investigation and information. Anything with two-dimensional output should be done only after we master DisplayForm. That's a bit down the line.

Sure. In any case, it is clear that ToString distinguishes BoxForms and PrintForms, and that an intermediate task to do is to build a routine that converts a BoxExpression into its String - encoded form.
Also, this investigation suggest that CharacterEncoding is not a parameter in MakeBoxes/format_element but in the boxes_to_format step.