hakril-PythonForWindows/docs/build/html/encoding.html

<!DOCTYPE html>

<html lang="en" data-content_root="./">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" /><meta name="viewport" content="width=device-width, initial-scale=1" />

    <title>19. Python, Windows &amp; encoding &#8212; PythonForWindows 1.0.3 documentation</title>
    <link rel="stylesheet" type="text/css" href="_static/pygments.css?v=fa44fd50" />
    <link rel="stylesheet" type="text/css" href="_static/classic.css?v=def86cc0" />
    <link rel="stylesheet" type="text/css" href="_static/css/mbasic.css?v=957880af" />

    <script src="_static/documentation_options.js?v=baaebd52"></script>
    <script src="_static/doctools.js?v=9a2dae69"></script>
    <script src="_static/sphinx_highlight.js?v=dc90522c"></script>

    <link rel="index" title="Index" href="genindex.html" />
    <link rel="search" title="Search" href="search.html" />
    <link rel="prev" title="18. Samples of code" href="sample.html" />
  </head><body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="sample.html" title="18. Samples of code"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">PythonForWindows 1.0.3 documentation</a> &#187;</li>
        <li class="nav-item nav-item-this"><a href=""><span class="section-number">19. </span>Python, Windows &amp; encoding</a></li>
      </ul>
    </div>

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">

  <section id="python-windows-encoding">
<span id="py-windows-encoding"></span><h1><span class="section-number">19. </span>Python, Windows &amp; encoding<a class="headerlink" href="#python-windows-encoding" title="Link to this heading">¶</a></h1>
<p>This part is only the result of my experimentations and readings and might not reflect the full picture.</p>
<p>My near-futur goal is to have PythonForWindows fully support unicode and use W (Wide) API.
This will result in most of the strings handled by PFW being unicode strings.</p>
<p>In python3 it should work as-is because Windows console handle Wide unicode and Py3 is using the <code class="docutils literal notranslate"><span class="pre">ConsoleWriteW</span></code> API directly.
The fact that python3 <code class="xref py py-class docutils literal notranslate"><span class="pre">str</span></code> type represent unicode string helps a lot.</p>
<p>But, As I want to maintains Python2.7 compatibilty and easyness of use, things beging to become tricky.</p>
<p>The main problem is the ability to print unicode (chinese / russian / …) in the console and object repr.</p>
<dl>
<dt>Example:</dt><dd><div class="doctest highlight-default notranslate"><div class="highlight"><pre><span></span><span class="gp">&gt;&gt;&gt; </span><span class="n">windows</span><span class="o">.</span><span class="n">crypto</span><span class="o">.</span><span class="n">Certificate</span><span class="o">.</span><span class="n">from_file</span><span class="p">(</span><span class="s2">&quot;omae.cer&quot;</span><span class="p">)</span>
<span class="go">&lt;Certificate &quot;お前はもう死んでい&quot; serial=&quot;19 da cc 2b a5 61 b6 98 4e 0d 6c 0c cb ce e6 99&quot;&gt;</span>
</pre></div>
</div>
</dd>
</dl>
<p>To achieve this in Py2 the required additionnal configuration of the console should be:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="nb">set</span> <span class="n">PYTHONIOENCODING</span><span class="o">=</span><span class="n">utf</span><span class="o">-</span><span class="mi">8</span> <span class="c1"># Set environnement variable PYTHONIOENCODING to utf-8. Allowing UTF8 encoding for python output (including sys.stdout).</span>
<span class="n">chcp</span> <span class="mi">65001</span> <span class="c1"># Setting the console code page to 65001 (UTF-8) Allowing the console to print the correct chars when receiving UTF-8 data</span>
</pre></div>
</div>
<p>For it to work optimaly in Py2.7, it require some tricks in both the codebase, the python configuration and console configuration.</p>
<p>My goal is to offer the better experience even without any Python/Console configuration.
But with the additionnal configuration it should be able to print anything without trouble.</p>
<section id="what-does-this-additional-configuration-does">
<h2><span class="section-number">19.1. </span>What does this additional configuration does ?<a class="headerlink" href="#what-does-this-additional-configuration-does" title="Link to this heading">¶</a></h2>
<section id="pythonioencoding">
<h3><span class="section-number">19.1.1. </span>PYTHONIOENCODING<a class="headerlink" href="#pythonioencoding" title="Link to this heading">¶</a></h3>
<p>As the official <a class="reference external" href="https://docs.python.org/2/using/cmdline.html#envvar-PYTHONIOENCODING">Python2 documentation</a> says:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Overrides</span> <span class="n">the</span> <span class="n">encoding</span> <span class="n">used</span> <span class="k">for</span> <span class="n">stdin</span><span class="o">/</span><span class="n">stdout</span><span class="o">/</span><span class="n">stderr</span><span class="p">,</span> <span class="ow">in</span> <span class="n">the</span> <span class="n">syntax</span> <span class="n">encodingname</span><span class="p">:</span><span class="n">errorhandler</span><span class="o">.</span> <span class="n">The</span> <span class="p">:</span><span class="n">errorhandler</span> <span class="n">part</span> <span class="ow">is</span> <span class="n">optional</span> <span class="ow">and</span> <span class="n">has</span> <span class="n">the</span> <span class="n">same</span> <span class="n">meaning</span> <span class="k">as</span> <span class="ow">in</span> <span class="nb">str</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span><span class="o">.</span>
</pre></div>
</div>
<p>The goal of setting this environnement variable is to have the stdout encoding set to <code class="docutils literal notranslate"><span class="pre">utf-8</span></code>.
This will allow a somewhat seamless encoding of any unicode string you try to print.</p>
<p>By default, the <code class="docutils literal notranslate"><span class="pre">sys.stdout.encoding</span></code> is set according to the current console output codepage (<a class="reference external" href="https://docs.microsoft.com/en-us/windows/console/getconsoleoutputcp">GetConsoleOutputCP</a>) at python initialisation.
This can be seen in <a class="reference external" href="https://github.com/python/cpython/blob/2.7/Python/pythonrun.c#L354">Py_InitializeEx</a>.</p>
<p>Most of the default codepage (437 in my case) cannot encode all unicode character.
These two elements together are the reason why priting an unicode string without any change in Python2 will lead to this type of problem:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ chcp
Active code page: 437

$ python -s -E
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; import sys
&gt;&gt;&gt; sys.stdout.encoding
&#39;cp437&#39;
&gt;&gt;&gt; print(u&quot;\u304a\u524d\u306f\u3082\u3046\u6b7b\u3093\u3067\u3044&quot;)
Traceback (most recent call last):
File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt;
File &quot;C:\Python27\lib\encodings\cp437.py&quot;, line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: &#39;charmap&#39; codec can&#39;t encode characters in position 0-8: character maps to &lt;undefined&gt;
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Note that the fails is in <code class="docutils literal notranslate"><span class="pre">cp437.py</span></code> directly linked to <code class="docutils literal notranslate"><span class="pre">sys.stdout.encoding</span></code></p>
</div>
<p>By setting <code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code> you force the <code class="docutils literal notranslate"><span class="pre">sys.stdout.encoding</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ chcp
Active code page: 437

$ set PYTHONIOENCODING=utf-8

$ python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; import sys
&gt;&gt;&gt; sys.stdout.encoding
&#39;utf-8&#39;
&gt;&gt;&gt; print(u&quot;\u304a\u524d\u306f\u3082\u3046\u6b7b\u3093\u3067\u3044&quot;)
πüèσëìπü»πééπüåµ¡╗πéôπüºπüä # Gibberish due to bad codepage in console
</pre></div>
</div>
<p>Well, it does not raise an exception anymore but it’s printing gibberish.
This is because, as-is the console still expects cp437 as an output.</p>
</section>
<section id="chcp-65001">
<h3><span class="section-number">19.1.2. </span>chcp 65001<a class="headerlink" href="#chcp-65001" title="Link to this heading">¶</a></h3>
<p>The <a class="reference external" href="https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/chcp">chcp</a> commande allow to display/change the active console code page.
The codepage 65001 stand for UTF-8.</p>
<p>Thus, setting the console code page to 65001 will tell it to expect UTF-8 as a program output. Which is perfect with our previous setup of <code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ set PYTHONIOENCODING=utf-8

$ chcp 65001
Active code page: 65001

$ python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)] on win32
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; import sys
&gt;&gt;&gt; sys.stdout.encoding
&#39;utf-8&#39;
&gt;&gt;&gt; print(u&quot;\u304a\u524d\u306f\u3082\u3046\u6b7b\u3093\u3067\u3044&quot;)
お前はもう死んでい
</pre></div>
</div>
</section>
</section>
<section id="the-case-of-repr">
<h2><span class="section-number">19.2. </span>The case of __repr__<a class="headerlink" href="#the-case-of-repr" title="Link to this heading">¶</a></h2>
<p>The case of UTF-8 and __repr__ in Python2.7 is more tricky.
I pay particular attention to this case because I am an heavy user of the interactive console and object <code class="docutils literal notranslate"><span class="pre">__repr__</span></code> to explore Windows.</p>
<p>the <code class="docutils literal notranslate"><span class="pre">__repr__</span></code> function cannot return an <code class="docutils literal notranslate"><span class="pre">unicode</span></code> object and must return a <code class="docutils literal notranslate"><span class="pre">str</span></code>.
But as I want to be able to output repr for objects with unicode attributes (like a お前はもう死んでい certificate), I need to encode my unicode __repr__.</p>
<p>In those cases, I encode the repr with the stdout encoding (with backslash escape for non-encodable characters).
This should assure that the result can always be written to stdout.</p>
<p>But based on the encoding of stdout and the code page 3 case may appear.
For example with an object having the following unicode repr:</p>
<blockquote>
<div><ul class="simple">
<li><p>&lt;Certificate “お前はもう死んでい” serial=”19 da cc 2b a5 61 b6 98 4e 0d 6c 0c cb ce e6 99”&gt;</p></li>
</ul>
</div></blockquote>
<p>The possibilities are:</p>
<blockquote>
<div><ul class="simple">
<li><dl class="simple">
<dt>stdout encoding do not handle full unicode (like cp437)</dt><dd><ul>
<li><p>repr will be backslash escaped to allow printing</p></li>
<li><p>&lt;Certificate “\u304a\u524d\u306f\u3082\u3046\u6b7b\u3093\u3067\u3044” serial=”19 da cc 2b a5 61 b6 98 4e 0d 6c 0c cb ce e6 99”&gt;</p></li>
</ul>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt>stdout encoding is utf-8 but code page is not (like cp437)</dt><dd><ul>
<li><p>console will output gibberish by trying to interpret utf-8 as a custom CodePage</p></li>
<li><p>&lt;Certificate “πüèσëìπü»πééπüåµ¡╗πéôπüºπüä” serial=”19 da cc 2b a5 61 b6 98 4e 0d 6c 0c cb ce e6 99”&gt;</p></li>
</ul>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt>stdout encoding is utf-8 and code page is 65001</dt><dd><ul>
<li><p>it works !</p></li>
<li><p>&lt;Certificate “お前はもう死んでい” serial=”19 da cc 2b a5 61 b6 98 4e 0d 6c 0c cb ce e6 99”&gt;</p></li>
</ul>
</dd>
</dl>
</li>
</ul>
</div></blockquote>
</section>
<section id="sample-of-test">
<h2><span class="section-number">19.3. </span>Sample of test<a class="headerlink" href="#sample-of-test" title="Link to this heading">¶</a></h2>
<p>I have created a sample <code class="docutils literal notranslate"><span class="pre">samples\encoding\check_encoding_config.py</span></code> that should help to understand and verify the current configuration of the console.
The code check and display the values of <code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code>, <code class="docutils literal notranslate"><span class="pre">sys.stdout.encoding</span></code> and the current console code page. It also tries to print an unicode string as well as an unicode object __repr__.</p>
<section id="no-setup">
<h3><span class="section-number">19.3.1. </span>No setup<a class="headerlink" href="#no-setup" title="Link to this heading">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code> is not set and code page is something like 437.</p>
<blockquote>
<div><ul class="simple">
<li><p>The printing of an unicode string will fail</p></li>
<li><dl class="simple">
<dt>The printing of an unicode __repr__ will display escaped unicode values</dt><dd><ul>
<li><p>PFW make its best to not raise an encoding related exception on __repr__ by checking <code class="docutils literal notranslate"><span class="pre">sys.stdout.encoding</span></code></p></li>
</ul>
</dd>
</dl>
</li>
</ul>
</div></blockquote>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ chcp 437
Active code page: 437

$ set PYTHONIOENCODING=

$ python samples\encoding\check_encoding_config.py
Python version is &lt;2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)]&gt;
Py2 python/console configuration analysis:
[*] env[PYTHONIOENCODING] = None
    [-] No env variable &lt;PYTHONIOENCODING&gt;.
        sys.stdout encoding will only depends on your console codepage. Leading to high probability of EncodingError if printing unicode string
[*] sys.stdout.encoding = cp437
    [-] Unoptimal stdout encoding
        Recommended fix is setting PYTHONIOENCODING == utf-8
[*] Console Codepage = 437
    [-] Non UTF-8 codepage for the current console
        Setting codepage to UTF8 (chcp 65001) will ensure currect output with PYTHONIOENCODING UTF-8
[-] Error printing unicode string: &#39;charmap&#39; codec can&#39;t encode characters in position 23-31: character maps to &lt;undefined&gt;
Unicode object repr: &lt;MyUtf8Object name=&quot;\u304a\u524d\u306f\u3082\u3046\u6b7b\u3093\u3067\u3044-\u043a\u0430\u043a\u0438\u0435_\u0444\u0436\u044e\u0449\u0434\u0444\u044f&quot;&gt;
</pre></div>
</div>
</section>
<section id="pythonioencoding-only">
<h3><span class="section-number">19.3.2. </span>PYTHONIOENCODING only<a class="headerlink" href="#pythonioencoding-only" title="Link to this heading">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code> is set to <code class="docutils literal notranslate"><span class="pre">utf-8</span></code> and code page is something like 437.</p>
<blockquote>
<div><ul class="simple">
<li><p>The printing of an unicode string will work but display gibberish</p></li>
<li><p>The printing of an unicode __repr__ will work but display gibberish</p></li>
</ul>
</div></blockquote>
<p>This is due to a mis-match between python output encoding and the console expected output.</p>
<p>Example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ chcp
Active code page: 437

$ set PYTHONIOENCODING=utf-8

$ python samples\encoding\check_encoding_config.py
Python version is &lt;2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)]&gt;
Py2 python/console configuration analysis:
[*] env[PYTHONIOENCODING] = utf-8
    [+] Optimal PYTHONIOENCODING
[*] sys.stdout.encoding = utf-8
[*] Console Codepage = 437
    [-] Non UTF-8 codepage for the current console
        Setting codepage to UTF8 (chcp 65001) will ensure currect output with PYTHONIOENCODING UTF-8
Unicode string print: &lt;πüèσëìπü»πééπüåµ¡╗πéôπüºπüä-╨║╨░╨║╨╕╨╡_╤ä╨╢╤Ä╤ë╨┤╤ä╤Å&gt;
Unicode object repr: &lt;MyUtf8Object name=&quot;πüèσëìπü»πééπüåµ¡╗πéôπüºπüä-╨║╨░╨║╨╕╨╡_╤ä╨╢╤Ä╤ë╨┤╤ä╤Å&quot;&gt;
</pre></div>
</div>
</section>
<section id="full-setup">
<h3><span class="section-number">19.3.3. </span>Full setup<a class="headerlink" href="#full-setup" title="Link to this heading">¶</a></h3>
<p><code class="docutils literal notranslate"><span class="pre">PYTHONIOENCODING</span></code> is set to <code class="docutils literal notranslate"><span class="pre">utf-8</span></code> and code page is 65001.
Everything should work:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ set PYTHONIOENCODING=utf-8

$ chcp 65001
Active code page: 65001

$ python samples\encoding\check_encoding_config.py
Python version is &lt;2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:19:22) [MSC v.1500 32 bit (Intel)]&gt;
Py2 python/console configuration analysis:
[*] env[PYTHONIOENCODING] = utf-8
    [+] Optimal PYTHONIOENCODING
[*] sys.stdout.encoding = utf-8
[*] Console Codepage = 65001
Unicode string print: &lt;お前はもう死んでい-какие_фжющдфя&gt;
Unicode object repr: &lt;MyUtf8Object name=&quot;お前はもう死んでい-какие_фжющдфя&quot;&gt;
</pre></div>
</div>
</section>
</section>
</section>


            <div class="clearer"></div>
          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <div>
    <h3><a href="index.html">Table of Contents</a></h3>
    <ul>
<li><a class="reference internal" href="#">19. Python, Windows &amp; encoding</a><ul>
<li><a class="reference internal" href="#what-does-this-additional-configuration-does">19.1. What does this additional configuration does ?</a><ul>
<li><a class="reference internal" href="#pythonioencoding">19.1.1. PYTHONIOENCODING</a></li>
<li><a class="reference internal" href="#chcp-65001">19.1.2. chcp 65001</a></li>
</ul>
</li>
<li><a class="reference internal" href="#the-case-of-repr">19.2. The case of __repr__</a></li>
<li><a class="reference internal" href="#sample-of-test">19.3. Sample of test</a><ul>
<li><a class="reference internal" href="#no-setup">19.3.1. No setup</a></li>
<li><a class="reference internal" href="#pythonioencoding-only">19.3.2. PYTHONIOENCODING only</a></li>
<li><a class="reference internal" href="#full-setup">19.3.3. Full setup</a></li>
</ul>
</li>
</ul>
</li>
</ul>

  </div>
  <div>
    <h4>Previous topic</h4>
    <p class="topless"><a href="sample.html"
                          title="previous chapter"><span class="section-number">18. </span>Samples of code</a></p>
  </div>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/encoding.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<search id="searchbox" style="display: none" role="search">
  <h3 id="searchlabel">Quick search</h3>
    <div class="searchformwrapper">
    <form class="search" action="search.html" method="get">
      <input type="text" name="q" aria-labelledby="searchlabel" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false"/>
      <input type="submit" value="Go" />
    </form>
    </div>
</search>
<script>document.getElementById('searchbox').style.display = "block"</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="py-modindex.html" title="Python Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="sample.html" title="18. Samples of code"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">PythonForWindows 1.0.3 documentation</a> &#187;</li>
        <li class="nav-item nav-item-this"><a href=""><span class="section-number">19. </span>Python, Windows &amp; encoding</a></li>
      </ul>
    </div>
    <div class="footer" role="contentinfo">
    &#169; Copyright 2015-2020, Clement Rouault.
      Created using <a href="https://www.sphinx-doc.org/">Sphinx</a> 7.3.7.
    </div>
  </body>
</html>