What’s New In Python 3.6

Release:3.6.0a1
Date:June 03, 2016

This article explains the new features in Python 3.6, compared to 3.5.

For full details, see the Misc/NEWS file.

Note

Prerelease users should be aware that this document is currently in draft form. It will be updated substantially as Python 3.6 moves towards release, so it’s worth checking back even after reading earlier versions.

Summary – Release highlights

New syntax features:

Windows improvements:

  • The py.exe launcher, when used interactively, no longer prefers Python 2 over Python 3 when the user doesn’t specify a version (via command line arguments or a config file). Handling of shebang lines remains unchanged - “python” refers to Python 2 in that case.

New Features

PEP 498: Formatted string literals

Formatted string literals are a new kind of string literal, prefixed with 'f'. They are similar to the format strings accepted by str.format(). They contain replacement fields surrounded by curly braces. The replacement fields are expressions, which are evaluated at run time, and then formatted using the format() protocol.

>>> name = "Fred"
>>> f"He said his name is {name}."
'He said his name is Fred.'

See PEP 498 and the main documentation at Formatted string literals.

PYTHONMALLOC environment variable

The new PYTHONMALLOC environment variable allows to set the Python memory allocators and/or install debug hooks.

It is now possible to install debug hooks on Python memory allocators on Python compiled in release mode using PYTHONMALLOC=debug. Effects of debug hooks:

  • Newly allocated memory is filled with the byte 0xCB
  • Freed memory is filled with the byte 0xDB
  • Detect violations of Python memory allocator API. For example, PyObject_Free() called on a memory block allocated by PyMem_Malloc().
  • Detect write before the start of the buffer (buffer underflow)
  • Detect write after the end of the buffer (buffer overflow)
  • Check that the GIL is held when allocator functions of PYMEM_DOMAIN_OBJ (ex: PyObject_Malloc()) and PYMEM_DOMAIN_MEM (ex: PyMem_Malloc()) domains are called.

Checking if the GIL is held is also a new feature of Python 3.6.

See the PyMem_SetupDebugHooks() function for debug hooks on Python memory allocators.

It is now also possible to force the usage of the malloc() allocator of the C library for all Python memory allocations using PYTHONMALLOC=malloc. It helps to use external memory debuggers like Valgrind on a Python compiled in release mode.

On error, the debug hooks on Python memory allocators now use the tracemalloc module to get the traceback where a memory block was allocated.

Example of fatal error on buffer overflow using python3.6 -X tracemalloc=5 (store 5 frames in traces):

Debug memory block at address p=0x7fbcd41666f8: API 'o'
    4 bytes originally requested
    The 7 pad bytes at p-7 are FORBIDDENBYTE, as expected.
    The 8 pad bytes at tail=0x7fbcd41666fc are not all FORBIDDENBYTE (0xfb):
        at tail+0: 0x02 *** OUCH
        at tail+1: 0xfb
        at tail+2: 0xfb
        at tail+3: 0xfb
        at tail+4: 0xfb
        at tail+5: 0xfb
        at tail+6: 0xfb
        at tail+7: 0xfb
    The block was made by call #1233329 to debug malloc/realloc.
    Data at p: 1a 2b 30 00

Memory block allocated at (most recent call first):
  File "test/test_bytes.py", line 323
  File "unittest/case.py", line 600
  File "unittest/case.py", line 648
  File "unittest/suite.py", line 122
  File "unittest/suite.py", line 84

Fatal Python error: bad trailing pad byte

Current thread 0x00007fbcdbd32700 (most recent call first):
  File "test/test_bytes.py", line 323 in test_hex
  File "unittest/case.py", line 600 in run
  File "unittest/case.py", line 648 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  File "unittest/suite.py", line 122 in run
  File "unittest/suite.py", line 84 in __call__
  ...

(Contributed by Victor Stinner in issue 26516 and issue 26564.)

Other Language Changes

  • None yet.

New Modules

  • None yet.

Improved Modules

asyncio

Since the asyncio module is provisional, all changes introduced in Python 3.6 have also been backported to Python 3.5.x.

Notable changes in the asyncio module since Python 3.5.0:

  • The ensure_future() function and all functions that use it, such as loop.run_until_complete(), now accept all kinds of awaitable objects. (Contributed by Yury Selivanov.)
  • New run_coroutine_threadsafe() function to submit coroutines to event loops from other threads. (Contributed by Vincent Michel.)
  • New Transport.is_closing() method to check if the transport is closing or closed. (Contributed by Yury Selivanov.)
  • The loop.create_server() method can now accept a list of hosts. (Contributed by Yann Sionneau.)
  • New loop.create_future() method to create Future objects. This allows alternative event loop implementations, such as uvloop, to provide a faster asyncio.Future implementation. (Contributed by Yury Selivanov.)
  • New loop.get_exception_handler() method to get the current exception handler. (Contributed by Yury Selivanov.)
  • New timeout() context manager to simplify timeouts handling code. (Contributed by Andrew Svetlov.)
  • New StreamReader.readuntil() method to read data from the stream until a separator bytes sequence appears. (Contributed by Mark Korenberg.)
  • The loop.getaddrinfo() method is optimized to avoid calling the system getaddrinfo function if the address is already resolved. (Contributed by A. Jesse Jiryu Davis.)

contextlib

The contextlib.AbstractContextManager class has been added to provide an abstract base class for context managers. It provides a sensible default implementation for __enter__() which returns self and leaves __exit__() an abstract method. A matching class has been added to the typing module as typing.ContextManager. (Contributed by Brett Cannon in issue 25609.)

datetime

The datetime.strftime() and date.strftime() methods now support ISO 8601 date directives %G, %u and %V. (Contributed by Ashley Anderson in issue 12006.)

faulthandler

On Windows, the faulthandler module now installs a handler for Windows exceptions: see faulthandler.enable(). (Contributed by Victor Stinner in issue 23848.)

os

A new close() method allows explicitly closing a scandir() iterator. The scandir() iterator now supports the context manager protocol. If a scandir() iterator is neither exhausted nor explicitly closed a ResourceWarning will be emitted in its destructor. (Contributed by Serhiy Storchaka in issue 25994.)

pickle

Objects that need calling __new__ with keyword arguments can now be pickled using pickle protocols older than protocol version 4. Protocol version 4 already supports this case. (Contributed by Serhiy Storchaka in issue 24164.)

readline

Added set_auto_history() to enable or disable automatic addition of input to the history list. (Contributed by Tyler Crompton in issue 26870.)

rlcompleter

Private and special attribute names now are omitted unless the prefix starts with underscores. A space or a colon is added after some completed keywords. (Contributed by Serhiy Storchaka in issue 25011 and issue 25209.)

Names of most attributes listed by dir() are now completed. Previously, names of properties and slots which were not yet created on an instance were excluded. (Contributed by Martin Panter in issue 25590.)

site

When specifying paths to add to sys.path in a .pth file, you may now specify file paths on top of directories (e.g. zip files). (Contributed by Wolfgang Langner in issue 26587).

socketserver

Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now support the context manager protocol. (Contributed by Aviv Palivoda in issue 26404.)

subprocess

subprocess.Popen destructor now emits a ResourceWarning warning if the child process is still running. Use the context manager protocol (with proc: ...) or call explicitly the wait() method to read the exit status of the child process (Contributed by Victor Stinner in issue 26741).

telnetlib

Telnet is now a context manager (contributed by Stéphane Wirtel in issue 25485).

typing

The typing.ContextManager class has been added for representing contextlib.AbstractContextManager. (Contributed by Brett Cannon in issue 25609.)

unittest.mock

The Mock class has the following improvements:

urllib.robotparser

RobotFileParser now supports the Crawl-delay and Request-rate extensions. (Contributed by Nikolay Bogoychev in issue 16099.)

warnings

A new optional source parameter has been added to the warnings.warn_explicit() function: the destroyed object which emitted a ResourceWarning. A source attribute has also been added to warnings.WarningMessage (contributed by Victor Stinner in issue 26568 and issue 26567).

When a ResourceWarning warning is logged, the tracemalloc is now used to try to retrieve the traceback where the detroyed object was allocated.

Example with the script example.py:

import warnings

def func():
    return open(__file__)

f = func()
f = None

Output of the command python3.6 -Wd -X tracemalloc=5 example.py:

example.py:7: ResourceWarning: unclosed file <_io.TextIOWrapper name='example.py' mode='r' encoding='UTF-8'>
  f = None
Object allocated at (most recent call first):
  File "example.py", lineno 4
    return open(__file__)
  File "example.py", lineno 6
    f = func()

The “Object allocated at” traceback is new and only displayed if tracemalloc is tracing Python memory allocations and if the warnings was already imported.

winreg

Added the 64-bit integer type REG_QWORD. (Contributed by Clement Rouault in issue 23026.)

zipfile

A new ZipInfo.from_file() class method allows making a ZipInfo instance from a filesystem file. A new ZipInfo.is_dir() method can be used to check if the ZipInfo instance represents a directory. (Contributed by Thomas Kluyver in issue 26039.)

The ZipFile.open() method can now be used to write data into a ZIP file, as well as for extracting data. (Contributed by Thomas Kluyver in issue 26039.)

zlib

The compress() function now accepts keyword arguments. (Contributed by Aviv Palivoda in issue 26243.)

fileinput

hook_encoded() now supports the errors argument. (Contributed by Joseph Hackman in issue 25788.)

Optimizations

  • The ASCII decoder is now up to 60 times as fast for error handlers surrogateescape, ignore and replace (Contributed by Victor Stinner in issue 24870).
  • The ASCII and the Latin1 encoders are now up to 3 times as fast for the error handler surrogateescape (Contributed by Victor Stinner in issue 25227).
  • The UTF-8 encoder is now up to 75 times as fast for error handlers ignore, replace, surrogateescape, surrogatepass (Contributed by Victor Stinner in issue 25267).
  • The UTF-8 decoder is now up to 15 times as fast for error handlers ignore, replace and surrogateescape (Contributed by Victor Stinner in issue 25301).
  • bytes % args is now up to 2 times faster. (Contributed by Victor Stinner in issue 25349).
  • bytearray % args is now between 2.5 and 5 times faster. (Contributed by Victor Stinner in issue 25399).
  • Optimize bytes.fromhex() and bytearray.fromhex(): they are now between 2x and 3.5x faster. (Contributed by Victor Stinner in issue 25401).
  • Optimize bytes.replace(b'', b'.') and bytearray.replace(b'', b'.'): up to 80% faster. (Contributed by Josh Snider in issue 26574).
  • Allocator functions of the PyMem_Malloc() domain (PYMEM_DOMAIN_MEM) now use the pymalloc memory allocator instead of malloc() function of the C library. The pymalloc allocator is optimized for objects smaller or equal to 512 bytes with a short lifetime, and use malloc() for larger memory blocks. (Contributed by Victor Stinner in issue 26249).
  • pickle.load() and pickle.loads() are now up to 10% faster when deserializing many small objects (Contributed by Victor Stinner in issue 27056).

Build and C API Changes

Deprecated

New Keywords

async and await are not recommended to be used as variable, class, function or module names. Introduced by PEP 492 in Python 3.5, they will become proper keywords in Python 3.7.

Deprecated Python modules, functions and methods

Deprecated functions and types of the C API

  • None yet.

Deprecated features

  • The pyvenv script has been deprecated in favour of python3 -m venv. This prevents confusion as to what Python interpreter pyvenv is connected to and thus what Python interpreter will be used by the virtual environment. (Contributed by Brett Cannon in issue 25154.)
  • When performing a relative import, falling back on __name__ and __path__ from the calling module when __spec__ or __package__ are not defined now raises an ImportWarning. (Contributed by Rose Ames in issue 25791.)

Deprecated Python behavior

Removed

API and Feature Removals

  • inspect.getmoduleinfo() was removed (was deprecated since CPython 3.3). inspect.getmodulename() should be used for obtaining the module name for a given path.
  • traceback.Ignore class and traceback.usage, traceback.modname, traceback.fullmodname, traceback.find_lines_from_code, traceback.find_lines, traceback.find_strings, traceback.find_executable_lines methods were removed from the traceback module. They were undocumented methods deprecated since Python 3.2 and equivalent functionality is available from private methods.
  • The tk_menuBar() and tk_bindForTraversal() dummy methods in tkinter widget classes were removed (corresponding Tk commands were obsolete since Tk 4.0).

Porting to Python 3.6

This section lists previously described changes and other bugfixes that may require changes to your code.

Changes in the Python API

  • The format of the co_lnotab attribute of code objects changed to support negative line number delta. By default, Python does not emit bytecode with negative line number delta. Functions using frame.f_lineno, PyFrame_GetLineNumber() or PyCode_Addr2Line() are not affected. Functions decoding directly co_lnotab should be updated to use a signed 8-bit integer type for the line number delta, but it’s only required to support applications using negative line number delta. See Objects/lnotab_notes.txt for the co_lnotab format and how to decode it, and see the PEP 511 for the rationale.
  • The functions in the compileall module now return booleans instead of 1 or 0 to represent success or failure, respectively. Thanks to booleans being a subclass of integers, this should only be an issue if you were doing identity checks for 1 or 0. See issue 25768.
  • Reading the port attribute of urllib.parse.urlsplit() and urlparse() results now raises ValueError for out-of-range values, rather than returning None. See issue 20059.
  • The imp module now raises a DeprecationWarning instead of PendingDeprecationWarning.
  • The following modules have had missing APIs added to their __all__ attributes to match the documented APIs: calendar, csv, ElementTree, enum, fileinput, ftplib, logging, optparse, subprocess, tarfile, threading and wave. This means they will export new symbols when import * is used. See issue 23883.
  • When performing a relative import, if __package__ does not compare equal to __spec__.parent then ImportWarning is raised. (Contributed by Brett Cannon in issue 25791.)
  • When a relative import is performed and no parent package is known, then ImportError will be raised. Previously, SystemError could be raised. (Contributed by Brett Cannon in issue 18018.)
  • Servers based on the socketserver module, including those defined in http.server, xmlrpc.server and wsgiref.simple_server, now only catch exceptions derived from Exception. Therefore if a request handler raises an exception like SystemExit or KeyboardInterrupt, handle_error() is no longer called, and the exception will stop a single-threaded server. (Contributed by Martin Panter in issue 23430.)
  • spwd.getspnam() now raises a PermissionError instead of KeyError if the user doesn’t have privileges.
  • The socket.socket.close() method now raises an exception if an error (e.g. EBADF) was reported by the underlying system call. See issue 26685.
  • The decode_data argument for smtpd.SMTPChannel and smtpd.SMTPServer constructors is now False by default. This means that the argument passed to process_message() is now a bytes object by default, and process_message() will be passed keyword arguments. Code that has already been updated in accordance with the deprecation warning generated by 3.5 will not be affected.

Changes in the C API