Migrating Plone 5.2 to Python 3
Migrating Plone 5.2 to Python 3#
This chapter provides instructions and tips for porting Plone projects to Python 3.
If you want to upgrade add-ons to Python 3, the list of all Plone packages that still need to be ported can be found on the GitHub project board Python 3 porting state for Plone add-ons.
You should support Python 2 and 3 with the same codebase to allow it to be used in existing versions of Plone.
Plone 5.2 supports Python 2.7, Python 3.6, Python 3.7, and Python 3.8.
We use six and modernize for the first steps toward Python 3.
In general, you should follow these steps to port add-ons:
buildoutfor the add-on to be ported.
Update code with python-modernize.
Use plone.recipe.precompiler (also called
precompilerfor brevity) to find syntax errors.
Start the instance and find more errors.
Test functionality manually.
Run and fix all tests.
Update package information.
Update package buildout and test setup.
In the GitHub repository of the add-on:
Open a ticket with the title "Add support for Python 3".
Create a new branch named
Using released Plone 5.2#
Usually you can use the latest Plone 5.2 release.
The version pins for the latest release can be found for
pip at https://dist.plone.org/release/5.2-latest/requirements.txt and for
buildout at https://dist.plone.org/release/5.2-latest/versions.cfg.
Install Plone with Python 3.6, 3.7, or 3.8, and then add your add-ons as source using
Using core development buildout#
buildout.coredev, the latest development version of Plone can be used.
It contains everything for porting an add-on to Python 3.
Follow these steps:
# Clone coredev and use branch 5.2: git clone email@example.com:plone/buildout.coredev.git coredev_py3 cd coredev_py3 git checkout 5.2 # Create a py3 virtual environment with either Python 3.6, 3.7, or 3.8: python3.8 -m venv . # Install buildout: ./bin/pip install -r requirements.txt
Next create a file called
local.cfg in the root of the buildout.
This file will be used to add your add-on to the buildout.
Add your package as in the following example.
collective.package with the name of the add-on you want to port.
This example expects a branch with the name
python3 to exist for the package.
Adapt it for your use case.
[buildout] extends = buildout.cfg always-checkout = true allow-picked-versions = true custom-eggs += collective.package test-eggs += collective.package [test] auto-checkout += collective.package [sources] collective.package = git firstname.lastname@example.org:collective/collective.package.git branch=python3
With the file in place, run
Then the source of the add-on package will be checked out into the
./bin/buildout -c local.cfg
You can also add development tools like
Products.PrintingMailHost to your
Products.PDBDebugMode will help a lot with issues during porting to Python 3.
custom-eggs += collective.package Products.PDBDebugMode plone.reload Products.PrintingMailHost test-eggs += collective.package [test] auto-checkout += collective.package
Now everything is prepared to work on the migration of the package.
For small packages or packages that have few dependencies, it is a good idea to try starting your instance now.
If it does not start up, you should continue with the next steps instead of trying to fix each issue as it appears.
2. Automated fixing with modernize#
python-modernize is a utility that automatically prepares Python 2 code for porting to Python 3.
python-modernize, there is manual work ahead.
There are some problems that
python-modernize cannot fix on its own.
It also can make changes that are not really needed.
You need to closely review all changes after you run this tool.
python-modernize will warn you when it is not sure what to do with a possible problem.
Check this Cheat Sheet with idioms for writing Python 2/3 compatible code.
python-modernize adds an import of the compatibility library
six if needed.
The import is added as the last import, therefore it is often necessary to reorder the imports.
The easiest way is to use isort, which does this for you automatically.
Check the Python style guide for Plone for information about the order of imports and an example configuration for
six is used in the code, make sure that
six is added to the
install_requires list in the
setup.py of the package.
modernize into your Python 3 environment with
./bin/pip install modernize
isort into your Python 3 environment with
./bin/pip install isort
The following command is a dry-run. It shows all changes that
modernize would make.
./bin/python-modernize -x libmodernize.fixes.fix_import src/collective.package
-x option is used to exclude certain fixers.
The one that adds
from __future__ import absolute_import should not be used.
./bin/python-modernize -l for a complete list of fixers and the fixers documentation.
The following command applies all fixes to the files:
./bin/python-modernize -wn -x libmodernize.fixes.fix_import src/collective.package
You can use
isort to fix the order of imports:
./bin/isort -rc src/collective.package
After you run the commands above, you need to review all changes and fix what
modernizer did not get right.
You can make use of
plone.recipe.precompiler to identify syntax errors quickly.
This recipe compiles all Python code already at buildout-time, not at run-time.
You will see right away when there is some illegal syntax.
Add the following line to the section
./bin/buildout -c local.cfg to enable and use
parts += precompiler
precompile will be run every time you run buildout.
If you want to avoid running the complete buildout every time, you can use the
install keyword of buildout like this as a shortcut:
./bin/buildout -c local.cfg install precompiler
4. Start the instance#
As a next step, we recommend that you try to start the instance with your add-on. This will fail on all import errors (e.g., relative imports that are not allowed in Python 3). If it works then you can try to install the add-on.
You need to fix all issues that appear before you can do manual testing to check for big, obvious issues.
Common Issues during startup#
The following issues will abort your startup. You need to fix them before you are able to test the functionality by hand or run tests.
If you get an error message similar to the following.
TypeError: Class advice impossible in Python3. Use the @implementer class decorator instead.
This tells you that there is a class using an
implements statement which needs to be replaced by the
For example, code that is written as follows:
from zope.interface import implements class Group(form.BaseForm): implements(interface.IGroup)
needs to be replaced with:
from zope.interface import implementer @implementer(interfaces.IGroup) class Group(form.BaseForm):
The same is true for
provides(IFoo) and some other class advices.
These need to be replaced with their respective decorators, such as
Relative imports such as
import permissions are no longer permitted.
Instead, use fully qualified import paths, such as
from collective.package import permissions.
Syntax error on importing async#
Starting with Python 3.7, you can no longer have a module called
async (see https://github.com/celery/celery/issues/4849).
You need to rename all such files, folders, or packages (such as
5. Test functionality manually#
Now that the instance is running, you should do the following, and fix all errors as they appear.
Install the add-on.
Test basic functionality, for example, adding and editing content types and views.
Uninstall the add-on.
For this step, it is recommended that you have installed
Products.PDBDebugMode to help debug and fix issues.
6. Run Tests#
$ ./bin/test --all -s collective.package
Remember that you can run
./bin/test -s collective.package -D to enter a
pdb session when an error occurs.
With some luck, there will not be too many issues left with the code at this point.
If you are unlucky, then you have to fix doctests.
These should be changed so that Python 3 is the default.
For example, string types (or text) should be represented as
u'foo', and bytes types (or data) should be represented as
Search for examples of
Py23DocChecker in Plone's packages to find a pattern which allows updated doctests to pass in Python 2.
7. Update add-on information#
Add the following four entries of the classifiers list in
"Framework :: Plone :: 5.2", # ... "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8",
Make an entry in the
8. Create a test setup that tests in Python 2 and Python 3#
You need to update the
buildout of the add-on you are migrating to also support Plone 5.2 and Python 3.
buildout of most add-ons are different, we cannot offer advice that works for all add-ons.
But it is a good idea to create an empty new package with
bobtemplates.plone, and either copy the code of the add-on in there or the new skeleton files into the old add-on.
The least you can do is look at the files created by
bobtemplates.plone, and copy whatever is appropriate to the add-on you are working on.
$ ./bin/pip install bobtemplates.plone $ ./bin/mrbob -O some.addon bobtemplates.plone:addon
Always use the newest version of
Add-ons created like this contain a setup that allows testing in Python 2 and Python 3, and various Plone versions locally, and on Travis-CI using
Look at the files
9. Frequent Issues#
Text and Bytes#
This is by far the biggest issue when porting to Python 3. Read the Conservative Python 3 Porting Guide, Strings to be prepared.
As a rule of thumb, you can assume that in Python 3 everything should be text. Only in very rare cases will you need to handle bytes.
python-modernize will not fix all your text/bytes issues.
It only replaces all cases of
You need to make sure that the code you are porting will remain unchanged in Python 2 and (at least in most cases) use text in Python 3.
Try to modify the code in such a way that when dropping support for Python 2 you will be able to delete while lines. For example:
if six.PY2 and isinstance(value, six.text_type): value = value.encode('utf8') do_something(value)
You can use the helper methods
safe_encode in Plone 5.1).
python-modernize also does not touch the import statement
from StringIO import StringIO, even though this works only in Python 2.
You have to check whether you are dealing with text or binary data and use the appropriate import statement from
# For textual data from six import StringIO # For binary data from six import BytesIO
Here is a list of helpful references on the topic of porting Python 2 to Python 3.