UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte (while migrating sqlite3 to postgreSQL - djangocms)

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte (while migrating sqlite3 to postgreSQL - djangocms)

我一直在尝试将我的 djangocms 项目的数据库从默认的 sqlite3 迁移到 postgreSQL。但是在这个过程结束时我得到了这个错误:UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte。 我不确定如何处理这个问题,因为我没有数据库方面的经验,而且根据我所见,互联网上关于此类问题的大多数帖子都与数据库无关。

为了达到这一点我做了什么: 我用 sqlite 创建了一个本地 djangocms 网站。为了进行数据库迁移,我遵循了 here 中描述的过程:我将数据库转储到 db.json 文件中,然后在 pgAdmin 上创建了新的 postgres 数据库。我在我的 django settings.py 中修改了字段 DATABASES,安装了 psycopg2 和 运行 python manage.py makemigrationspython manage.py migrate 然后(在 Python shell 中)我运行:

from django.contrib.contenttypes.models import ContentType
ContentType.objects.all().delete()

python manage.py loaddata db.json.

之前

我还必须重新创建一个帮助我登录 djangocms 管理员的超级用户。但是果然我的网站没有了(连djangocms的布局都错了)

如有任何帮助,我们将不胜感激! 非常感谢。

以下是回溯:

(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py loaddata db.json
Traceback (most recent call last):
  File " C:\Users\...\vEnv\myWebsite\project\website\manage.py", line 22, in <module>
    main()
  File " C:\Users\...\vEnv\myWebsite\project\website\manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\management\commands\loaddata.py", line 172, in load_label
    for obj in objects:
  File " C:\Users\...\vEnv\myWebsite\lib\site-packages\django\core\serializers\json.py", line 67, in Deserializer
    stream_or_string = stream_or_string.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 119485: invalid start byte

之前的操作及提示结果:

(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py makemigrations
No changes detected

(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py migrate
Operations to perform:
  Apply all migrations: admin, aldryn_background_image, auth, bootstrap4_alerts, bootstrap4_badge, bootstrap4_card, bootstrap4_carousel, bootstrap4_collapse, bootstrap4_content, bootstrap4_grid, bootstrap4_jumbotron, bootstrap4_link, bootstrap4_listgroup, bootstrap4_media, bootstrap4_picture, bootstrap4_tabs, bootstrap4_utilities, cms, cmsplugin_soundcloud, contenttypes, djangocms_file, djangocms_googlemap, djangocms_icon, djangocms_link, djangocms_picture, djangocms_style, djangocms_text_ckeditor, djangocms_video, djangocms_vimeo, djangocms_wow, easy_thumbnails, filer, menus, sessions, sites
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying admin.0003_logentry_add_action_flag_choices... OK
  Applying filer.0001_initial... OK
  Applying filer.0002_auto_20150606_2003... OK
  Applying sites.0001_initial... OK
  Applying cms.0001_initial... OK
  Applying cms.0002_auto_20140816_1918... OK
  Applying cms.0003_auto_20140926_2347... OK
  Applying cms.0004_auto_20140924_1038... OK
  Applying cms.0005_auto_20140924_1039... OK
  Applying cms.0006_auto_20140924_1110... OK
  Applying cms.0007_auto_20141028_1559... OK
  Applying cms.0008_auto_20150208_2149... OK
  Applying cms.0008_auto_20150121_0059... OK
  Applying cms.0009_merge... OK
  Applying cms.0010_migrate_use_structure... OK
  Applying cms.0011_auto_20150419_1006... OK
  Applying cms.0012_auto_20150607_2207... OK
  Applying cms.0013_urlconfrevision... OK
  Applying aldryn_background_image.0001_initial... OK
  Applying aldryn_background_image.0002_backgroundimage_background_attachment... OK
  Applying aldryn_background_image.0003_remove_backgroundimage_background_attachment... OK
  Applying aldryn_background_image.0004_backgroundimage_background_attachment... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying auth.0009_alter_user_last_name_max_length... OK
  Applying auth.0010_alter_group_name_max_length... OK
  Applying auth.0011_update_proxy_permissions... OK
  Applying auth.0012_alter_user_first_name_max_length... OK
  Applying cms.0014_auto_20160404_1908... OK
  Applying cms.0015_auto_20160421_0000... OK
  Applying cms.0016_auto_20160608_1535... OK
  Applying bootstrap4_alerts.0001_initial... OK
  Applying bootstrap4_badge.0001_initial... OK
  Applying bootstrap4_card.0001_initial... OK
  Applying filer.0003_thumbnailoption... OK
  Applying filer.0004_auto_20160328_1434... OK
  Applying filer.0005_auto_20160623_1425... OK
  Applying filer.0006_auto_20160623_1627... OK
  Applying filer.0007_auto_20161016_1055... OK
  Applying filer.0008_auto_20171117_1313... OK
  Applying filer.0009_auto_20171220_1635... OK
  Applying filer.0010_auto_20180414_2058... OK
  Applying filer.0011_auto_20190418_0137... OK
  Applying bootstrap4_carousel.0001_initial... OK
  Applying bootstrap4_carousel.0002_bootstrap4carousel_carousel_aspect_ratio... OK
  Applying bootstrap4_carousel.0003_auto_20180610_1102... OK
  Applying bootstrap4_carousel.0004_auto_20190703_0831... OK
  Applying bootstrap4_collapse.0001_initial... OK
  Applying bootstrap4_content.0001_initial... OK
  Applying bootstrap4_content.0002_added_figure... OK
  Applying bootstrap4_grid.0001_initial... OK
  Applying bootstrap4_grid.0002_auto_20180709_0808... OK
  Applying bootstrap4_grid.0003_migrate_column_size... OK
  Applying bootstrap4_grid.0004_remove_bootstrap4gridcolumn_column_size... OK
  Applying bootstrap4_jumbotron.0001_initial... OK
  Applying bootstrap4_link.0001_initial... OK
  Applying bootstrap4_link.0002_add_icons... OK
  Applying bootstrap4_link.0003_icon_updates... OK
  Applying bootstrap4_link.0004_auto_20190703_0831... OK
  Applying bootstrap4_listgroup.0001_initial... OK
  Applying bootstrap4_media.0001_initial... OK
  Applying djangocms_picture.0001_initial... OK
  Applying djangocms_picture.0002_auto_20151018_1927... OK
  Applying djangocms_picture.0003_migrate_to_filer... OK
  Applying djangocms_picture.0004_adapt_fields... OK
  Applying djangocms_picture.0005_reset_null_values... OK
  Applying djangocms_picture.0006_remove_null_values... OK
  Applying djangocms_picture.0007_fix_alignment... OK
  Applying djangocms_picture.0008_picture_use_responsive_image... OK
  Applying bootstrap4_picture.0001_initial... OK
  Applying bootstrap4_picture.0002_bootstrap4picture_use_responsive_image... OK
  Applying bootstrap4_picture.0003_auto_20181212_1055... OK
  Applying bootstrap4_picture.0004_auto_20190703_0831... OK
  Applying bootstrap4_tabs.0001_initial... OK
  Applying bootstrap4_tabs.0002_auto_20180610_1106... OK
  Applying bootstrap4_utilities.0001_initial... OK
  Applying cms.0017_pagetype... OK
  Applying cms.0018_create_pagenode... OK
  Applying cms.0019_set_pagenode... OK
  Applying cms.0020_old_tree_cleanup... OK
  Applying cms.0021_auto_20180507_1432... OK
  Applying cms.0022_auto_20180620_1551... OK
  Applying cmsplugin_soundcloud.0001_initial... OK
  Applying cmsplugin_soundcloud.0002_auto_20210621_1556... OK
  Applying cmsplugin_soundcloud.0003_auto_20210621_1619... OK
  Applying cmsplugin_soundcloud.0004_auto_20210621_1639... OK
  Applying cmsplugin_soundcloud.0005_auto_20210621_1732... OK
  Applying djangocms_file.0001_initial... OK
  Applying djangocms_file.0002_auto_20151202_1551... OK
  Applying djangocms_file.0003_remove_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_file.0004_set_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_file.0005_auto_20160119_1534... OK
  Applying djangocms_file.0006_migrate_to_filer... OK
  Applying djangocms_file.0007_adapted_fields... OK
  Applying djangocms_file.0008_add_folder... OK
  Applying djangocms_file.0009_fixed_null_fields... OK
  Applying djangocms_file.0010_removed_null_fields... OK
  Applying djangocms_file.0011_auto_20181211_0357... OK
  Applying djangocms_googlemap.0001_initial... OK
  Applying djangocms_googlemap.0002_auto_20160622_1031... OK
  Applying djangocms_googlemap.0003_auto_20160825_1829... OK
  Applying djangocms_googlemap.0004_adapted_fields... OK
  Applying djangocms_googlemap.0005_create_nested_plugins... OK
  Applying djangocms_googlemap.0006_remove_fields... OK
  Applying djangocms_googlemap.0007_reset_null_values... OK
  Applying djangocms_googlemap.0008_removed_null_fields... OK
  Applying djangocms_googlemap.0009_googlemapmarker_icon... OK
  Applying djangocms_googlemap.0010_auto_20190718_1021... OK
  Applying djangocms_icon.0001_initial... OK
  Applying djangocms_icon.0002_auto_20190218_2017... OK
  Applying djangocms_link.0001_initial... OK
  Applying djangocms_link.0002_auto_20140929_1705... OK
  Applying djangocms_link.0003_auto_20150212_1310... OK
  Applying djangocms_link.0004_auto_20150708_1133... OK
  Applying djangocms_link.0005_auto_20151003_1710... OK
  Applying djangocms_link.0006_remove_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_link.0007_set_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_link.0008_link_attributes... OK
  Applying djangocms_link.0009_auto_20160705_1344... OK
  Applying djangocms_link.0010_adapted_fields... OK
  Applying djangocms_link.0011_fixed_null_values... OK
  Applying djangocms_link.0012_removed_null... OK
  Applying djangocms_link.0013_fix_hostname... OK
  Applying djangocms_link.0014_link_file_link... OK
  Applying djangocms_link.0015_auto_20190621_0407... OK
  Applying djangocms_picture.0009_auto_20181212_1003... OK
  Applying djangocms_picture.0010_auto_20190627_0432... OK
  Applying djangocms_picture.0011_auto_20190314_1536... OK
  Applying djangocms_style.0001_initial... OK
  Applying djangocms_style.0002_set_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_style.0003_adapted_fields... OK
  Applying djangocms_style.0004_use_positive_small_integer_field... OK
  Applying djangocms_style.0005_reset_null_values... OK
  Applying djangocms_style.0006_removed_null_fields... OK
  Applying djangocms_style.0007_style_template... OK
  Applying djangocms_text_ckeditor.0001_initial... OK
  Applying djangocms_text_ckeditor.0002_remove_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_text_ckeditor.0003_set_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_text_ckeditor.0004_auto_20160706_1339... OK
  Applying djangocms_video.0001_initial... OK
  Applying djangocms_video.0002_set_related_name_for_cmsplugin_ptr... OK
  Applying djangocms_video.0003_field_adaptions... OK
  Applying djangocms_video.0004_move_to_attributes... OK
  Applying djangocms_video.0005_migrate_to_filer... OK
  Applying djangocms_video.0006_field_adaptions... OK
  Applying djangocms_video.0007_create_nested_plugin... OK
  Applying djangocms_video.0008_reset_null_values... OK
  Applying djangocms_video.0009_removed_null_values... OK
  Applying djangocms_video.0010_videoplayer_parameters... OK
  Applying djangocms_vimeo.0001_initial... OK
  Applying djangocms_vimeo.0002_update_related_name... OK
  Applying djangocms_wow.0001_initial... OK
  Applying djangocms_wow.0002_auto_20210611_1217... OK
  Applying djangocms_wow.0003_auto_20210621_1750... OK
  Applying easy_thumbnails.0001_initial... OK
  Applying easy_thumbnails.0002_thumbnaildimensions... OK
  Applying filer.0012_file_mime_type... OK
  Applying menus.0001_initial... OK
  Applying sessions.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK

(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py shell
Python 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()
(376, {'auth.Permission': 300, 'contenttypes.ContentType': 76})

当我登录网站时:

(myWebsite) C:\Users\...\vEnv\myWebsite\project\website>python manage.py runserver
Performing system checks...
System check identified no issues (0 silenced).
August 25, 2021 - 17:21:21
Django version 3.1.11, using settings 'regSite.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CTRL-BREAK.
[25/Aug/2021 17:21:33] "GET /en/ HTTP/1.1" 200 13923
[25/Aug/2021 17:21:33] "GET /static/cms/css/3.8.0/cms.base.css HTTP/1.1" 302 0
[25/Aug/2021 17:21:33] "GET /static/cms/js/dist/3.8.0/bundle.toolbar.min.js HTTP/1.1" 302 0
[25/Aug/2021 17:21:33] "GET /en/static/cms/css/3.8.0/cms.base.css/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:33] "GET /en/static/cms/js/dist/3.8.0/bundle.toolbar.min.js/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:33] "GET /static/cms/css/3.8.0/cms.welcome.css HTTP/1.1" 302 0
[25/Aug/2021 17:21:34] "GET /en/static/cms/css/3.8.0/cms.welcome.css/ HTTP/1.1" 404 179
[25/Aug/2021 17:21:34] "GET /static/cms/js/dist/3.8.0/bundle.toolbar.min.js HTTP/1.1" 302 0
[25/Aug/2021 17:21:34] "GET /en/static/cms/js/dist/3.8.0/bundle.toolbar.min.js/ HTTP/1.1" 404 179

这里是 link,您可以在其中找到更改数据库的步骤。它是关于 SQLite 到 Mysql,但我认为这个过程会非常相似。

注意:在对数据库执行任何操作之前,请不要忘记备份数据。

感谢 Ian Shelvington 和 Tim Roberts,我在记事本中打开数据库并使用 UTF-8 编码另存为。然后我又运行python manage.py loaddata db.json

我的解决方案是自定义命令:

proj\proj\management\commands\db_load_portable_data.py

django 3.2,python3.10

import json

from django.core.management import BaseCommand, call_command
from django.conf import settings


def fix_dumpdata_encoding(fixture_path: str):
    with open(fixture_path) as f:
        data = json.load(f)
    with open(fixture_path, 'w', encoding='utf-8') as f:
        json.dump(data, f, ensure_ascii=False, indent=4)


class Command(BaseCommand):
    def handle(self, *args, **opts):
        home_dir = settings.BASE_DIR
        dump_path = 'proj/fixtures/tmp'
        app_model_set = (
            'auth.User',
            'auth.Permission',
            'auth.Group',
        )
        for app_model in app_model_set:
            output_path = f'{home_dir}/{dump_path}/{app_model}.json'
            print(output_path)
            call_command('dumpdata', app_model,
                         indent=4, natural_foreign=True, natural_primary=True, output=output_path)
            fix_dumpdata_encoding(output_path)
            print()
        print(f'Dump done, path: {dump_path}')