Error with DVC on Google Colab - dvc.scm.CloneError: Failed to clone repo

Question

我在尝试运行 Google Colab 上的“dvc pull”时遇到问题。我有两个存储库（我们称它们为 A 和 B），其中存储库 A 用于我的机器学习代码，存储库 B 用于我的数据集。

我已经使用 DVC（使用 gdrive 作为我的远程存储）成功地将我的数据集推送到存储库 B，并且我还成功地运行“dvc 导入”（以及“dvc pull/update") 在我的存储库 A 的本地项目上。

当我将 colab 用于运行我的项目时，问题就来了。所以我做了以下事情：

在 colab 上创建了一个新笔记本
成功 git- 克隆了我的机器学习项目（存储库 A）
运行 "!pip install dvc"
运行 "!dvc pull -v"（这是导致错误的原因）

在第 4 步，我得到了错误（这是完整的堆栈跟踪。请注意，出于保密原因，我更改了堆栈跟踪中的 repo URL）

2022-03-08 08:53:31,863 DEBUG: Adding '/content/<my_project_A>/.dvc/config.local' to gitignore file.
2022-03-08 08:53:31,866 DEBUG: Adding '/content/<my_project_A>/.dvc/tmp' to gitignore file.
2022-03-08 08:53:31,866 DEBUG: Adding '/content/<my_project_A>/.dvc/cache' to gitignore file.
2022-03-08 08:53:31,916 DEBUG: Creating external repo https://gitlab.com/<my-dataset-repo-B>.git@3a3f4559efabff8ec74486da39b86688d1b98d75
2022-03-08 08:53:31,916 DEBUG: erepo: git clone 'https://gitlab.com/<my-dataset-repo-B>.git' to a temporary dir
Everything is up to date.
2022-03-08 08:53:32,154 ERROR: failed to pull data from the cloud - Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x7y7xgedvc-clone'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/backend/gitpython.py", line 185, in clone
    tmp_repo = clone_from()
  File "/usr/local/lib/python3.7/dist-packages/git/repo/base.py", line 1148, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, multi_options, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/git/repo/base.py", line 1079, in _clone
    finalize_process, decode_streams=False)
  File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 176, in handle_process_output
    return finalizer(process)
  File "/usr/local/lib/python3.7/dist-packages/git/util.py", line 386, in finalize_process
    proc.wait(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/git/cmd.py", line 502, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --no-single-branch --progress https://gitlab.com/<my-dataset-repo-B>.git /tmp/tmp2x7y7xgedvc-clone

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/dvc/scm.py", line 104, in clone
    return Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/__init__.py", line 121, in clone
    backend.clone(url, to_path, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/scmrepo/git/backend/gitpython.py", line 190, in clone
    raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x7y7xgedvc-clone'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/dvc/command/data_sync.py", line 41, in run
    glob=self.args.glob,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/pull.py", line 38, in pull
    run_cache=run_cache,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/fetch.py", line 50, in fetch
    revs=revs,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/__init__.py", line 437, in used_objs
    with_deps=with_deps,
  File "/usr/local/lib/python3.7/dist-packages/dvc/repo/index.py", line 190, in used_objs
    filter_info=filter_info,
  File "/usr/local/lib/python3.7/dist-packages/dvc/stage/__init__.py", line 660, in get_used_objs
    for odb, objs in out.get_used_objs(*args, **kwargs).items():
  File "/usr/local/lib/python3.7/dist-packages/dvc/output.py", line 918, in get_used_objs
    return self.get_used_external(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/output.py", line 973, in get_used_external
    return dep.get_used_objs(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/dependency/repo.py", line 94, in get_used_objs
    used, _ = self._get_used_and_obj(**kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/dependency/repo.py", line 108, in _get_used_and_obj
    locked=locked, cache_dir=local_odb.cache_dir
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 35, in external_repo
    path = _cached_clone(url, rev, for_write=for_write)
  File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 155, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev, for_write=for_write)
  File "/usr/local/lib/python3.7/dist-packages/funcy/decorators.py", line 45, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/usr/local/lib/python3.7/dist-packages/funcy/flow.py", line 274, in wrap_with
    return call()
  File "/usr/local/lib/python3.7/dist-packages/funcy/decorators.py", line 66, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.7/dist-packages/dvc/external_repo.py", line 220, in _clone_default_branch
    git = clone(url, clone_path)
  File "/usr/local/lib/python3.7/dist-packages/dvc/scm.py", line 106, in clone
    raise CloneError(str(exc))
dvc.scm.CloneError: Failed to clone repo 'https://gitlab.com/<my-dataset-repo-B>.git' to '/tmp/tmp2x7y7xgedvc-clone'
------------------------------------------------------------
2022-03-08 08:53:32,161 DEBUG: Analytics is enabled.
2022-03-08 08:53:32,192 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp4x5js0dk']'
2022-03-08 08:53:32,193 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp4x5js0dk']'

顺便说一句，这就是我克隆 git 存储库 (repo A)

的方式

!git config - global user.name "Zharfan"
!git config - global user.email "zharfan@myemail.com"
!git clone https://<MyTokenName>:<MyToken>@link-to-my-repo-A.git

有谁知道为什么？任何帮助将不胜感激。提前致谢！

Answer 1

总结评论线程中的讨论。

很可能是因为 DVC 无法访问 GitLab 上的私有存储库。（错误消息含糊不清，应予以修复。）

以同样的方式你将无法运行:

!git clone https://gitlab.com/org/<private-repo>

它也是 returns 一个相当晦涩的错误：

Cloning into '<private-repo>'...
fatal: could not read Username for 'https://gitlab.com': No such device or address

（我认为这与 Colab 中的 tty 设置有关？）

解决此问题的最佳方法是使用 SSH，例如 here。

Error with DVC on Google Colab - dvc.scm.CloneError: Failed to clone repo

Error with DVC on Google Colab - dvc.scm.CloneError: Failed to clone repo

git

dataset

google-colaboratory

dvc