在 FTP 服务器上扩展 Python 的 os.walk 功能

Question

如何让 os.walk 遍历 FTP 数据库（位于远程服务器上）的目录树？现在代码的结构方式是（提供注释）：

import fnmatch, os, ftplib

def find(pattern, startdir=os.curdir): #find function taking variables for both desired file and the starting directory
    for (thisDir, subsHere, filesHere) in os.walk(startdir): #each of the variables change as the directory tree is walked
        for name in subsHere + filesHere: #going through all of the files and subdirectories
            if fnmatch.fnmatch(name, pattern): #if the name of one of the files or subs is the same as the inputted name
                fullpath = os.path.join(thisDir, name) #fullpath equals the concatenation of the directory and the name
                yield fullpath #return fullpath but anew each time

def findlist(pattern, startdir = os.curdir, dosort=False):
    matches = list(find(pattern, startdir)) #find with arguments pattern and startdir put into a list data structure
    if dosort: matches.sort() #isn't dosort automatically False? Is this statement any different from the same thing but with a line in between
    return matches

#def ftp(
#specifying where to search.

if __name__ == '__main__':
    import sys
    namepattern, startdir = sys.argv[1], sys.argv[2]
    for name in find(namepattern, startdir): print (name)

我在想我需要定义一个新函数（即 def ftp()）以将此功能添加到上面的代码中。但是，恐怕 os.walk 函数在默认情况下只会遍历代码来自运行的计算机的目录树。

有什么方法可以扩展 os.walk 的功能，使其能够遍历远程目录树（通过 FTP）？

Answer 1

我假设这就是你想要的......虽然我真的不知道

ssh = paramiko.SSHClient()
ssh.connect(server, username=username, password=password)
ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command("locate my_file.txt")
print ssh_stdout

这将要求远程服务器具有 mlocate 软件包 `sudo apt-get install mlocate;sudo updatedb();

Answer 2

您所需要的只是利用 python 的 ftplib 模块。由于 os.walk() 基于广度优先搜索算法，因此您需要在每次迭代中找到目录和文件名，然后从第一个目录继续递归遍历。我实现了 this algorithm about 2 years ago for using as the heart of FTPwalker，这是通过 FTP.

遍历超大目录树的最佳包

from os import path as ospath


class FTPWalk:
    """
    This class is contain corresponding functions for traversing the FTP
    servers using BFS algorithm.
    """
    def __init__(self, connection):
        self.connection = connection

    def listdir(self, _path):
        """
        return files and directory names within a path (directory)
        """

        file_list, dirs, nondirs = [], [], []
        try:
            self.connection.cwd(_path)
        except Exception as exp:
            print ("the current path is : ", self.connection.pwd(), exp.__str__(),_path)
            return [], []
        else:
            self.connection.retrlines('LIST', lambda x: file_list.append(x.split()))
            for info in file_list:
                ls_type, name = info[0], info[-1]
                if ls_type.startswith('d'):
                    dirs.append(name)
                else:
                    nondirs.append(name)
            return dirs, nondirs

    def walk(self, path='/'):
        """
        Walk through FTP server's directory tree, based on a BFS algorithm.
        """
        dirs, nondirs = self.listdir(path)
        yield path, dirs, nondirs
        for name in dirs:
            path = ospath.join(path, name)
            yield from self.walk(path)
            # In python2 use:
            # for path, dirs, nondirs in self.walk(path):
            #     yield path, dirs, nondirs
            self.connection.cwd('..')
            path = ospath.dirname(path)

现在要使用这个 class，您可以简单地使用 ftplib 模块创建一个连接对象，并将该对象传递给 FTPWalk 对象，然后循环遍历 walk() 函数：

In [2]: from test import FTPWalk

In [3]: import ftplib

In [4]: connection = ftplib.FTP("ftp.uniprot.org")

In [5]: connection.login()
Out[5]: '230 Login successful.'

In [6]: ftpwalk = FTPWalk(connection)

In [7]: for i in ftpwalk.walk():
            print(i)
   ...:     
('/', ['pub'], [])
('/pub', ['databases'], ['robots.txt'])
('/pub/databases', ['uniprot'], [])
('/pub/databases/uniprot', ['current_release', 'previous_releases'], ['LICENSE', 'current_release/README', 'current_release/knowledgebase/complete', 'previous_releases/', 'current_release/relnotes.txt', 'current_release/uniref'])
('/pub/databases/uniprot/current_release', ['decoy', 'knowledgebase', 'rdf', 'uniparc', 'uniref'], ['README', 'RELEASE.metalink', 'changes.html', 'news.html', 'relnotes.txt'])
...
...
...

Answer 3

我在 FTP 上需要像 os.walk 这样的函数，但那里没有，所以我认为编写它会很有用，为了将来参考，您可以找到最新版本 here

顺便说一句，这里是执行此操作的代码：

def FTP_Walker(FTPpath,localpath):
    os.chdir(localpath)
    current_loc = os.getcwd()
    for item in ftp.nlst(FTPpath):
        if not is_file(item):
            yield from FTP_Walker(item,current_loc)

        elif is_file(item):
            yield(item)
            current_loc = localpath
        else:
            print('this is a item that i could not process')
    os.chdir(localpath)
    return


def is_file(filename):
    current = ftp.pwd()
    try:
        ftp.cwd(filename)
    except Exception as e :
        ftp.cwd(current)
        return True

    ftp.cwd(current)
    return False

使用方法：

首先连接到您的主机：

host_address = "my host address"
user_name = "my username"
password = "my password"


ftp = FTP(host_address)
ftp.login(user=user_name,passwd=password)

现在你可以这样调用函数了：

ftpwalk = FTP_Walker("FTP root path","path to local") # I'm not using path to local yet but in future versions I will improve it. so you can just path an '/' to it

然后要打印和下载文件，您可以这样做：

for item in ftpwalk:
ftp.retrbinary("RETR "+item, open(os.path.join(current_loc,item.split('/')[-1]),"wb").write) #it is downloading the file 
print(item) # it will print the file address

（我会尽快为它编写更多功能，所以如果您需要一些特定的东西或有任何对用户有用的想法，我会很高兴听到）

Answer 4

我写了一个库pip install walk-sftp。虽然事件名为 walk-sftp，但我包含了一个 WalkFTP class，它允许您按 start_date 文件和 end_date 文件进行过滤。您甚至可以传入一个 processing_function returns True 或 False 来查看您清理和存储数据的过程是否有效。它还有一个使用 pickle 的日志参数（传递文件名）并跟踪任何进度，这样您就不会覆盖或必须跟踪日期，从而使回填更容易。

https://pypi.org/project/walk-sftp/

在 FTP 服务器上扩展 Python 的 os.walk 功能

Extending Python's os.walk function on FTP server

python

algorithm

ftp

os.walk

使用方法：