在 Clojure 中的 Ring Web 应用程序中生成并流式传输 zip 文件

Question

我有一个 Ring 处理程序需要：

压缩几个文件
将 Zip 流式传输到客户端。

现在我可以正常工作了，但只有第一个压缩条目被流式传输，然后是 stalls/stops。我感觉跟flushing/streaming有关系是错误的

这是我的（compojure）处理程序：

(GET "/zip" {:as request}
            :query-params [order-id   :- s/Any]
            (stream-lessons-zip (read-string order-id) (:db request) (:auth-user request)))

这是 stream-lessons-zip 函数：

(defn stream-lessons-zip
  []
  (let [lessons ...];... not shown

  {:status 200
   :headers {"Content-Type" "application/zip, application/octet-stream"
             "Content-Disposition" (str "attachment; filename=\"files.zip\"")
   :body (futil/zip-lessons lessons)}))

我使用管道输入流来进行流式传输，如下所示：

(defn zip-lessons
 "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons]
(let [paths (map #(select-keys % [:file_path :file_name]) lessons)]
(ring-io/piped-input-stream
  (fn [output-stream]
    ; build a zip-output-stream from a normal output-stream
    (with-open [zip-output-stream (ZipOutputStream. output-stream)]
      (doseq [{:keys [file_path file_name] :as p} paths]
        (let [f (cio/file file_path)]
          (.putNextEntry zip-output-stream (ZipEntry. file_name)) 
          (cio/copy f zip-output-stream)
          (.closeEntry zip-output-stream))))))))

所以我已经确认 'lessons' 向量包含大约 4 个条目，但 zip 文件只包含 1 个条目。此外，Chrome 似乎没有 'finalize' 下载，即。它认为它仍在下载。

我该如何解决这个问题？

Answer 1

听起来 http-kit 不支持使用阻塞 IO 生成有状态流。非状态流可以这样实现：

http://www.http-kit.org/server.html#async

不接受使用阻塞 IO 引入有状态流的 PR：

https://github.com/http-kit/http-kit/pull/181

听起来探索的选项是使用 ByteArrayOutputStream 将 zip 文件完全渲染到内存，然后 return 生成的缓冲区。如果此端点的流量不高并且它生成的 zip 文件不大（< 1 gb），那么这可能会起作用。

Answer 2

所以，几年过去了，但该代码仍在生产环境中运行（即它有效）。所以我当时让它起作用了，但忘了在这里提到它（并且忘记了为什么它起作用，老实说，..它非常trial/error）。

现在是代码：

(defn zip-lessons
  "Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
  [lessons {:keys [firstname surname order_favorite_name company_name] :as annotation
            :or {order_favorite_name ""
                 company_name ""
                 firstname ""
                 surname ""}}]
  (debug "zipping lessons" (count lessons))
  (let [paths (map #(select-keys % [:file_path :file_name :folder_number]) lessons)]
    (ring-io/piped-input-stream
      (fn [output-stream]
        ; build a zip-output-stream from a normal output-stream
        (with-open [zip-output-stream (ZipOutputStream. output-stream)]
          (doseq [{:keys [file_path file_name folder_number] :as p} paths]
            (let [f (cio/as-file file_path)
                  baos (ByteArrayOutputStream.)]
              (if (.exists f)
                (do
                  (debug "Adding entry to zip:" file_name "at" file_path)
                  (let [zip-entry (ZipEntry. (str (if folder_number (str folder_number "/") "") file_name))]
                    (.putNextEntry zip-output-stream zip-entry)

                   
                    (.close baos)
                    (.writeTo baos zip-output-stream)
                    (.closeEntry zip-output-stream)
                    (.flush zip-output-stream)
                    (debug "flushed")))
                (warn "File '" file_name "' at '" file_path "' does not exist, not adding to zip file!"))))
          (.flush zip-output-stream)
          (.flush output-stream)
          (.finish zip-output-stream)
          (.close zip-output-stream))))))

在 Clojure 中的 Ring Web 应用程序中生成并流式传输 zip 文件

Generate and stream a zip-file in a Ring web app in Clojure

zip

clojure

stream

ring