当长度已知时从 Lazy ByteString 构造 RequestBodyStream

Question

我正在尝试调整此 AWS S3 upload code to handle Lazy ByteString where length is already known (so that it is not forced to be read in its entirety in memory - it comes over the network where length is sent beforehand). It seems I have to define a GivesPopper function over Lazy ByteString to convert it to RequestBodyStream. Because of the convoluted way GivesPopper is defined, I am not sure how to write it for Lazy ByteString. Will appreciate pointers on how to write it. Here 是如何编写以从文件中读取的：

let file ="test"
-- streams large file content, without buffering more than 10k in memory
let streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240

上面代码中的

streamer 是 GivesPopper () 类型，如果我没看错的话。给定一个已知长度 len 的 Lazy ByteString，在其上编写 GivesPopper 函数的好方法是什么？我们一次可以读取一个块。

Answer 1

这是您要找的吗？

import qualified Data.ByteString as S
import qualified Data.ByteString.Lazy as L
import System.IO

file = "test"
-- original streamer for feeding a sink from a file
streamer :: (IO S.ByteString -> IO r) -> IO r
streamer sink = withFile file ReadMode $ \h -> sink $ S.hGet h 10240

-- feed a lazy ByteString to sink    
lstreamer :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
lstreamer lbs sink = sink (return (L.toStrict lbs))

lstreamer 类型检查，但可能不会完全按照您的要求进行。每次接收器调用它时，它只是 returns 相同的数据。另一方面 S.hGet h ... 最终会 return 空字符串。

这是一个解决方案，它使用 IORef 来跟踪我们是否应该开始 returning 空字符串：

import Data.IORef

mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
  ref <- newIORef False
  let fetch :: IO S.ByteString
      fetch = do sent <- readIORef ref
                 writeIORef ref True
                 if sent
                   then return S.empty
                   else return (L.toStrict lbs)
  sink fetch

这里fetch是获取下一个块的动作。第一次调用它时，您将获得原始的惰性字节串（严格化）。后续调用将始终 return 空字符串。

更新

一次分发少量的方法如下：

mklstream :: L.ByteString -> (IO S.ByteString -> IO r) -> IO r
mklstream lbs sink = do
  ref <- newIORef (L.toChunks lbs)
  let fetch :: IO S.ByteString
      fetch = do chunks <- readIORef ref
                 case chunks of
                   [] -> return S.empty
                   (c:cs) -> do writeIORef ref cs
                                return c
  sink fetch

当长度已知时从 Lazy ByteString 构造 RequestBodyStream

Constructing RequestBodyStream from Lazy ByteString when length is known

haskell

bytestring

http-conduit