上传到S3的批处理文件
Batch file to upload to S3
我需要开发一个批处理文件，它将多个小文本文件 (4KB – 100KB) 发送到 S3 存储桶中的一个文件夹。受来自 http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash 的 bash 脚本的启发，我开发了下面列出的批处理脚本。此脚本使用以下依赖项：
openssl.exe（以及两个库 libeay32.dll 和 ssleay32.dll）
b64.exe (https://sourceforge.net/projects/base64/)
curl.exe
grep.exe
效果不错；但是，有时一个文件或多个文件无法上传，我会收到“HTTP/1.1 403 Forbidden / SignatureDoesNotMatch。”这似乎是一个签名问题，可能是在创建签名时。这种行为似乎没有规律可循。为了测试，我使用 8 个文本文件上传。我可以 运行 多次，导致每次都上传所有文件，而下一次脚本将只上传 8 个中的 5、6 或 7 个。我有点不知所措，为什么会这样。如有任何建议，我们将不胜感激。
@echo off
setlocal enableDelayedExpansion
for /f %%f in ('dir /b *.txt') do (
REM Set date/time variables
for /f "tokens=1-4 delims=/ " %%a in ('date /t') do set dow=%%a&&set day=%%c&& set month=%%b&&set year=%%d
for /f "tokens=1-4 delims=:,. " %%h in ('echo %time%') do set hour=%%h&set min=%%i&set sec1=%%j&set sec2=%%k
REM Obtain three letter month value for DateValue HTTP Header
IF !month! == 01 set mname=Jan
IF !month! == 02 set mname=Feb
IF !month! == 03 set mname=Mar
IF !month! == 04 set mname=Apr
IF !month! == 05 set mname=May
IF !month! == 06 set mname=Jun
IF !month! == 07 set mname=Jul
IF !month! == 08 set mname=Aug
IF !month! == 09 set mname=Sep
IF !month! == 10 set mname=Oct
IF !month! == 11 set mname=Nov
IF !month! == 12 set mname=Dec
REM Set time as a 4 digit value if leading zero does not exist
for /f "tokens=1" %%u in ('echo %time%') do set t=%%u
IF "!hour:~1,1!"==":" set t=0!hour!
REM Obtain the ActiveBias value and convert to decimal
for /f "tokens=3" %%a in ('reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v ActiveTimeBias ^| grep -i "ActiveTimeBias"') do set /a abias=%%a
REM Set the + or - sign variable to reflect the timezone offset
IF "!abias:~0,1!"=="-" (set si=+) ELSE (set si=-)
for /f "tokens=1 delims=-" %%t in ('echo !abias!') do set tzc=%%t
REM Calculate to obtain floating points (decimal values)
set /a tzd=100*!tzc!/60
REM Calculate the active bias to obtain the hour
set /a tze=!tzc!/60
REM Set the minutes based on the result of the floating point calculation
IF "!tzd!"=="0" (set en=00 && set si=)
IF "!tzd:~1!"=="00" (set en=00) ELSE IF "!tzd:~2!"=="00" (set en=00 && set tz=!tzd:~0,2!)
IF "!tzd:~1!"=="50" (set en=30) ELSE IF "!tzd:~2!"=="50" (set en=30 && set tz=!tzd:~0,2!)
IF "!tzd:~1!"=="75" (set en=45) ELSE IF "!tzd:~2!"=="75" (set en=45 && set tz=!tzd:~0,2!)
REM Adding a 0 to the beginning of a single digit hour value
IF !tze! LSS 10 (set tz=0!tze!)
REM Set the date/timestamp to meet required format
set dateValue=!dow!, !day! !mname! !year! !hour!:!min!:!sec1! !si!!tz!!en!
REM Preparing the HTTP header field
set file=%%f
set bucket=yourbucketname
set resource=/!bucket!/upload/!file!
set contentType=text/plain
REM You MUST have two returns after set NL=^
setlocal enableDelayedExpansion
set NL=^
set stringToSign=PUT!NL!!NL!!contentType!!NL!!dateValue!!NL!!resource!
<nul set /p ".=!stringToSign!" > put.tmp
set S3KEY="Your S3 Key Here"
set S3SECRET="Your S3 Secrect Key Here"
for /f "tokens=*" %%a in ('type put.tmp ^| openssl sha1 -hmac !S3SECRET! -binary ^| b64 -e') do set signature=%%a
REM Sending the data
curl -vvv --no-alpn --http2 -1 -S -X PUT -T "!file!" -H "Host: !bucket!.s3.amazonaws.com" -H "Date: !dateValue!" -H "Content-Type: !contentType!" -H "Authorization: AWS !S3KEY!:!signature!" "https://!bucket!.s3.amazonaws.com/upload/!file!"
REM Reset Variables
set "day="
set "month="
set "year="
set "retry="
set "dow="
set "hour="
set "min="
set "sec1="
set "sec2="
set "dateValue="
set "file="
set "mname="
set "bucket="
set "resource="
set "contentType="
set "stringToSign="
set "S3KEY="
set "S3SECRET="
set "t="
del put.tmp
)
为什么不使用具有显式 S3 支持的 Windows 命令行客户端？无论如何，您的批处理文件离不开第 3 方依赖项。
亚马逊有自己的aws-cli。
如果您需要更轻量级的解决方案，请使用 WinSCP：
winscp.com /log=s3.log /command ^
"open s3://%S3KEY%:%S3SECRET%@s3.amazonaws.com/" ^
"put *.txt /%BUCKET%/" ^
"exit"
你需要URL-encode special characters in the credentials. WinSCP GUI can generate an S3 script template，就像上面那个一样。
或者，自 WinSCP 5.19 起，您可以使用不需要任何编码的 -username and -password 开关：
winscp.com /log=s3.log /command ^
"open s3://s3.amazonaws.com/ -username=%S3KEY% -password=%S3SECRET%" ^
"put *.txt /%BUCKET%/" ^
"exit"
(我是WinSCP的作者)
我需要开发一个批处理文件，它将多个小文本文件 (4KB – 100KB) 发送到 S3 存储桶中的一个文件夹。受来自 http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash 的 bash 脚本的启发，我开发了下面列出的批处理脚本。此脚本使用以下依赖项：
openssl.exe（以及两个库 libeay32.dll 和 ssleay32.dll）
b64.exe (https://sourceforge.net/projects/base64/)
curl.exe
grep.exe
效果不错；但是，有时一个文件或多个文件无法上传，我会收到“HTTP/1.1 403 Forbidden / SignatureDoesNotMatch。”这似乎是一个签名问题，可能是在创建签名时。这种行为似乎没有规律可循。为了测试，我使用 8 个文本文件上传。我可以 运行 多次，导致每次都上传所有文件，而下一次脚本将只上传 8 个中的 5、6 或 7 个。我有点不知所措，为什么会这样。如有任何建议，我们将不胜感激。
@echo off
setlocal enableDelayedExpansion
for /f %%f in ('dir /b *.txt') do (
REM Set date/time variables
for /f "tokens=1-4 delims=/ " %%a in ('date /t') do set dow=%%a&&set day=%%c&& set month=%%b&&set year=%%d
for /f "tokens=1-4 delims=:,. " %%h in ('echo %time%') do set hour=%%h&set min=%%i&set sec1=%%j&set sec2=%%k
REM Obtain three letter month value for DateValue HTTP Header
IF !month! == 01 set mname=Jan
IF !month! == 02 set mname=Feb
IF !month! == 03 set mname=Mar
IF !month! == 04 set mname=Apr
IF !month! == 05 set mname=May
IF !month! == 06 set mname=Jun
IF !month! == 07 set mname=Jul
IF !month! == 08 set mname=Aug
IF !month! == 09 set mname=Sep
IF !month! == 10 set mname=Oct
IF !month! == 11 set mname=Nov
IF !month! == 12 set mname=Dec
REM Set time as a 4 digit value if leading zero does not exist
for /f "tokens=1" %%u in ('echo %time%') do set t=%%u
IF "!hour:~1,1!"==":" set t=0!hour!
REM Obtain the ActiveBias value and convert to decimal
for /f "tokens=3" %%a in ('reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation /v ActiveTimeBias ^| grep -i "ActiveTimeBias"') do set /a abias=%%a
REM Set the + or - sign variable to reflect the timezone offset
IF "!abias:~0,1!"=="-" (set si=+) ELSE (set si=-)
for /f "tokens=1 delims=-" %%t in ('echo !abias!') do set tzc=%%t
REM Calculate to obtain floating points (decimal values)
set /a tzd=100*!tzc!/60
REM Calculate the active bias to obtain the hour
set /a tze=!tzc!/60
REM Set the minutes based on the result of the floating point calculation
IF "!tzd!"=="0" (set en=00 && set si=)
IF "!tzd:~1!"=="00" (set en=00) ELSE IF "!tzd:~2!"=="00" (set en=00 && set tz=!tzd:~0,2!)
IF "!tzd:~1!"=="50" (set en=30) ELSE IF "!tzd:~2!"=="50" (set en=30 && set tz=!tzd:~0,2!)
IF "!tzd:~1!"=="75" (set en=45) ELSE IF "!tzd:~2!"=="75" (set en=45 && set tz=!tzd:~0,2!)
REM Adding a 0 to the beginning of a single digit hour value
IF !tze! LSS 10 (set tz=0!tze!)
REM Set the date/timestamp to meet required format
set dateValue=!dow!, !day! !mname! !year! !hour!:!min!:!sec1! !si!!tz!!en!
REM Preparing the HTTP header field
set file=%%f
set bucket=yourbucketname
set resource=/!bucket!/upload/!file!
set contentType=text/plain
REM You MUST have two returns after set NL=^
setlocal enableDelayedExpansion
set NL=^
set stringToSign=PUT!NL!!NL!!contentType!!NL!!dateValue!!NL!!resource!
<nul set /p ".=!stringToSign!" > put.tmp
set S3KEY="Your S3 Key Here"
set S3SECRET="Your S3 Secrect Key Here"
for /f "tokens=*" %%a in ('type put.tmp ^| openssl sha1 -hmac !S3SECRET! -binary ^| b64 -e') do set signature=%%a
REM Sending the data
curl -vvv --no-alpn --http2 -1 -S -X PUT -T "!file!" -H "Host: !bucket!.s3.amazonaws.com" -H "Date: !dateValue!" -H "Content-Type: !contentType!" -H "Authorization: AWS !S3KEY!:!signature!" "https://!bucket!.s3.amazonaws.com/upload/!file!"
REM Reset Variables
set "day="
set "month="
set "year="
set "retry="
set "dow="
set "hour="
set "min="
set "sec1="
set "sec2="
set "dateValue="
set "file="
set "mname="
set "bucket="
set "resource="
set "contentType="
set "stringToSign="
set "S3KEY="
set "S3SECRET="
set "t="
del put.tmp
)
为什么不使用具有显式 S3 支持的 Windows 命令行客户端？无论如何，您的批处理文件离不开第 3 方依赖项。
亚马逊有自己的aws-cli。
如果您需要更轻量级的解决方案，请使用 WinSCP：
winscp.com /log=s3.log /command ^
"open s3://%S3KEY%:%S3SECRET%@s3.amazonaws.com/" ^
"put *.txt /%BUCKET%/" ^
"exit"
你需要URL-encode special characters in the credentials. WinSCP GUI can generate an S3 script template，就像上面那个一样。
或者，自 WinSCP 5.19 起，您可以使用不需要任何编码的 -username and -password 开关：
winscp.com /log=s3.log /command ^
"open s3://s3.amazonaws.com/ -username=%S3KEY% -password=%S3SECRET%" ^
"put *.txt /%BUCKET%/" ^
"exit"
(我是WinSCP的作者)