I want to stream multipart file directly to AWSS3 instead of saving file to disk at first. Receiving file of type *multipart.FileHeader in my golang REST API. Now I want to stream this to AWSS3, using s3manager upload method where it's required to pass *bytes.Reader in body param. I have seen an example of uploading file to AWSS3, written in a way that opening a file os.Open("filepath") of type *os.File from disk,getting *bytes.Reader out of it and passing to AWSS3. Now I am struggling to convert this *multipart.FileHeader to *os.File.
2 Answers
Call Open on the multipart.FileHeader
. It will return a multipart.File
which will provide a reader to the file (which will be held in memory).
-
To add a little more
ParseMultipartForm
will let you control when to have the file in memory, and when they are written to temp. From the docs: a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.– djdDec 4, 2016 at 11:17 -
1
-
@RaviTejaBathula if you really need an *os.File, you need to write a file somewhere. However, there should be no reason to really require an *os.File.– JimBDec 4, 2016 at 17:15
-
@RaviTejaBathula if you want to prevent the files from hitting disk at all, you have two options. (1) set
maxMemory
to a sufficiently large value that it can hold the files in memory (you can use other parts of thehttp
package to limit the size of the incoming request). (2) use*http.Request.MultiPartReader
to access the part's directly.– djdDec 5, 2016 at 0:15 -
@djd I succeeded. I turned incoming file data to bytes and passed to bytes.NewReader() which gave me io.Reader. Thank you for your time. I will optimize code with your idea now. Dec 5, 2016 at 10:26
If you have multipart.File, then you can use next solution:
file, ok := multipartFile.(*os.File)
if !ok {
responseError(w, errors.New("invalid file", errors.InvalidArguments))
return
}
*os.File
or really anio.Reader
? The file is only for a file that is physically on disk, but you're also saying that you don't want the file to be on disk?