Questions tagged [s3fs]
s3fs is a FUSE filesystem which allows to mount an Amazon S3 bucket as a local filesystem.
203
questions
0
votes
0
answers
22
views
Linking openssl to rc4 error: -d -no-salt command not found
Image to terminal error, -d -no-salt not found
I'm fairly new to RC4 so bear with me. I tried linking the library directly to the .c file but I must have done something wrong. When I echo $...
0
votes
0
answers
22
views
How to use holodeck with S3
I would like to use http://holodeck-b2b.org/ with a self-hosted S3-Storage as filesystem.
I'm thinking of mount options like Goofys, s3fs or rclone.
But also whether there is possibly any fork or ...
0
votes
0
answers
100
views
s3fs mount to shared volume between init container and default container
I want to mount from a s3 bucket to a container and then further access the mounted files inside pod.
I can successfully mount the files to init container, however the mounted files are not accessible ...
0
votes
1
answer
38
views
Directory/Prefix with 40,000 objects locks windows explorer and other applications
Here at work we use s3fs because our product uses Oracle and the legacy code load files uploaded thru web direct from Oracle pl/sql, so we need the Oracle Database to see the same file system the web ...
0
votes
1
answer
133
views
S3Fs directory listings slow - caching somehow possible?
I'm wondering if there is any way to practically speedup directory listings of a s3fs mount? I have a WebDAV server, only for read operations, that basically access my s3fs mount. The problem is that ...
0
votes
1
answer
72
views
Use s3fs only to upload new files, don't care about existing ones already on bucket
I was hoping to use s3fs to upload new files into S3. On the documentation I saw that it doesn't work well when there are multiple clients uploading/syncing to the same bucket.
I really don't care ...
1
vote
2
answers
150
views
How can I play HLS streams from Wasabi in Ant Media Server?
I want to play my HLS streams from Wasabi. I enabled S3 options in Ant Media Server Dashboard. But it seems that Ant Media Server uploads HLS files after the stream ends. How can I play HLS chunks on ...
0
votes
0
answers
51
views
S3FS file transfer fails after 500MB
I am trying to transfer a file of size 900MB into S3 directory which is mounted using S3FS on EC2 instance. I installed S3FS using default config without changing any parameters.
Once file size hits ...
0
votes
1
answer
243
views
AWS: Mounting S3 bucket to EC2 via s3fs (iam role) executes but does not work
I set up a RoboMaker project on AWS. When I am on the virtual desktop of my development environment, I would like to work with the files from my S3 bucket. I figured mounting the S3 bucket via s3fs is ...
0
votes
1
answer
354
views
s3fs use credentials and config within $HOME/.aws as opposed to a /passwd-s3fs file
I'm looking at the readme of s3fs repo.
I wouldlike to mount an S3 dir locally using this tool. The readme says:
s3fs supports the standard AWS credentials file stored in ${HOME}/.aws/credentials. ...
1
vote
1
answer
224
views
Is it possible to install s3fs-fuse inside a aws lambda function?
I want to extract images via ffmpeg to s3 bucket but not sure actually how to do this as s3 is different from local filesystem.
So after searching a bit i found s3fs-fuse but not sure how i can ...
0
votes
1
answer
150
views
FFMPEG trouble with mounted s3fs (fuse) drive "failed to rename file __.stream.tmp to __.stream: Operation not supported"
*** PLease this question is not about my ffmpeg command, but the fact that is stops when the output files goes to a mounted s3fs...
CONTEXT : convert a video when the output file goes to a mounted ...
1
vote
1
answer
170
views
Can s3fs mount different bucket and different account simultaneously?
Can s3fs mount different buckets with different account simultaneously so that I can transfer A account bucket’s folder to B account bucket?
Please told me how to do that.
Or having another way to ...
0
votes
1
answer
525
views
s3fs: How to mount S3 buckets on AWS EC2 instances behind proxy, using IMDS v2
We are having some trouble to mount an AWS S3 bucket (using s3fs v1.90) into an AWS EC2 instance which:
is running Ubuntu 18.04
requires IMDS v2 session tokens
is behind a proxy
The HTTP response ...
1
vote
1
answer
358
views
How to make S3FS use IMDS v2 when mounting s3 buckets from EC2 instance
We are unable to mount a S3 bucket on an AWS EC2 instance that requires IMDS v2 (IMDS v1 works fine, though!). We're using s3fs-fuse v1.90 which is supposed to support IMDS v2 (on Ubuntu 18.04).
This ...
0
votes
1
answer
73
views
Upload large size image files from S3 to cloud application through lambda with nodeJS 14
I'm trying to upload a large image file from S3 to the Cloud application. First I download objects from S3 using aws-sdk for nodejs and then upload them to the cloud application. image is downloaded ...
0
votes
1
answer
290
views
Can I mount a S3 bucket mounted to Linux EC2 to a Windows EC2 with Samba(smb)?
I have a situation where in I have to mount AWS S3 bucket to a Linux EC2 instance(Linux) using FUSE(S3FS).
Once the Bucket it mounted, it need to be mounted in an Windows EC2 instance with Samba(smb).
...
1
vote
0
answers
137
views
s3fs does not store downloaded files in the specified cache directory
I created a s3 bucket and mounted it locally using s3fs with -ouse_cache=/tmp as option. I copied a 50MB zip to the mounted directory and it got uploaded to s3. Then I tried to copy that s3fs file to ...
0
votes
1
answer
714
views
How to Manufacture "Transport endpoint is not connected" Problem in S3FS (for Testing Workarounds)
Do you know of a reliable way to induce a "Transport endpoint is not connected" trouble state in S3FS?
Yes, I know that S3FS is dodgy and that S3 is not meant for mounting as a normal file ...
1
vote
1
answer
194
views
Why s3fs is showing "invalid option -- 'j'" when I have no Js using Docker?
I´m building a docker image using that has a log file that must be mounted as an S3 folder. I´m using s3fs for the trick; however, I cannot run it, I´m constantly getting this bug:
s3fs: invalid ...
0
votes
0
answers
283
views
Pod with S3FS stuck in terminating status
I need to mount an s3 bucket in a kubernetes pod. I am using this guide to help me. It works perfectly, however, the pod is stuck indefinitely in the status of "terminating" when giving the ...
0
votes
1
answer
525
views
s3fs mount non empty folder
I have mounted the s3 bucket to /images on EC2 everything was fine and working.
Now I noticed that /images got unmounted, what I mean is uploaded files are no longer appearing on the s3 bucket, ...
0
votes
1
answer
552
views
chmod/chown is slow on files stored using S3FS on EC2
I have an EC2 server with s3fs mount.
I noticed that it takes more than 40s to 1min if I'm trying to change permission or owner.
$ ls -ltr directory
-rwxrwxrwx 1 apache apache 6444069321 Feb 6 15:54 ...
0
votes
0
answers
454
views
Files from S3 bucket are not getting listed when i mount s3 bucket to Kubernetes Pod using s3fs
I am trying to mount Amazon S3 bucket with the Kubernetes pod using the s3fs. I followed this article
https://icicimov.github.io/blog/virtualization/Kubernetes-shared-storage-with-S3-backend/
for ...
2
votes
1
answer
4k
views
How to access Amazon S3 bucket to the Kubernetes pods using IAM roles instead of Access key & secret keys?
I am trying to mount S3 bucket using s3fs-fuse to the Kubernetes pod. My S3 bucket is protected by IAM roles and i dont have Access Keys and Secret Keys to access S3 bucket. I know how to access a ...
0
votes
2
answers
61
views
Centralized & versioned file based system for a web application deployed in Kubernetes
I am trying to create an Centralized file based repository where I can upload all the configuration files needed for an application to run which is deployed as a pod inside the Kubernetes. Any ...
0
votes
1
answer
774
views
Mount a bucket using S3FS doesn't work as non-root user
I'm trying to mount an Exoscale bucket to an Exoscale VM running Ubuntu 20.04 using s3fs as the ubuntu user created by default by Exoscale. After reading the s3fs README and a few online tutorials ...
0
votes
1
answer
2k
views
fuse: device not found, try 'modprobe fuse' first while mounting s3 bucket on a docker container
I have to mount the s3 bucket over docker container so that we can store its contents in an s3 bucket.
I found https://www.youtube.com/watch?v=FFTxUlW8_QQ&ab_channel=ValaxyTechnologies video which ...
0
votes
0
answers
195
views
Accessing S3 subfolders as non root user using S3FS
I can mount S3 buckets using S3FS by running:
s3fs -o iam_role="myrole" -o use_cache=/tmp/myBucket myBucket /mnt/myBucket -o allow_other
However, when I try to navigate inside this bucket, ...
0
votes
2
answers
357
views
files copying from remote server to s3 does not show status on s3 bucket
I am uploading almost 7 TB of files and folders from my remote server to the s3 bucket but I can not see any files on the s3 bucket. only a few files I can see on s3 that was copied successfully.
I ...
1
vote
2
answers
1k
views
Error - "Transport endpoint is not connected" while using s3fs with kubernetes when pod fails
I have an EKS cluster where having one daemon which mounts a s3 bucket to all pods.
Whenever there is some issue or pod restarts, then the mount volume is not accessible and throws the below error.
...
3
votes
1
answer
2k
views
s3fs -- bucket name contains illegal character
I am trying to mount an S3 bucket onto my ubuntu instance . But I want to only mount a "directory" or key in the s3 bucket.
bucketname : s3://my-bucket/
the directory or part of the s3 ...
0
votes
1
answer
1k
views
s3fs get file throw Input/output error or Couldn't read from remote file "/mnt/s3/" : Failure
I trying to setup sftp with s3fs but facing accessing errors for files.
I create docker image with mounting S3 bucket in entrypoint, I can ssh\sftp to its container and see files and folder from S3 ...
1
vote
1
answer
556
views
Rex-ray AWS S3 external volume docker (docker-compose not working)
I am trying to use postgres and pgadmin with rex-ray external volume on AWS S3.
I did:
Docker plugin install rexray/s3fs:0.11.4 S3FS_ACCESSKEY=XXXXXXXXXXXXX S3FS_SECRETKEY=XXXXXXXXXXXXXXXXX
And then ...
0
votes
0
answers
43
views
Digital Ocean storage thrown Operation not allowed when I try to follow inside dir
I used digital ocean storage in my ubuntu 18.04 by standart library s3fs for mount my storage,
like this
s3fs minimoj-consumer /home/ivan/hosts/docker-symfony/symfony/do1 -o passwd_file=/home/ivan/...
0
votes
1
answer
464
views
Potential Issues making chmod 777 on a directory located in s3 Bucket
Suppose I have an s3 bucket mounted on my ec2 instance. My goal is to keep my static files on s3 and access them from my app which is on the ec2, so I have done everything as it should be but the ...
0
votes
1
answer
482
views
Problem installing S3FS on the RHEL 6 system
We're trying to move off a physical RHEL 6 server, with the help of some S3 storage. I'm following the special RHEL 6 install process call out in the install instructions. The fuse 2.8.4 install ...
1
vote
1
answer
305
views
Unable to mount S3 on EC2 using RIOFS - HTTP error: 400 (Bad Request)
I am trying to mount S3 bucket on AWS EC2 instance using RIOFS. I am unsuccessful.
Here is the logs
12:47:53 [main] (main main.c:753) Using config file: /home/ubuntu/.config/riofs/riofs.conf.xml
12:47:...
0
votes
1
answer
581
views
Regarding mounting S3 bucket in Amazon linux permanently
I have bucket named test bucket and I need to mount this bucket under /var/www/html/upload. the bucket is having 33 GB data. I am using below command to mount
/usr/bin/s3fs -o allow_other test ...
0
votes
1
answer
368
views
After mounting S3 bucket system date of the directory is shown as 1970
I'm mounting S3 bucket using s3fs command, after mount the directory shows the system date as 1970.
My google search could not lead to a fix, looking for a help.
S3 Command:
s3fs rsqatestbucket2 /...
0
votes
1
answer
43
views
Do local files remain accessible offline?
If I mount a local directory and point it to an S3 bucket using s3fs, does it eventually receive the files stored in S3 and do the files remain accessible for reading even if S3 or internet is not ...
0
votes
2
answers
4k
views
How to mount AWS s3 using S3FS to allow full access to any user
I have installed the latest s3fs on Centos 7 in AWS, and can mount my S3 bucket and read from it. My requirement is to allow any user to read, write, create and delete files and folders including ...
0
votes
2
answers
342
views
Mount multiple buckets on Ubuntu server, how to configure multiple access keys?
I have followed WhiteBoard Coder: Install and set up s3fs in Ubuntu 16.04 to mount an s3 bucket to mount an S3 bucket on Ubuntu server.
Now what I want to do is to mount a second bucket on the same ...
0
votes
0
answers
251
views
s3fs mount in Redhat 7.7 with IAM role dont work
I am trying to use s3fs mount following the link as below using IAM role on the EC2 instance spin out of the RHEL 7.7 AMI.
https://medium.com/tensult/aws-how-to-mount-s3-bucket-using-iam-role-on-ec2-...
0
votes
2
answers
747
views
I can't upload files bigger then 20 M to my S3 bucket
I recently created a S3 bucket at Scaleway.
I mount it using s3fs without apparent problem.
I have problems uploading some "mid size" files.
If the size under 20 M it's ok but for with larger files (...
1
vote
2
answers
1k
views
s3fs timeout issue on an AWS Lambda function within a VPN
s3fs seems to fail from time to time when reading from an S3 bucket using an AWS Lambda function within a VPN. I am using s3fs==0.4.0 and pandas==1.0.1.
import s3fs
import pandas as pd
def ...
0
votes
1
answer
605
views
s3fs: Could not load mime types for curl library
I have a strange problem. Last thursday my code has worked, so I'm a little bit confused.
What I want: Mount an S3-Bukket via S3FS to an EC2-Instance.
What thursday worked: Successfully mounted that ...
1
vote
1
answer
955
views
s3fs - Maximum file size limit
I need some explanation regarding s3fs, as after implementation I have some tricky issues.
How can we check the bucket settings after creating a bucket with default settings? What is the command, or ...
0
votes
2
answers
1k
views
Is there any python module through which I can mount a S3 bucket into a filesystem?
s3fs-FUSE is a CLI tool which can do this
But s3fs library does not actually mount it to the filesystem, instead it gives us apis to access the S3 buckets as file system
I'm looking for some python ...
-2
votes
1
answer
194
views
Can we mount s3 bucket which already has some files and directories in it using S3FS?
Am trying to mount S3 bucket with some data to an EC2 instance. Am not getting any error, but df -h is not showing it as mounted.
Can you suggest any other better open source software for the same ...