Questions tagged [large-files]

Large files, whether binary or text, can sometimes be problematic even for an experienced programmer. This tag should be used if issues arise relating to opening and/or writing large files in a text editor, managing resources that run to gigabytes, or strategic decisions for large amounts of data.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

(OleDbCommand) GetSchema() causes large time delays when using a massive Excel file

I have an Excel file that has records in the hundreds of thousands (file size == 336 MB, # of records == 361707) and I want to read the schema of this file into a data table. When I use the ...
oBtooce's user avatar
0 votes
0 answers
17 views

How to stream more than 4GB zip files while generating the zip file in python

I have n number of audio files in s3. If I make a django api request, it should get all the audio files from s3 and generate a zip file and download it as zip file. While generating the zip file, it ...
sheshu N's user avatar
-4 votes
2 answers
85 views

apply sed only to the part of the file after last match in loop - shell / bash [closed]

I have a couple of large files (~1Gb) of such structure: fooA iug9wa fooA lauie fooA nwgoieb fooB wilgb fooB rqgebepu fooB ifbqeiu ... fooN ibfiygb fooN yvsiy fooN aeviu I would like to replace in ...
MartynaM's user avatar
2 votes
1 answer
81 views

create php code for large files openssl encrypt using AES-256-CTR compatible with openssl command line

I am trying to create a PHP method to replicate the OpenSSL command line function so I can encrypt using PHP and then decrypt it using command line . i created PHP method for encrypting files taking ...
Bilal H's user avatar
  • 23
0 votes
0 answers
69 views

replicate openssl encrypt/decrypt between PHP and Command Line for large files?

I am trying to create a PHP method to replicate the OpenSSL command line function so I can encrypt using the command line and then decrypt it using PHP or the opposite by encrypting using PHP and then ...
Bilal H's user avatar
  • 23
1 vote
1 answer
61 views

Using Powershell, how do I filter a JSON to exclude certain key names?

I am trying to reduce the size of a JSON which is 700MB. It's a slightly smaller version of this: https://kaikki.org/dictionary/All%20languages%20combined/by-pos-name/kaikki_dot_org-dictionary-all-by-...
Ne Mo's user avatar
  • 188
0 votes
2 answers
105 views

lxml iterparse eats up memory for an 4GB XML file, even when clear() is used

The purpose of the script is to extract how many articles/books have been published for each year and get this information from element in the xml file dblp-2023-10-01.xml. The file can be found here:...
Elias S's user avatar
0 votes
1 answer
60 views

Large File Upload Results in Corrupted Files

I am currently working on a file upload feature in my application, and I am encountering an issue where the uploaded file appears to be corrupted. The application is designed to handle large file ...
Saqif Haque's user avatar
0 votes
0 answers
55 views

How can I setup Django to serve files on a file server?

I am working on a Django website where files are going to be sold. These files can be large in size, up to 5GB. So, I believe that in addition to a VPS for serving the website, I also need to set up a ...
Shaw's user avatar
  • 63
0 votes
0 answers
42 views

Implementation of an OData v4 Endpoint for Large File Transfer in ASP.NET Core 6 Web API

I am a novice in developing REST APIs in OData and am working on a project that requires the creation of an OData v4 endpoint in ASP.NET Core 6 Web API, using the Microsoft.AspNetCore.OData (v8.2.3) ...
Khan75's user avatar
  • 1
0 votes
0 answers
180 views

How to Upload a Large File to MinIO using Python SDK

I'm curious about how to upload large files in chunks using the MinIO Python SDK in a FastAPI backend. I've implemented chunked uploading using .stream() to the server according to the following post: ...
kitkatdog's user avatar
0 votes
0 answers
73 views

Have any size restriction on download a blob file above 100MB in android webview?

Hello I am sharing my codebase that I used in Webview and JavaScriptInterface. I couldn't download a large file using my codebase. My codebase is working when the file size is around 50MB. If you have ...
Mimo Saha's user avatar
1 vote
1 answer
61 views

Is there any limitation for python pprint printing large data?

Sorry if my English is bad, I speak Korean as mother tongue. What I was trying is to pprint a large list, of which size is like len(repr(lst)) == 64xxx, to a text file. but when I tried something like ...
k9wan's user avatar
  • 57
0 votes
0 answers
14 views

How and where to execute code for "clearing Jupyter Notebooks Output in All Cells"

I have created an infinte loop and now my jupyter notebook is stuck. I have found a solution in this article: https://saturncloud.io/blog/how-to-clear-jupyter-notebooks-output-in-all-cells-from-the-...
Tuan's user avatar
  • 1
0 votes
2 answers
100 views

Jenkins hangs while deploying large files to Artifactory

When our Jenkins project pipelines produce jar files with sizes close to 60 MB or more and try to deploy them to Artifactory, they just hang and never finish. For smaller jar files - below 10 MB - ...
Steffen Lomholt Hansen's user avatar
0 votes
0 answers
12 views

netflix zuul1, a strange phenomenon of uploading large files with multipart file?

My project uses zuul1, has a file upload module, and the client call interface, as others have reported, is http://zuul-proxy/zuul/upload-module/file However, I found that when the file is large, the ...
lightling's user avatar
0 votes
0 answers
29 views

Facing too much Insufficient Memory exception while working with memory mapped-IO files

I am trying to create memory-mapped IO files for having sized ( 1-100 GB ) files. These files will contain random data that will be generated during the process. So, I am trying to generate partitions ...
Usman's user avatar
  • 2,745
1 vote
0 answers
63 views

Why doesn't my program work with large files(650M) correctly? K&R exersice 5-13 tail

Exercise 5-13. Write the program tail, which prints the last n lines of its input. By default, n is set to 10, let us say, but it can be changed by an optional argument so that "tail -n" ...
hansoko's user avatar
  • 321
0 votes
1 answer
56 views

DOMException Error when splitting large file blob into chunks

can someone please help me to debug this function const chunkify = async (file: Blob) => { const totalSize = file.size; const chunkSize = 1024 * 1024 * 100; // 100MB const chunks = [] as ...
Deji's user avatar
  • 92
0 votes
0 answers
132 views

Send large files with Telegram API BOT

Hi everyone on a node project I'm trying to send files to my private channel on Telegram via bot API. However, I get an error when I try to send a video of even 100MB. Reading on the net I noticed ...
mirco crescente's user avatar
0 votes
3 answers
231 views

How to compare (diff) two large CSV files where order does not matter

I'm struggling with comparing (diff'ing) 2 large CSV files. The order of rows doesn't matter I don't need to print the differences or anything, just a True or False. For example: File 1 a,b,c,d e,f,...
luci5r's user avatar
  • 605
0 votes
0 answers
43 views

Problem with UnicodeDecodeError in massive file

I'm trying to read a 7GB csv zipped file using pandas, reading by chunks. The file is UTF-8-BOM, so I'm using that encoding in the read_csv. After 2 hours of reading, it always raises this error: ...
Souni's user avatar
  • 31
0 votes
0 answers
20 views

I have trouble uploading a 2gb folder to github?

So this is my first big project. To keep it short it's a website for a studio that has a lot of images and the file size when uploading to Github is around 2.16gb. I'm uploading it to Github, because ...
adham's user avatar
  • 5
0 votes
1 answer
80 views

Logic Apps parse large json files (up to 300MB)

I have an Azure Logic app which runs daily and downloads up to 300MB of json data (approximately 300k objects). I then need to iterate over the items and transform before I push to an Azure Service ...
Marius Agur's user avatar
0 votes
0 answers
16 views

I have initialized git LFS in my repo and added extensions in the gitattribute for it to track but the size is still not reducing, what do I do?

My bit bucket repository has exceeded the** 4GB limit** so I initialized git LFS in my repo and added **extensions **in the gitattribute for it to track but the size only reduced by a few MB so I ...
Eyram Amusu's user avatar
0 votes
1 answer
24 views

How can I Add column Name in very Large Text file. Data is there in last column but I am missing the column Name

I Have a very large text file that I am unable to open in Notepad or excel. I am missing the column name in the last column. Data is there.
user3860954's user avatar
0 votes
0 answers
23 views

Optimal Approach for Iterating and Processing Large CSV Files with Calculations in Python

Is there an efficient way to iterate through a large CSV file in Python, performing calculations on specific columns and storing the results in a new file? I attempted to process a substantial CSV ...
Afeeqa Hana's user avatar
0 votes
0 answers
113 views

jQuery large file upload very slow, ASP.NET Core

I use jQuery unobtrusive ajax (data-ajax="true") in an ASP.NET Core MVC site to upload a large file (multipart data) to the server. I configured jQuery like this: $.ajax({ ...
mz1378's user avatar
  • 2,079
0 votes
1 answer
44 views

Asp.net core file upload by Unobtrusive Ajax not Working

I have an action method: [RequestSizeLimit(int.MaxValue)] [RequestFormLimits(ValueLengthLimit = int.MaxValue, MultipartBodyLengthLimit = int.MaxValue)] [HttpPost] [MultipartFormData] [...
mz1378's user avatar
  • 2,079
0 votes
0 answers
114 views

Getting an error on NextJS app when trying to push to gitHub. I added a long video to my assets, but then deleted it and I'm still getting the error

This is my error: Compressing objects: 100% (19/19), done. Writing objects: 100% (19/19), 203.40 MiB | 4.76 MiB/s, done. Total 19 (delta 15), reused 0 (delta 0), pack-reused 0 remote: Resolving deltas:...
tmowreen97's user avatar
0 votes
2 answers
85 views

How can I download a large file from SageMaker Studio Lab

I have created a YOLO weight file (300MB) using Sage Maker Studio Lab, and I want to download it to my local PC. I am able to download small files, but the "download" button does not work ...
Wataame's user avatar
1 vote
1 answer
190 views

Springboot large file upload throw "Out Of Memory"

I am using Springboot (2.x) with Jetty Server and trying to upload MULTIPLE large files(~2.2GB) PARALLEL to S3 via direct Streaming, still getting Out Of Memory Issue. So, my question is Is SpringBoot ...
Anshita Singh's user avatar
0 votes
0 answers
246 views

AWSGlue: Decrypt- large files(50GB) using pyhthon-gnupg

Context: We are receiving files which are PGP encrypted , which needs to be decrypted in AWS Glue using pyhon-gnupg module Issue: AWSGlue with PySpark is able to decrypt files upto 2GB files but we ...
Abhishek Jain's user avatar
0 votes
1 answer
29 views

Untar files larger than 2GB (w/o using libarchive if possible)

I'm successfully using https://github.com/libarchive/libarchive/blob/master/contrib/untar.c as dependency-free code to read TAR files, but that code fails with a .tar file that contains a 10GB single ...
ddevienne's user avatar
  • 1,812
0 votes
0 answers
24 views

Jupyter notebook truncates very large FITS file

When I try importing the mastar-goodspec-v3_1_1-v1_7_7.fits file (about 7 GB) on a Jupyter notebook, I get a warning saying that only x GB were imported. Unsurprisingly, when I try to do anything this ...
periwwinkle's user avatar
0 votes
4 answers
286 views

Split large file in R into smaller files with a loop

I have a csv file with 12,626,756 rows that I need to split into smaller files so a colleague can open them in Excel. I want to create a loop that splits the file into files that fit within Excel's ...
DrPaulVella's user avatar
0 votes
1 answer
505 views

How to convert a large csv file (~1TB) into a polars dataframe in a shorter time?

I have a very large CSV file with size 1TB. I want to convert into a polars dataframe, but it takes > ~15mins to generate the dataframe. How can I do this more efficiently? There are high chances ...
megha's user avatar
  • 23
0 votes
0 answers
32 views

Error when taking a screenshot of a very large window

I am using IDLE for Python on Mac (Catalina 10.15.7) and I would like to be able to take a screenshot (Command + Shift + 4 + Space + Enter) of the preview window for function inputs, shown here: This ...
David Chew's user avatar
-2 votes
2 answers
97 views

How to manage large files in Git and GitLab [closed]

I am working on a Django project that has 3 Large files. 1st file: 4.3 GB 2nd file: ~ 600 MB 3rd file: ~ 30 MB The Problem is that Git can't handle large files and crashes. I used Git LFS to track ...
Mahdi Jafari's user avatar
0 votes
1 answer
154 views

Split very large CSV file to Smaller files on the basis of size

I have large 133GB CSV file in my with header in my local system .I want to split this file in 5 parts and each file should contain the header. I have tried doing it through git bash command split -b ...
Radha's user avatar
  • 11
0 votes
0 answers
24 views

Python: how to read + transform 10gb file of concatenated JSONs

I just ran a large batch transform job on AWS Sagemaker and ended up with a 10GB file in the following format: {"vectors": [[1024 items], ..., [1024 items]]}{"vectors": [[1024 ...
sonny's user avatar
  • 303
0 votes
0 answers
21 views

C# XMLReader acts up after attempting to read a too big element

I have a very simple XML, but where one child element is a DNA string. This string can be anything from a few hundred characters long to 3 billion (like in the human genome). Of course reading a ...
Oystein's user avatar
0 votes
2 answers
333 views

c# net7 IFormFile: upload large files

I want to receive files in different sizes. I already get the problem with a 30 MB file. If I use Postman as a client then I get problems with a 7 MB file when I, in intervals of 2 seconds, send the ...
Vesen's user avatar
  • 1
0 votes
1 answer
74 views

Compressing extremely large files for archiving purposes on computing cluster - best practice?

I am in the following situation: Current status: ~40TB of data has to be archived Mostly netCDF4 file format, but folders contain some text files No files larger than ~100MB Plan: Compress and ...
Niklas's user avatar
  • 175
0 votes
1 answer
225 views

Loading a large CSV File into Oracle Table using peoplecode

I am using file layout method and using Peoplesoft suggested code. It works fine with a small csv file but I need to import a file that is 921,000kb. It took a day and is still running. I cannot use ...
SForum's user avatar
  • 3
-1 votes
2 answers
41 views

How to extract text from very large XML files in Python without interrupting tags while parsing incrementally?

EDIT: After I tried the first answer, I still get text broken at interrupting tags (just as before). For reference, here is the code: for event, element in ET.iterparse(path): if element....
user21871257's user avatar
0 votes
0 answers
51 views

How can I efficiently read and upload large files in Amazon S3 using AWS SDKV2 java?

S3 object read function does not read byte according to the buffer declared. I want to read a file from a bucket and upload that file into another bucket. But as file can be huge I want to read and ...
Rajat Dobriyal's user avatar
0 votes
0 answers
33 views

How to upload large scanned files to mongodb - about 1 gigabyte

I am scanning multiple large files for archiving, files size is too large about 500 megabytes to 2 gigabyte. What is the best way to store these files to mongodb database. Should I save files ...
Basem S H Saabneh's user avatar
0 votes
0 answers
84 views

Python moderngl and PIL with big image (14000x9600) only loads red channel

I am trying to load big .png files (~10-16 MB, 14000x9600) with PIL. When I load them into a texture I can only use one color channel (I try 3 or 4 and it tells me it doesn't match in size). The part ...
Senfguerkli's user avatar
0 votes
1 answer
145 views

How to load whole large (100 GB) json to memory with python

I have json file, which size is 100 gb. It scheme looks like: json_f = {"main_index":{"0":3,"1":7},"lemmas":{"0":["test0", "test0"]...
SollPicher's user avatar

1
2 3 4 5
35