Questions tagged [batch-processing]
Batch-processing refers to programmatic execution with minimum human interaction.
3,912
questions
0
votes
0
answers
5
views
Spark streaming trigger(once=True) is not stopping spark job
I have streaming query with trigger once.
But as per spark documentation once all records are read from event-hub,spark job should stop.
But this is not happening.
dfout=df_events.writeStream.format(&...
1
vote
0
answers
3
views
How to break a nested Python for-loop into many individual jobs and then run them on HTCondor?
I have a nested for loop with this kind of logic:
As, Bs, Cs = [...], [...], [...]
for a in As:
for b in Bs:
for c in Cs:
result = function(a, b, c)
I want to break this into ...
0
votes
0
answers
6
views
linux batchscript for moving logs from one folder to another folder depends upon date
I need to run linux batch script for moving logs from one folder and paste it in another folder. The folder must be created with the date corresponding to logs created date and logs has to be deleted ...
0
votes
1
answer
29
views
How can I extracting the last four pages from 1000+ pdfs in batch?
I have around 1000+ pdfs. I need to extract the last 4 pages of each document and then save each file.
What is the easiest way to do this?
I don't have access to Adobe Pro and the number of documents ...
-3
votes
0
answers
53
views
Why is my .vbs running as intended when I execute in Windows but errors out when I try to schedule it as a scheduled task? [duplicate]
I have a .vbs code set up that I have no issues with when I execute in Windows, however I run into CON issues when I try to run it as a .bat (w and c). I am getting the following error:
Line 81
Char:...
0
votes
0
answers
11
views
digitalocean batch processing
Wondering if digitalocean has a solution/alternative like aws has for batch job processing?
Something that automatically spins up X droplets(instances) does the job and then shuts down?
Trying to ...
0
votes
2
answers
35
views
how to generate batch files in a loop in Python (not a loop in a batch file) with slightly changed parameters per iteration
I am a researcher that needs to run files for a set of years on a SLURM system (high performance computing center). The available nodes for long compute times have a long queue. I have 42 years to run,...
0
votes
0
answers
17
views
How to process batch files with Python so that the file names are recorded into each row of excel one by one?
This Python code is used to batch-read multiple txt files in a folder and organize their contents into an excel file. The code works by looping through all the files in a folder, reading the content ...
-1
votes
1
answer
38
views
Calculate least squares in batches
I want to calculate a Gaussian process. For this I need to calculate the following equation:
f = X @ Z^-1 @ y
To handle the inverse I use c = Z^-1 @ y which is the solution of the problem Z @ c = y. ...
0
votes
1
answer
28
views
Spring batch vs Kafka Streams [closed]
I have to implement a solution that consists of processing a large amount of data by applying business requirement rules. The input and the output will be a file.
I haven't been using Kafka before, I ...
0
votes
0
answers
33
views
Easytrive Plus and DB2 - Read a file as input to matching against DB2 table
I am running a batch job that reads in a file in with a list of templates. I am trying to find an example since the one I was provided will not work. I have tried to find any example of reading in a ...
0
votes
0
answers
19
views
How can I download multiple email streams using Microsoft Graph SDK's batched requests?
I'm trying to download multiple emails using a single Microsoft Graph SDK call.
This is the code I'm using:
foreach (var emailId in emailIds)
{
var request = graphServiceClient.Users[...
-1
votes
0
answers
28
views
More than %9 variables on a .BAT [duplicate]
I need to send 11 variables to a .BAT
Powershell.exe -executionpolicy remotesigned -File C:\CrLocal\LegajosTemp\Shell.ps1 %1 %2 %3 %4 %5 %6 %7 %8 %9 %10% %11%
But in the %10 and %11 take %1
how can i ...
0
votes
1
answer
17
views
I am getting ORA-01008: not all variables bound : Error when I am trying to read a oracle DB via SQLPagingQueryProviderFactoryBean
Below is the snippet of the code of Spring boot JDBC item reader which call paging Query query provider.
final SqlPagingQueryProviderFactoryBean testVar = new SqlPagingQueryProviderFactoryBean(); ...
0
votes
0
answers
62
views
Wildfly 26.1.3 Domain Batch JBeret WFLYCTL0030: No resource definition is registered for address
I am trying to run JBeret Batch on a Wildfly Domain Cluster locally, but I keep getting the error
"WFLYCTL0030: No resource definition is registered for address [ (\"deployment\" => \...
0
votes
1
answer
53
views
Windows Batch For-Loop 'input line too long'
My FOR-loops recently created some annoying error messages on both my Windows 10 and 11 computers. I could sware they worked before.
The following three command lines work fine and create the expected ...
-2
votes
1
answer
36
views
How to perform batch operations in YII Framework?
Now I'm working with a big project written on PHP + YII framework.
Now there is a task where it is required to import data from a big json-file into the database.
I'm afraid that this process would ...
-1
votes
2
answers
42
views
How to parse command outptut and store in a file
I am executing command and parsing the command based on space, Formatting it and storing in a file.
This is how I have written the batch script
for /f "tokens=2,4,5" %G IN ('command') DO ...
0
votes
0
answers
46
views
How to get the serial port number in Makefile?
Sometimes, you need to know the serial port number when developing MCUs. For example, I plug the USB cable of the board into the USB port of my computer, then one serial port number appeared in the ...
1
vote
1
answer
458
views
How to do batch img2img job of many img/prompt pairs -- automatic1111 stable diffusion
there
With automatic1111 stable diffuison, I need to re-draw 100 images. Thanks to clip-interrogator, I've generated prompt text for each one of them. Next, I should to run img2img. However, I noticed ...
0
votes
1
answer
40
views
Batch / offline methods of converting RESP files to Redis RDB file
Assuming we want to automate the process of creating RDB files (and don't want to use Redis server for this purpose) what options are available?
The current process involves importing (with redis-cli)...
6
votes
5
answers
218
views
Can we reliably keep HTTP/S connection open for a long time?
My team maintains an application (written on Java) which processes long running batch jobs. These jobs needs to be run on a defined sequence. Hence, the application starts a socket server on a pre-...
-1
votes
1
answer
22
views
Running multi-node parallel job in AWS Batch using R
I am trying to build a multi-node parallel job in AWS Batch running an R script. My R script runs independently multiple statistical models for multiple users. Hence, I want to split and distribute ...
0
votes
1
answer
38
views
some questions about purpose of both ItemStreamReader and ItemStreamWriter in Spring Batch
I'm studying about Spring Batch.
I'm using ItemReader and ItemWriter in my Spring Batch Project.
However, my project's the biggest problem is that all data reading logic is in constructor without ...
-2
votes
1
answer
26
views
I am trying to create a .bat script to edit a bunch of xml files. I am encounterring a lot of errors
What I want is:
if XML File contains CASES.PRIORITY and/or CASES.SUBJECT remove both of them and if file contains Category__c and Sub_Category__c then end. Else if they don't exist, add them below ...
-1
votes
0
answers
22
views
How do some program scan all of etherscan tokens old and new, and provide real time notifications when certain conditions are met?
I recently was interested in creating a telegram bot using python to track all ethereum tokens on eth scan (there are about 100 million tokens, and there is about 100+ tokens launched everyday).
I see ...
0
votes
0
answers
35
views
read method of ItemReader & stepExecutionListner's custom implementation is not called
read method of ItemReader & stepExecutionListner's custom implementation is not called. The basic code structure is as below -
public class myCustomReader implements ItemReader<List<String&...
0
votes
0
answers
28
views
Identify Input Record/Row/Line No. causing the Exception in an Spring Batch process
I have been surfing several websites since few days to find a generic solution to this problem : How to identify which input record is causing the Batch process to fail in exception cases?
I have a ...
1
vote
0
answers
11
views
At a radio ip address (interface login) , I want the browser to auto fill the user and password and login, repeat until login is sucessful
I have hundreds of radio equipment and cameras to log into to change settings , firmware , etc.
I repetitively retype the same 6 or 8 different usernames and passwords until I find the correct ones ...
0
votes
0
answers
19
views
ImageJ Macro script to batch process with color thresholding fails to save output tif files
I've written a script in ImageJ Macro in order to batch process a large number of images with color thresholding and particle analysis, modified from this tutorial. Most of the time the thresholding ...
0
votes
0
answers
32
views
AWS Batch from on prem
I have an on premises schedule that's too dense and there are a lot of errors. I'd like to explore options for migrating some of the workload to the cloud. Can I connect to on-premises resources ...
0
votes
0
answers
49
views
For Loop /D Not Working for Network Attached Drives
TLDR: When I do the following For Loop in the CMD line, I get different results depending on what disk is Current.
for /d %i in (*) do @echo %i
It works if I am on any directory in the C: drive.
...
0
votes
0
answers
30
views
What's the point of Keras its Unstateful LSTM model?
Keras distinguishes "Stateful" and "Unstateful" LSTM's, by default having them configured as "Unstateful". This is something that confused me a lot as LSTM's by ...
0
votes
0
answers
65
views
Laravel batches for execution
I am fairly new with Laravel and I am planning to use massively the queues and batches. Today, while I was working on massing data import for performance testing, I found an issue with what we build ...
0
votes
0
answers
73
views
FFmpeg Batch extracting frames from multiple videos on Windows
I want to extract frames/images per video that will import from a directory and then save those frames into a new folder and the folder name as import video filename.
I tried the below code on Linux ...
1
vote
1
answer
66
views
Optimize batch transform inference on sagemaker
With current batch transform inference I see a lot of bottlenecks,
Each input file can only have close to 1000 records
Currently it is processing 2000/min records on 1 instance of ml.g4dn.12xlarge
...
0
votes
0
answers
24
views
Spring-Batch ItermReader using REST Endpoint as source
Could not find a source as REST endpoint using which ItemReader in spring-batch can can invoke it thousand times and read json.
I tried going through https://docs.spring.io/spring-batch/docs/current/...
1
vote
0
answers
36
views
Batch call to PlaylistItems: insert YouTube API v3 with Google API Python client library produces intermittent HttpError 500
I'm using the Google API Python client library to run a batch of PlaylistItems: insert calls per the batch method described in the docs. When it's all put together it looks like this:
youtube = ...
1
vote
2
answers
54
views
How to export multiple queries from DataGrip
I have a file that contains multiple select queries.
I run these on a weekly basis to generate internal company reports
For example (these are not the actual queries):
-- Select number of users
SELECT ...
0
votes
0
answers
20
views
Pipelined vs blocking data exchange in a Flink job
I've been reading about pipelined region scheduling in Flink and am a bit confused about what they mean. My understanding of it is that a Streaming job is always pipelined whereas a Batch job can ...
0
votes
1
answer
56
views
Vertex AI Airflow Operators don't render XCom pulls (specifically CreateBatchPredictionJobOperator)
I am trying to run a batch predict job task using the Vertex AI Airflow Operator CreateBatchPredictionJobOperator. This requires pulling a model id from XCom which was pushed by a previous custom ...
0
votes
0
answers
96
views
Error Running Python Script from Batch File from the remote machine FileNotFoundError: [Errno 2] No such file or directory: '[]'
I have a python script in a server. I am able to execute this from the local server using a batch script. But when I try to execute this same batch file from remote server I get this error.
"can'...
0
votes
0
answers
17
views
one query I need to use multiple transaction blocks inside a single jdbc batch, how to do that any example?
Because I have insert statement which should be happen only if update statement executed fine, otherwise proceed to other set of insert and update query inside batch and do the same
0
votes
0
answers
16
views
JPA saveAll is not inserting in two tables in batch and very slow [duplicate]
I am using spring JPA/hibernate to insert data into postgres database.
Actually I need to insert records into two tables
I have below Entity class
@Entity
@Table(name="TECHNOLOGY")
@...
1
vote
1
answer
87
views
Can we get data processed in Spring Batch after batch job is completed?
I am using spring batch for reading data from db and process the same and do spome process in writer.
if batch size is less than the records read by reader then spring batch runs in multiple batches.I ...
0
votes
0
answers
26
views
Image_dataset_from_directory with portrait and landscape images
I use tf.keras.utils.image_dataset_from_directory to create an Import Lane to my NN.
Importing the image using the right split and lable works without any issues as long as I only have either portrait-...
0
votes
0
answers
34
views
How can I convert a list of float values to a string of hex?
To start off, I'm not a programmer or script writer. I'm modding a 3d model file of a game and it involves using a hex editor to input values into the float32 box in a hex editor like HxD.
So i'm ...
1
vote
1
answer
84
views
woocoomerce bulk product variant update with batch processing
I want to make update multiple product variants with batch processing
with the woocommerce standard functions. I learned that there is a REST API endpoint https://woocommerce.github.io/woocommerce-...
0
votes
0
answers
27
views
How do I configure a Spring Batch job that executes/ is triggered when a flag is set or value is available from the Database
Possible Solution: Have two Spring jobs. One job is a scheduled one that runs every 15 mins and reads the value from the DB table. The second job is triggered "on demand" that is when the ...
0
votes
0
answers
18
views
RabbitMq batch consuming messages
I am sending messages to rabbitmq queue using mqtt protocol. I want to consume the messages using AMQP protocol. Instead of consuming the messages one by one. I want to consume the messages in batches ...