Questions tagged [amazon-data-pipeline]

Simple service to transfer data between Amazon data storage services, kick off Elastic MapReduce jobs, and connect with outside data services.

amazon-data-pipeline
Filter by
Sorted by
Tagged with
1 vote
0 answers
87 views

How to aggregate documents using optional keys in opensearch data prepper pipeline

I’m trying to create an ingestion pipeline using Opensearch Data Prepper to batch upload CDR records with its corresponding QoE data to the opensearch index. After each call ends, the SBC sends three ...
Damo's user avatar
  • 11
1 vote
0 answers
31 views

OpenSearch pipeline to convert string to json

I am trying to accomplish a migration from MySQL to OpenSearch. Because, it is a kind of migration from relational data to NoSQL, I firstly created view to combine data to a json string (DMS is not ...
NecmiK's user avatar
  • 13
0 votes
1 answer
437 views

How can I transfer a database from DynamoDB to Aurora PostgreSQL?

Hello Stack Overflow community, I'm currently working on a project where I need to migrate data from DynamoDB to Aurora PostgreSQL. I've attempted to connect PostgreSQL to Lambda resolver in the past ...
Furkan Yılmaz's user avatar
1 vote
0 answers
37 views

API doesn't support batch/bulk operations

I have a CSV file with 1.5 millions records, I need to call API to get the users email_address, unfortunately the API documents shows it doesn't support batch operation. Currently , for 1.5 millions ...
Olivia Xu's user avatar
0 votes
1 answer
168 views

Mark a AWS data pipeline as succeeded via AWS CLI

I have a lot of pipelines on AWS that use data pipeline. I wont be able to meet the May 12 deadline to move my pipelines. I am looking for alternatives to monitor my pipelines without the console. So ...
CGBoy's user avatar
  • 33
0 votes
1 answer
92 views

Data pipelines om AWS for high mem (glue, MWAA, step funktions)

I am trying to setup a data pipeline in AWS hopefully using serverless and hosted service. However, one of the steps require large amount of ram (120GB) which cannot be broken down into smaller chunks....
Niklas B's user avatar
  • 1,889
10 votes
2 answers
6k views

Is aws datapipeline service being deprecated?

When I navigate to aws datapipeline console it shows this banner, Please note that Data Pipeline service is in maintenance mode and we are not planning to expand the service to new regions. We plan ...
Dinuka Salwathura's user avatar
-1 votes
1 answer
327 views

File conversion in AWS

I am trying to find the most efficient way to process files in AWS. Read a json, xml, csv from S3 bucket Map it to another type of json, xml, csv Save it to S3 bucket Right now we are using Java ...
user3417479's user avatar
  • 1,890
0 votes
2 answers
1k views

Options to export selective data from one dynamodb table to another table in same region

Need to move data from one dynamodb table to another table after doing a transformation What is the best approach to do that Do I need to write a script to read selective data from one table and put ...
Parwinder Kumar's user avatar
0 votes
1 answer
75 views

How would you use AWS Data Pipeline, Elastic MapReduce, and Redshift to perform ETL and data warehousing?

I'm very new to data warehousing and AWS. For school, we have to make a presentation on how data warehousing can be performed using the following three technologies: Redshift AWS Data Pipeline ...
Abi Manoharan's user avatar
2 votes
0 answers
878 views

How to connect AWS RDS for SQL Server to ODBC data sources via Linked Server connections?

Setup Currently we are using SQL Server installed on an EC2 instance as our central data warehouse. We pull in data from a long list of data sources. This is done via SQL Agent Jobs that execute ...
DSC's user avatar
  • 385
0 votes
0 answers
513 views

Redshift to redshift ongoing sync using AWS data pipeline

We would like to explore the possibility of replicating 'some of the tables' our data warehouse on AWS Redshift into another Redshift cluster in an ongoing manner. Don't ask me why as the reasons ...
rk0905's user avatar
  • 5
0 votes
1 answer
305 views

Cannot run AWS Data Pipeline job due to ListObjectsV2 operation: Access Denied

I've written some CDK code to programmatically create a data pipeline that backs up a DynamoDB table into an S3 bucket on a daily basis. But it keeps running into this error: amazonaws.datapipeline....
Fisher Coder's user avatar
  • 3,462
0 votes
1 answer
222 views

AWS CDK DataPipeline how to import an existing data pipeline

I have a datapipeline running already, previously manually created, now I'd like to use CDK code to manage it. How can I do so? (use aws cdk typescript library find/import this datapipeline and manage ...
Fisher Coder's user avatar
  • 3,462
0 votes
1 answer
84 views

AWS Data Pipeline keeps running into FileAlreadyExistsException

I basically followed this tutorial to set up a simple DataPipeline to export my DynamoDB table to S3. But whenever I tried to run it, it keeps throwing Details :Exception in thread "main" ...
Fisher Coder's user avatar
  • 3,462
0 votes
1 answer
29 views

How can I attaching an additional EBS volume to EC2 from Data Pipeline?

I need to attached an additional EBS volume to my ec2 instance from data pipeline. I think currently data pipeline service does not support an option to specify ebs volumes to attach to a ec2 resource....
potato's user avatar
  • 1
0 votes
0 answers
199 views

SQL Server bcp tool on AWS GLUE job

any one has tried sql server bcp utility on aws Glue job (Python shell or Spark type)? bcp tool needs to be installed using Sudo yum commands however those are not supported on Glue. sudo yum install ...
Sam's user avatar
  • 410
5 votes
2 answers
6k views

AWSGlue AccessDeniedException, Status Code 400

I am trying to build a data pipeline for a data engineering project With the help of S3, Glue, Athena, etc., I am stuck when setting up glue crawler for indexing over data. Even I set up the role ...
Zaheer UD Din Baber's user avatar
0 votes
1 answer
199 views

Run Pytest on Glue

How do I invoke pytest on aws glue ? something like below python -m pytest -m reg -ra --myxml=./myout.xml I couldn't find much info on how to pass these markers while invoking pytest. Only option I ...
Sam's user avatar
  • 410
0 votes
0 answers
47 views

Unable to run Elasticmap reduce pyspark job on AWS Datapipeline

Unable to run EMR activity on AWS Data pipeline. Added the job step and bootstrap action step to execute the spark job but it's not able to pick it up. added job step spark-submit, --deploy-mode, ...
ASHISH's user avatar
  • 3
1 vote
0 answers
844 views

Is there a way I can replicate a DynamoDb table but add a sort key to it?

I have some DynamoDb tables populated by data and which don't have a sort key configured. I read DDB doesn't let you add a sort key only at table creation and the only solution is to create a new ...
Go_dot's user avatar
  • 31
0 votes
1 answer
902 views

How to export dynamodb to s3 as a single file?

I have a dynamodb table which will need to be exported to a s3 bucket every 24 hours using data pipeline. This will in turn be used by a sparkjob to query the data. The problem is that whenever I am ...
Arnav Jasrotia's user avatar
1 vote
0 answers
168 views

How to integrate an Ingestion Flow Pipeline with AWS Glue?

I want to create an Entire Data pipeline on AWS. These are the components that I have planned on using: Data Source: This is a remote CSV file (let's say a file in GitHub). Data Lake: This will be an ...
Imperial_J's user avatar
4 votes
2 answers
1k views

AWS Data Pipeline didn't showing EC2 instance role

I am trying to get data from S3 to Dynamodb using AWS Data Pipeline. The issue I am facing is that my "Data Pipeline" wasn't showing EC2 instance role even though I have created one in the ...
Rehan CH's user avatar
  • 111
0 votes
1 answer
189 views

AWS dynamodb export using data pipeline not working for eu-north-1?

I've prepared data pipeline (in eu-west-1 region) based on template that would export dynamodb table to S3. My table is located in eu-north-1 region, but when putting that under the parameter ...
fewfew's user avatar
  • 281
1 vote
0 answers
26 views

Transform rows in CSVs in map reduce way on AWS

Just wonder if there is a map-reduce way on AWS data pipeline or AWS EMR so that I can transform GZIPPED CSV files on S3 into DynamoDB backup format? I think it could be done by an EMRActivity in data ...
user1888955's user avatar
0 votes
0 answers
179 views

Export DynamoDB data to S3 and get the presigned url

While I have alternative to write a lambda for this where I can do, def lambda_handler: file = scan_db_and_push_data_in_csv() s3_client.upload(file) return s3_client_get_presigned_url() But ...
amazing_milkha's user avatar
0 votes
0 answers
13 views

AWS data pipeline get data from hive to s3 [duplicate]

I have created a aws data pipeline to get the data from hive to s3. I have created a emr cluster and set up the hive and using jdbc connection trying to connect hive from the pipeline. Below is the ...
jyo's user avatar
  • 11
0 votes
1 answer
194 views

Batch file processing in AWS using Data Pipeline

I have a requirement of reading a csv batch file that was uploaded to s3 bucket, encrypt data in some columns and persist this data in a Dynamo DB table. While persisting each row in the DynamoDB ...
Shanaka's user avatar
  • 1,700
0 votes
2 answers
274 views

Include header row while extracting the table as csv to s3 in data pipeline AWS service

while extracting the entire table from RDS to s3 in csv format. Header column is not included. what option should we choose to include the Header row in the csv file.
Nivethitha Thiyagarajan's user avatar
1 vote
2 answers
168 views

Estimate duration of DynamoDB data export via Data Pipeline

My DynamoDB table has around 100 million (30GB) items and I provisioned it with 10k RCUs. I'm using a data pipeline job to export the data. The DataPipeline Read Throughput Ratio set to 0.9. How do I ...
Harika K's user avatar
1 vote
1 answer
1k views

ERROR: Unable to validate instance profile 'DataPipelineDefaultResourceRole' because no role exists with that name

I am trying to create the data pipeline in order to take a backup of DynamoDB. I had followed the below documentation for the same. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/...
Dhaval Soni's user avatar
3 votes
1 answer
814 views

How to export DynamoDB table data without the point in time recovery?

I am trying to export data from a DynamoDB table for the last 15 days, but unfortunately, the point in time recovery is not active. So I can't use the new DynamoDB export to S3 feature because it's ...
Reactive_learner's user avatar
1 vote
1 answer
925 views

How to use Web activity in Azure Data Factory to count the total number of pages in source data?

I am doing REST API call to get source data which is JSON and file size is around 1GB. I would like to use pagination concept. I see in this link https://learn.microsoft.com/en-us/answers/questions/...
Dinesh Madhup's user avatar
1 vote
0 answers
126 views

AWS Data Pipeline - SQLActivity into S3 (MySQL DB)

What have I tried As you can see here, this is my configuration on trying to save the output data from query to s3 as CSV. The query was successful (from what I've check on the logs) but it seems ...
Tenten Ponce's user avatar
  • 2,466
1 vote
2 answers
2k views

DynamoDB backup and restore using Data pipelines. How long does it take to backup and recover?

I'm planning to use Data pipelines as a backup and recovery tool for our DynamoDB. We will be using amazon's prebuilt pipelines to backup to s3, and use the prebuilt recovery pipeline to recover to a ...
Vineeth Sai's user avatar
  • 1,517
0 votes
1 answer
169 views

AWS Data Pipeline DynamoDB to S3 503 SlowDown Error

We have a Data pipeline that does a nightly copy of our DynamoDB to S3 buckets so we can run reports on the data with Athena. Occasionally the pipeline will fail with a 503 SlowDown error. The ...
WillK's user avatar
  • 161
2 votes
1 answer
491 views

AWS Data Pipeline Dynamo to Redshift

I have an issue: I need to migrate data from DynamoDB to Redshift. The problem is that I receive such exception: ERROR: Unsupported Data Type: Current Version only supports Strings and Numbers Detail:...
4d61726b's user avatar
  • 457
1 vote
0 answers
94 views

Send data pipeline logs to CloudWatch

I have a cloud formation template for the data pipeline. The only option I see under that is to send logs to the S3 bucket like: - Key: "pipelineLogUri" ...
NoviceMe's user avatar
  • 3,186
1 vote
0 answers
663 views

How to copy selected records from one DynamoDB table to another?

I have two DynamoDB tables, I want to copy just 100 records out of 5000 to another table. Is there any way to do so? I currently have a script which copies the entire contents of the table but I just ...
Nirja Ayare's user avatar
0 votes
2 answers
165 views

Export dynamodb on demand data table by data pipeline

I am trying to export data from dynamoDb to S3 by using data pipeline. My table is on-demand provisioned and contains data of 10gb. How much rcu it would consume? Is there a way I could reduce the ...
Shaggy31's user avatar
0 votes
1 answer
222 views

Activating data pipeline when new files arrived on S3 using SNS

How to activating data pipeline when new files arrived on S3.For EMR scheduling using triggered using SNS when new files arrived on S3.
Salim K A's user avatar
0 votes
2 answers
483 views

AWS data pipeline unable to create through serverless yaml template

I was creating data pipeline for dynamo db export to s3. The template given for serverless yaml is not working on "PAY_PER_REQUEST" billing mode Created one using aws console itr worked fine,...
Asfar Irshad's user avatar
0 votes
2 answers
1k views

AWS Data Pipeline restore to DynamoDB table error with "is in status 'CANCELLED' with reason 'Job terminated' "

I have configured a AWS Data Pipeline to export a DynamoDB table to a S3 bucket (using the template) in a different account. That export is working just fine, but I have some issues when I try to ...
farp332's user avatar
  • 736
0 votes
1 answer
482 views

Datapipeline validation error while adding EMR configuration in datapipeline object list in cloud formation template

If I add a "configuration" object in the datapipeline object list I get the error: Pipeline Definition failed to validate because of following Errors: [{ObjectId = 'SampleEMRCluster', errors ...
AzureMinotaur's user avatar
0 votes
1 answer
257 views

Export DynamoDB to S3 data using AWS Datapipeline

I have a DynamoDB table storing 1Gb of data. RCU and WCU are 1000 each. I set up a Data pipeline to export this 1 GB of data to s3. The entire 1GB of data is exported to s3 in partitions. My question ...
mightyMouse's user avatar
1 vote
1 answer
490 views

AWS Data Pipeline - csv(s3) to RDS Oracle

I want to load my s3 csv file data in RDS Oracle existing table. But in the console, i am not able to find any templates for it. Does it supports oracle also. If yes, How can i achieve this.
Nimmo's user avatar
  • 105
2 votes
1 answer
332 views

AWS data pipeline name tag option for EC2 resource

I'm running a shell activity in EC2 resource sample json for creating EC2 resource. { "id" : "MyEC2Resource", "type" : "Ec2Resource", "actionOnTaskFailure" : "terminate", "actionOnResourceFailure"...
Dev's user avatar
  • 423
0 votes
1 answer
843 views

How to Export million records DynamoDB table as a CSV using Data Pipeline?

I have a million record DynamoDB table. I am using the Data Pipeline to export the DynamoDb table to S3. But the Data Pipeline exports the table as a group of raw json files in DynamoDB JSON format. ...
Shiva Teja's user avatar
2 votes
1 answer
2k views

How to catch Spark error from shell script

I have a pipeline in AWS Data Pipeline that runs a shell script named shell.sh: $ spark-submit transform_json.py Running command on cluster... [54.144.10.162] Running command... [52.206.87.30] ...
gogolaygo's user avatar
  • 309

1
2 3 4 5
10