Questions tagged [meltano]
21
questions
1
vote
1
answer
72
views
Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK
Does anyone have experience developing SQL singer-taps using the Meltano SDK?
Problem
My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap.
Context
I am ...
0
votes
0
answers
116
views
Poetry configuration error when adding Meltano tap-csv extractor
I'm attempting to add the tap-csv extractor to my Meltano project in a virtual environment but run into the following error when running meltano add extractor tap-csv:
RuntimeError
The Poetry ...
0
votes
0
answers
149
views
Meltano tap-mysql throwing `No RECORD message received` error
I'm building an ETL pipeline using Meltano that extracts data from a db using tap-mysql extractor and dumps the data into an S3 bucket in AWS.
Here's what I'm doing - testing locally with this meltano....
1
vote
1
answer
138
views
Meltano with MWAA
I have an AWS MWAA environment to run an ETL Pipeline, and I was planning to consider using Meltano in it.
I seem to find no example(s) or documentation that is specific to MWAA. So, I was wondering ...
0
votes
1
answer
158
views
Meltano ETL not deleting rows in destination PostgreSQL database after deleting from source PostgreSQL database
I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (...
0
votes
1
answer
217
views
Where files from tap are kept in Meltano
I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet.
I want to extract data from tap-marketo from data A to date B in the past.
I saw that we can only define ...
0
votes
1
answer
217
views
Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit
I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql".
how could I ...
1
vote
1
answer
442
views
Meltano: tag-postgres to target-postgres: data type such as uuid is converted to varchar
I'm actually working on a Meltano project where I have to extract data from one postgres database and load to the final warehouse (also a postgres database) using "Key-based Incremental ...
1
vote
1
answer
707
views
Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout'
Running Meltano commands raises an error:
__init__() got an unexpected keyword argument 'request_timeout'
This is happening on a simple:
$ meltano state clear
# OR
$ meltano elt
2
votes
1
answer
630
views
How to debug a Singer Tap using VS Code
In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code.
What's the best way to get the VS Code test features working?
5
votes
2
answers
1k
views
How to override a stream's primary key properties or incremental replication key in Meltano?
There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
2
votes
1
answer
295
views
Unable to connect to AWS redshift via Meltano
hope you all are doing great.
Can any one support me with an issue regarding Meltano? I want to perform EL (Extract Load) where I extract data from shopify and load it to AWS Redshift. To extract data ...
1
vote
1
answer
473
views
With Singer taps and targets, why are columns created in the target even if they are deselected?
When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap.
Why does this this happen and how can we resolve ...
4
votes
1
answer
391
views
How to kill a running EL pipeline in Meltano
When running an Extract/Load pipeline with Meltano, what's the best way (or ways) to kill a running job?
Generally these would be run via Airflow but would be nice to have a process that worked also ...
3
votes
1
answer
112
views
How to manually edit STATE tracking in Meltano?
There are several scenarios where we it would be useful to manually edit job state in Meltano. For example, if a table is accidentally deleted in the target SQL warehouse, the user may want to remove ...
1
vote
1
answer
309
views
How to provide a relative start date for Meltano CI/CD pipelines on Singer
When we are running a Meltano build/test cycle (such as in a CI/CD pipeline), we want our Singer pipelines to run as follows:
Don't use pre-captured state bookmarks that might permit a stream to ...
7
votes
1
answer
1k
views
Using dbt and Meltano, how can I prevent multiple dbt job runs from conflicting with each other?
When running dbt jobs in Meltano, dbt run jobs may collide with each other if run out of a triggered context - for instance, when an on-demand job collides with a scheduled job or a CI-based job.
If ...
2
votes
1
answer
2k
views
apache superset query error on datetime with timezone column
I have a table with a column of type TIMESTAMP WITH TIME ZONE and I try to simply SELECT * FROM table, but I get this message:
postgresql error: '>=' not supported between instances of 'datetime....
1
vote
1
answer
156
views
Singer-IO JIRA tap - Not getting records
I am getting only Projects and Versions Stream RECORDS only; Catalog(properties.json) contain 11 streams.
How to get other Streams RECORDS also? I run this command:
tap-jira -c config.json -p ...
0
votes
1
answer
524
views
Cron job with an alias
While I work on setting up Meltano, I'd like to create a simple cron job to schedule my Pipelinewise tap/target on my Ubuntu server. Transferwise references using the following crontab:
*/5 * * * * ...
0
votes
1
answer
201
views
Meltano Kafka Monk.io integration
I'm playing a bit with Monk.io and Kafka - Meltano integration.
So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano.
I'm using https://github.com/lensesio/fast-...