Questions tagged [meltano]
16
questions
0
votes
1
answer
50
views
Where files from tap are kept in Meltano
I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet.
I want to extract data from tap-marketo from data A to date B in the past.
I saw that we can only define ...
0
votes
0
answers
122
views
Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit
I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql".
how could I ...
1
vote
1
answer
232
views
Meltano: tag-postgres to target-postgres: data type such as uuid is converted to varchar
I'm actually working on a Meltano project where I have to extract data from one postgres database and load to the final warehouse (also a postgres database) using "Key-based Incremental ...
1
vote
1
answer
363
views
Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout'
Running Meltano commands raises an error:
__init__() got an unexpected keyword argument 'request_timeout'
This is happening on a simple:
$ meltano state clear
# OR
$ meltano elt
2
votes
1
answer
387
views
How to debug a Singer Tap using VS Code
In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code.
What's the best way to get the VS Code test features working?
3
votes
1
answer
613
views
How to override a stream's primary key properties or incremental replication key in Meltano?
There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
2
votes
1
answer
215
views
Unable to connect to AWS redshift via Meltano
hope you all are doing great.
Can any one support me with an issue regarding Meltano? I want to perform EL (Extract Load) where I extract data from shopify and load it to AWS Redshift. To extract data ...
1
vote
1
answer
297
views
With Singer taps and targets, why are columns created in the target even if they are deselected?
When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap.
Why does this this happen and how can we resolve ...
2
votes
1
answer
250
views
How to kill a running EL pipeline in Meltano
When running an Extract/Load pipeline with Meltano, what's the best way (or ways) to kill a running job?
Generally these would be run via Airflow but would be nice to have a process that worked also ...
3
votes
1
answer
59
views
How to manually edit STATE tracking in Meltano?
There are several scenarios where we it would be useful to manually edit job state in Meltano. For example, if a table is accidentally deleted in the target SQL warehouse, the user may want to remove ...
1
vote
1
answer
197
views
How to provide a relative start date for Meltano CI/CD pipelines on Singer
When we are running a Meltano build/test cycle (such as in a CI/CD pipeline), we want our Singer pipelines to run as follows:
Don't use pre-captured state bookmarks that might permit a stream to ...
6
votes
1
answer
973
views
Using dbt and Meltano, how can I prevent multiple dbt job runs from conflicting with each other?
When running dbt jobs in Meltano, dbt run jobs may collide with each other if run out of a triggered context - for instance, when an on-demand job collides with a scheduled job or a CI-based job.
If ...
2
votes
1
answer
2k
views
apache superset query error on datetime with timezone column
I have a table with a column of type TIMESTAMP WITH TIME ZONE and I try to simply SELECT * FROM table, but I get this message:
postgresql error: '>=' not supported between instances of 'datetime....
1
vote
1
answer
124
views
Singer-IO JIRA tap - Not getting records
I am getting only Projects and Versions Stream RECORDS only; Catalog(properties.json) contain 11 streams.
How to get other Streams RECORDS also? I run this command:
tap-jira -c config.json -p ...
0
votes
1
answer
384
views
Cron job with an alias
While I work on setting up Meltano, I'd like to create a simple cron job to schedule my Pipelinewise tap/target on my Ubuntu server. Transferwise references using the following crontab:
*/5 * * * * ...
0
votes
1
answer
173
views
Meltano Kafka Monk.io integration
I'm playing a bit with Monk.io and Kafka - Meltano integration.
So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano.
I'm using https://github.com/lensesio/fast-...