Questions tagged [meltano]

The tag has no usage guidance, but it has a tag wiki.

Filter by
Sorted by
Tagged with
1 vote
1 answer
72 views

Debugging and troubleshooting singer-tap development for a SQL source using the Meltano SDK

Does anyone have experience developing SQL singer-taps using the Meltano SDK? Problem My biggest problem is troubleshooting the exceptions I am experiencing when testing my tap. Context I am ...
Justin Cohen's user avatar
0 votes
0 answers
116 views

Poetry configuration error when adding Meltano tap-csv extractor

I'm attempting to add the tap-csv extractor to my Meltano project in a virtual environment but run into the following error when running meltano add extractor tap-csv: RuntimeError The Poetry ...
Danial B's user avatar
0 votes
0 answers
149 views

Meltano tap-mysql throwing `No RECORD message received` error

I'm building an ETL pipeline using Meltano that extracts data from a db using tap-mysql extractor and dumps the data into an S3 bucket in AWS. Here's what I'm doing - testing locally with this meltano....
b-onmo's user avatar
  • 13
1 vote
1 answer
138 views

Meltano with MWAA

I have an AWS MWAA environment to run an ETL Pipeline, and I was planning to consider using Meltano in it. I seem to find no example(s) or documentation that is specific to MWAA. So, I was wondering ...
b-onmo's user avatar
  • 13
0 votes
1 answer
158 views

Meltano ETL not deleting rows in destination PostgreSQL database after deleting from source PostgreSQL database

I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (...
vuonglv's user avatar
  • 31
0 votes
1 answer
217 views

Where files from tap are kept in Meltano

I have the following combination of tap/target in Meltano: tap-marketo and target-s3-parquet. I want to extract data from tap-marketo from data A to date B in the past. I saw that we can only define ...
marz's user avatar
  • 93
0 votes
1 answer
217 views

Meltano: (tag and tagret: postgresql) temporary file size exceeds temp_file_limit

I'm getting the issue while running Meltano with tag and target "postgresql" as source database, without changing the temp_file_limit params by the tap "postgresql". how could I ...
Dev024's user avatar
  • 13
1 vote
1 answer
442 views

Meltano: tag-postgres to target-postgres: data type such as uuid is converted to varchar

I'm actually working on a Meltano project where I have to extract data from one postgres database and load to the final warehouse (also a postgres database) using "Key-based Incremental ...
Dev024's user avatar
  • 13
1 vote
1 answer
707 views

Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout'

Running Meltano commands raises an error: __init__() got an unexpected keyword argument 'request_timeout' This is happening on a simple: $ meltano state clear # OR $ meltano elt
ricardossr's user avatar
2 votes
1 answer
630 views

How to debug a Singer Tap using VS Code

In creating new Singer Taps using the Meltano SDK, it's not always clear how to setup testing in VS Code. What's the best way to get the VS Code test features working?
aaronsteers's user avatar
  • 2,337
5 votes
2 answers
1k views

How to override a stream's primary key properties or incremental replication key in Meltano?

There are times when we want to override a stream's key properties (primary key) or it's incremental replication key. What's the best way to do this in Meltano?
aaronsteers's user avatar
  • 2,337
2 votes
1 answer
295 views

Unable to connect to AWS redshift via Meltano

hope you all are doing great. Can any one support me with an issue regarding Meltano? I want to perform EL (Extract Load) where I extract data from shopify and load it to AWS Redshift. To extract data ...
Zaki Khurshid's user avatar
1 vote
1 answer
473 views

With Singer taps and targets, why are columns created in the target even if they are deselected?

When using singer taps and targets, we sometimes see the target creating columns for columns that are explicitly filtered out (deselected) in the tap. Why does this this happen and how can we resolve ...
aaronsteers's user avatar
  • 2,337
4 votes
1 answer
391 views

How to kill a running EL pipeline in Meltano

When running an Extract/Load pipeline with Meltano, what's the best way (or ways) to kill a running job? Generally these would be run via Airflow but would be nice to have a process that worked also ...
aaronsteers's user avatar
  • 2,337
3 votes
1 answer
112 views

How to manually edit STATE tracking in Meltano?

There are several scenarios where we it would be useful to manually edit job state in Meltano. For example, if a table is accidentally deleted in the target SQL warehouse, the user may want to remove ...
aaronsteers's user avatar
  • 2,337
1 vote
1 answer
309 views

How to provide a relative start date for Meltano CI/CD pipelines on Singer

When we are running a Meltano build/test cycle (such as in a CI/CD pipeline), we want our Singer pipelines to run as follows: Don't use pre-captured state bookmarks that might permit a stream to ...
aaronsteers's user avatar
  • 2,337
7 votes
1 answer
1k views

Using dbt and Meltano, how can I prevent multiple dbt job runs from conflicting with each other?

When running dbt jobs in Meltano, dbt run jobs may collide with each other if run out of a triggered context - for instance, when an on-demand job collides with a scheduled job or a CI-based job. If ...
aaronsteers's user avatar
  • 2,337
2 votes
1 answer
2k views

apache superset query error on datetime with timezone column

I have a table with a column of type TIMESTAMP WITH TIME ZONE and I try to simply SELECT * FROM table, but I get this message: postgresql error: '>=' not supported between instances of 'datetime....
Erik's user avatar
  • 2,880
1 vote
1 answer
156 views

Singer-IO JIRA tap - Not getting records

I am getting only Projects and Versions Stream RECORDS only; Catalog(properties.json) contain 11 streams. How to get other Streams RECORDS also? I run this command: tap-jira -c config.json -p ...
Sun's user avatar
  • 3,454
0 votes
1 answer
524 views

Cron job with an alias

While I work on setting up Meltano, I'd like to create a simple cron job to schedule my Pipelinewise tap/target on my Ubuntu server. Transferwise references using the following crontab: */5 * * * * ...
motherbuzzard's user avatar
0 votes
1 answer
201 views

Meltano Kafka Monk.io integration

I'm playing a bit with Monk.io and Kafka - Meltano integration. So, I would like to create a Monk.io Kafka cluster and provision a new connection on Meltano. I'm using https://github.com/lensesio/fast-...
shonster88's user avatar