bug: deprecation of dialect+driver removed the only way to use custom connectivity (e.g. for AWS RDS IAM)

[mateusz]

Target Version

master

Python Version

3.9

PostgreSQL Version

16.8

Operating System

Any

Description

Removal of dialect+driver means I can no longer connect to AWS RDS using AWS IAM authentication by using a customised psycopg driver variant. There is no alternate way to implement this functionality going forward as it now hardcodes the driver.

Is there a possibility to reconsider this deprecation, as it's useful?

Link to Slack/Linen

No response

[edgarrmondragon]

Hi @mateusz, I don't have a hard date where I plan to remove it but I'm curious what's this customised psycopg driver variant do on top of psycopg?

[mateusz]

@edgarrmondragon thanks for your answer - here is how I use it (apologies, this is not a proper module, I just pulled this code out of a project): https://github.com/mateusz/awspostgres/blob/master/awspostgres/dialect.py.

Then I set the dialect+driver config of target-postgres "awspostgres+psycopg2" and it just works! 🤩

As a sidenote, this works also in https://github.com/MeltanoLabs/tap-postgres (although is undocumented)!

[edgarrmondragon]

@edgarrmondragon thanks for your answer - here is how I use it (apologies, this is not a proper module, I just pulled this code out of a project): https://github.com/mateusz/awspostgres/blob/master/awspostgres/dialect.py.

Then I set the dialect+driver config of target-postgres "awspostgres+psycopg2" and it just works! 🤩

As a sidenote, this works also in https://github.com/MeltanoLabs/tap-postgres (although is undocumented)!

Thanks for sharing more details @mateusz! Looking at the code, I think an implementation based on the psycopg dialect could be added, and that way you could leverage some psycopg3-only features of this target (e.g. use_copy):

from sqlalchemy.dialects.postgresql.psycopg import PGDialect_psycopg
from sqlalchemy.dialects.postgresql.psycopg2 import PGDialect_psycopg2


class AWS_PGDialect(PGDialect_psycopg2):
    ...

class AWS_PGDialect_psycopg(PGDialect_psycopg):
    ....

[mateusz]

Even if I implemented psycopg3-compatible dialect, how would I make target-postgres use it after the dialect+driver option becomes deprecated? I don't seem to be able to locate any other relevant options. Apologies if I'm missing something obvious! 🤔

[edgarrmondragon]

Even if I implemented psycopg3-compatible dialect, how would I make target-postgres use it after the dialect+driver option becomes deprecated?

Sorry for the confusion, I didn't mean that it'd be available after the option is removed. Rather that it'd be more compatible with the currently favored dialect+driver combination.

[mateusz]

OK, so I will lose the ability to use RDS IAM with this target, thanks for clarifying.

I guess we can close this ticket, as there will be no resolution possible?

[edgarrmondragon]

OK, so I will lose the ability to use RDS IAM with this target, thanks for clarifying.

Let me reiterate that the setting will not be removed in the foreseeable future, so feel free to continue using it. I might decide to keep the setting and remove the deprecation warning, but document that variants of postgresql+psycopg are preferred.

[mateusz]

Thanks 😊 I will for sure continue to use this until it disappears!