You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+21Lines changed: 21 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -27,6 +27,7 @@ This tap is tested with all actively supported [Python](https://devguide.python.
27
27
| password | False | None | Password used to authenticate. Note if sqlalchemy_url is set this will be ignored. |
28
28
| database | False | None | Database name. Note if sqlalchemy_url is set this will be ignored. |
29
29
| max_record_count | False | None | Optional. The maximum number of records to return in a single stream. |
30
+
| replication_slot_name | False | tappostgres | Optional. Name of the replication slot created with `pg_create_logical_replication_slot`|
30
31
| sqlalchemy_url | False | None | Example postgresql://[username]:[password]@localhost:5432/[db_name]|
31
32
| filter_schemas | False | None | If an array of schema names is provided, the tap will only process the specified Postgres schemas and ignore others. If left blank, the tap automatically determines ALL available Postgres schemas. |
32
33
| dates_as_string | False | 0 | Defaults to false, if true, date, and timestamp fields will be Strings. If you see ValueError: Year is out of range, try setting this to True. |
@@ -267,44 +268,64 @@ Note also that using log-based replication will cause the replication key for al
267
268
1. Ensure you are using PostgresSQL 9.4 or higher.
268
269
1. Need to access the master postgres instance
269
270
1. Install the wal2json plugin for your database. Example instructions are given below for a Postgres 15.0 database running on Ubuntu 22.04. For more information, or for alternative versions/operating systems, refer to the [wal2json documentation](https://github.com/eulerto/wal2json)
271
+
270
272
- Update and upgrade apt if necessary.
273
+
271
274
```bash
272
275
sudo apt update
273
276
sudo apt upgrade -y
274
277
```
278
+
275
279
- Prepare by making prerequisite installations.
280
+
276
281
```bash
277
282
sudo apt install curl ca-certificates
278
283
sudo install -d /usr/share/postgresql-common/pgdg
279
284
```
285
+
280
286
- Import the repository keys for the Postgres Apt repository
- Use the Postgres Apt repository to install wal2json
299
+
289
300
```bash
290
301
sudo apt update
291
302
sudo apt-get install postgresql-server-dev-15
292
303
export PATH=/usr/lib/postgresql/15/bin:$PATH
293
304
sudo apt-get install postgresql-15-wal2json
294
305
```
306
+
295
307
1. Configure your database with wal2json enabled.
308
+
296
309
- Edit your `postgresql.conf` configuration file so the following parameters are appropriately set.
310
+
297
311
```
298
312
wal_level = logical
299
313
max_replication_slots = 10
300
314
max_wal_senders = 10
301
315
```
316
+
302
317
- Restart PostgresSQL
318
+
303
319
- Create a replication slot for tap-postgres.
320
+
304
321
```sql
305
322
SELECT * FROM pg_create_logical_replication_slot('tappostgres', 'wal2json');
306
323
```
324
+
325
+
You can also create the replication slot with any name you want, and set the `replication_slot_name` setting to that value.
326
+
307
327
1. Ensure your configuration for tap-postgres specifies host, port, user, password, and database manually, without relying on an sqlalchemy url.
328
+
308
329
1. Use the following metadata modification in your `meltano.yml` for the streams you wish to have as log-based. Note that during log-based replication, we do not support any replication key other than `_sdc_lsn`.
0 commit comments