Skip to content

feat: Translator for external deps, minor improvement to AssetOut #41

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
May 30, 2025
Merged

feat: Translator for external deps, minor improvement to AssetOut #41

merged 11 commits into from
May 30, 2025

Conversation

lucargir
Copy link
Contributor

@lucargir lucargir commented Apr 22, 2025 •
edited
Loading

  • Use the translator to map external dependencies
  • Update the internal asset keys to have a prefix to make sure all Dagster internal asset keys are unique
  • Remove the translator parameter from methods where it was not used
  • Update the function that gets the internal asset key and the translator to rely on the FQN of the table instead of just the model
  • Minor changes:
    • updated refs to be absolute
    • included asset tags
    • added possibility to group assets through translator

@lucargir lucargir marked this pull request as draft April 22, 2025 11:33
@lucargir lucargir marked this pull request as ready for review April 22, 2025 11:34
@lucargir lucargir mentioned this pull request Apr 22, 2025
Open
@lucargir lucargir force-pushed the main branch 4 times, most recently from dcb5c85 to d07b2b0 Compare April 23, 2025 13:24
@lucargir
feat: move translator to config, add support for kinds label and spec…
f679a3e 
…ify group through translator

The SQLMeshDagsterTranslator has been simplified to return an AssetKey, the name of
the AssetKey or a string equivalent, which deprecates utils.sqlmesh_model_name_to_key
and utils.key_to_sqlmesh_model_name.

The kinds label is now also added, to show technology labels on the UI, only when
Dagster's version allows for it.

Groups can now be specified through the translator, instead of having a fixed method.
@ravenac95
Copy link
Member

@lucargir thanks for the PR!

@lucargir
Copy link
Contributor Author

@ravenac95 the test is failing on the "Table with name test_source does not exist", but I haven't changed anything there. Locally, make test returns no errors, so not sure what it might be 🤷

@ravenac95
Copy link
Member

@lucargir hrm.. Interesting, let me try to look into it tomorrow. I also add some feedback then too!

@ravenac95
Copy link
Member

Can you also rebase to the most recent version? I added an additional test to assist with preventing regressions for our upgrades (and also am upgrading now to sqlmesh v0.178.2)

@ravenac95
Copy link
Member

ravenac95 commented Apr 30, 2025 •
edited
Loading

Oh I have a hunch that I don't have time yet to trace through to validate, but if it works locally I'm not exactly sure what's happening. Potentially I'd do a full deletion of all duckdb state and start over though the tests are intended to do that so that in and of itself is a problem if this is the cause of the fault.

My hunch is that there is something with the translator that has changed to relationship of the external table references to the assets in dagster, because on a fresh run like in the testing it's not initializing the data before running sqlmesh in dagster.

@lucargir
Copy link
Contributor Author

@ravenac95 just rebased, can you re-run the tests? the local tests did pass, so maybe it was this.

@ravenac95
Copy link
Member

@lucargir hrm it seems it's still failing on the same issue. If I have a second tomorrow I'll take a look

ravenac95
ravenac95 reviewed Apr 30, 2025
View reviewed changes
ravenac95
ravenac95 reviewed Apr 30, 2025
View reviewed changes
ravenac95
ravenac95 reviewed Apr 30, 2025
View reviewed changes
ravenac95
ravenac95 reviewed Apr 30, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
ravenac95
ravenac95 reviewed May 1, 2025
View reviewed changes
@lucargir lucargir changed the title feat: Translator as config, minor improvement to AssetOut feat: Translator for external deps, minor improvement to AssetOut May 7, 2025
ravenac95
ravenac95 reviewed May 8, 2025
View reviewed changes
@ravenac95
Copy link
Member

Apologies for the delay, I've been meaning to review this again but we've been a little busy on this end.

@lucargir lucargir requested a review from ravenac95 May 27, 2025 09:54
@ravenac95
Copy link
Member

@lucargir this looks good! Although I don't necessarily like that we moved around the dagster assets. I'll keep it for now :)! Let's get this merged in

@ravenac95 ravenac95 added this pull request to the merge queue May 30, 2025
Merged via the queue into opensource-observer:main with commit 092e46e May 30, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ravenac95 ravenac95 Awaiting requested review from ravenac95

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lucargir @ravenac95