Skip to content

Pause migration progress workflow schedule #3995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 25, 2025
Merged

Conversation

FastLee
Copy link
Contributor

@FastLee FastLee commented Apr 11, 2025

As the workflow is experimental and fails if no UCX catalog was created by the customer, we are pausing this workflow.
We suggest that the user unpause it after running the create-ucx-catalog command.

Copy link

github-actions bot commented Apr 11, 2025

✅ 9/9 passed, 4 skipped, 18m28s total

Running from acceptance #8574

@FastLee FastLee force-pushed the fix/pause_progress_workflow branch from 3bdd200 to c55595f Compare April 21, 2025 13:53
Copy link
Contributor

@pritishpai pritishpai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor change

@FastLee FastLee force-pushed the fix/pause_progress_workflow branch from c55595f to def1d24 Compare April 25, 2025 13:47
@FastLee FastLee enabled auto-merge April 25, 2025 13:47
@FastLee FastLee requested a review from pritishpai April 25, 2025 13:47
@FastLee FastLee added this pull request to the merge queue Apr 25, 2025
Merged via the queue into main with commit 5a09b25 Apr 25, 2025
8 checks passed
@FastLee FastLee deleted the fix/pause_progress_workflow branch April 25, 2025 14:03
@FastLee FastLee restored the fix/pause_progress_workflow branch April 25, 2025 14:30
gueniai added a commit that referenced this pull request May 6, 2025
* Adds requirement for matching account groups to be created before assessment to the docs ([#4017](#4017)). The account group setup requirements have been clarified to ensure successful assessment and group migration workflows, mandating that account groups matching workspace local groups are created beforehand, which can be achieved manually or programmatically via various methods. The assessment workflow has been enhanced to retrieve workspace assets and securable objects from the Hive metastore for compatibility assessment with UC, storing the results in the inventory database for further analysis. Additionally, the documentation now stresses the necessity of running the `validate-groups-membership` command prior to initiating the group migration workflow, and recommends running the `create-account-groups` command beforehand if the required account groups do not already exist, to guarantee a seamless execution of the assessment and migration processes.
* Fixed Service Principal instructions for installation ([#3967](#3967)). The installation requirements for UCX have been updated to reflect changes in Service Principal support, where it is no longer supported for workspace installations, but may be supported for account-level installations. As a result, account-level identity setup now requires connection via Service Principal with Account Admin and Workspace Admin privileges in all workspaces. All other installation requirements remain unchanged, including the need for a Databricks Premium or Enterprise workspace, network access to the Databricks Workspace and the Internet, a created Unity Catalog Metastore, and a PRO or Serverless SQL Warehouse for rendering reports. Additionally, users with external Hive Metastores, such as AWS Glue, must consult the relevant guide for specific instructions to ensure proper setup.
* Fixed migrate tables when default catalog is set ([#4012](#4012)). The handling of the default catalog in the Hive metastore has been enhanced to ensure correct behavior when the default catalog is set. Specifically, the `DESCRIBE SCHEMA EXTENDED` and `SHOW TBLPROPERTIES` queries have been updated to include the `hive_metastore` prefix when fetching database descriptions and constructing table identifiers, respectively, unless the table is located in a mount point, in which case the `delta` prefix is used. This change addresses a previously reported issue with migrating tables when the default catalog is set, ensuring that table properties are correctly fetched and tables are properly identified. The update has been applied to multiple test cases, including those for skipping tables, upgraded tables, and mapping tables, to guarantee correct execution of queries with the default catalog name, which is essential when the default catalog is set to `hive_metastore`.
* Limit crawl workflows task in assessment to workflows that ran in the last 30 days ([#3963](#3963)). The JobInfo class has been enhanced with a new `last_run` attribute to store the timestamp of the last pipeline execution, allowing for better monitoring and assessment. The `from_job` method has been updated to initialize this attribute consistently. Additionally, the `assess_workflows` method now filters workflows to only include those that have run within the last 30 days, achieved through the introduction of a `last_run_days` parameter in the `refresh_report` method. This parameter enables time-based filtering of job runs, and a new inner function `lint_job_limited` handles the filtering logic. The `lint_job` method has also been updated to accept the `last_run_days` parameter and check if a job has run within the specified time frame. Furthermore, a new test method `test_workflow_linter_refresh_report_time_bound` has been added to verify the correct functioning of the `WorkflowLinter` class when limited to recent workflow runs, ensuring that it produces the expected results and writes to the correct tables.
* Pause migration progress workflow schedule ([#3995](#3995)). The migration progress workflow schedule is now paused by default, with its `pause_status` set to `PAUSED`, to prevent automatic execution and potential failures due to missing prerequisites. This change is driven by the experimental nature of the workflow, which may fail if a UCX catalog has not been created by the customer. To ensure successful execution, users are advised to unpause the workflow after running the `create-ucx-catalog` command, allowing them to control when the workflow runs and verify that necessary prerequisites are in place.
* Skip integration tests with legacy dashboard creation as it is deprecated ([#4009](#4009)). The configuration has been updated with a new section for `mypy_extensions` containing an empty list, in addition to the existing `mypy-extensions` section, introducing redundancy. Furthermore, several integration tests related to legacy dashboard creation have been skipped due to deprecation, as indicated by Databricks no longer supporting this feature. The skipped tests include those for dashboard creation, migration, query linter functionality, and job progress encoder failures, all of which have been marked with a skip reason citing the deprecation of legacy dashboard creation. These changes are temporary and will be revisited in collaboration with the product team to potentially enable these tests only for Labs workspaces, with the ultimate goal of accommodating the deprecation of legacy dashboard creation and ensuring compatibility with future developments.
* Warns instead of an error while finding an acc group in workspace ([#4016](#4016)). The behavior of the account group reflection functionality has been updated to handle duplicate groups more robustly. When encountering a group that already exists in the workspace, the function now logs a warning instead of an error, allowing it to continue executing uninterrupted. This change accommodates the introduction of nested account groups from workspace local groups, which can lead to groups being present in the workspace that are also being migrated. The warning message clearly indicates that the group is being skipped due to its existing presence in the workspace, providing transparency into the reflection process.
@gueniai gueniai mentioned this pull request May 6, 2025
gueniai added a commit that referenced this pull request May 6, 2025
* Adds requirement for matching account groups to be created before
assessment to the docs
([#4017](#4017)). The
account group setup requirements have been clarified to ensure
successful assessment and group migration workflows, mandating that
account groups matching workspace local groups are created beforehand,
which can be achieved manually or programmatically via various methods.
The assessment workflow has been enhanced to retrieve workspace assets
and securable objects from the Hive metastore for compatibility
assessment with UC, storing the results in the inventory database for
further analysis. Additionally, the documentation now stresses the
necessity of running the `validate-groups-membership` command prior to
initiating the group migration workflow, and recommends running the
`create-account-groups` command beforehand if the required account
groups do not already exist, to guarantee a seamless execution of the
assessment and migration processes.
* Fixed Service Principal instructions for installation
([#3967](#3967)). The
installation requirements for UCX have been updated to reflect changes
in Service Principal support, where it is no longer supported for
workspace installations, but may be supported for account-level
installations. As a result, account-level identity setup now requires
connection via Service Principal with Account Admin and Workspace Admin
privileges in all workspaces. All other installation requirements remain
unchanged, including the need for a Databricks Premium or Enterprise
workspace, network access to the Databricks Workspace and the Internet,
a created Unity Catalog Metastore, and a PRO or Serverless SQL Warehouse
for rendering reports. Additionally, users with external Hive
Metastores, such as AWS Glue, must consult the relevant guide for
specific instructions to ensure proper setup.
* Fixed migrate tables when default catalog is set
([#4012](#4012)). The
handling of the default catalog in the Hive metastore has been enhanced
to ensure correct behavior when the default catalog is set.
Specifically, the `DESCRIBE SCHEMA EXTENDED` and `SHOW TBLPROPERTIES`
queries have been updated to include the `hive_metastore` prefix when
fetching database descriptions and constructing table identifiers,
respectively, unless the table is located in a mount point, in which
case the `delta` prefix is used. This change addresses a previously
reported issue with migrating tables when the default catalog is set,
ensuring that table properties are correctly fetched and tables are
properly identified. The update has been applied to multiple test cases,
including those for skipping tables, upgraded tables, and mapping
tables, to guarantee correct execution of queries with the default
catalog name, which is essential when the default catalog is set to
`hive_metastore`.
* Limit crawl workflows task in assessment to workflows that ran in the
last 30 days
([#3963](#3963)). The
JobInfo class has been enhanced with a new `last_run` attribute to store
the timestamp of the last pipeline execution, allowing for better
monitoring and assessment. The `from_job` method has been updated to
initialize this attribute consistently. Additionally, the
`assess_workflows` method now filters workflows to only include those
that have run within the last 30 days, achieved through the introduction
of a `last_run_days` parameter in the `refresh_report` method. This
parameter enables time-based filtering of job runs, and a new inner
function `lint_job_limited` handles the filtering logic. The `lint_job`
method has also been updated to accept the `last_run_days` parameter and
check if a job has run within the specified time frame. Furthermore, a
new test method `test_workflow_linter_refresh_report_time_bound` has
been added to verify the correct functioning of the `WorkflowLinter`
class when limited to recent workflow runs, ensuring that it produces
the expected results and writes to the correct tables.
* Pause migration progress workflow schedule
([#3995](#3995)). The
migration progress workflow schedule is now paused by default, with its
`pause_status` set to `PAUSED`, to prevent automatic execution and
potential failures due to missing prerequisites. This change is driven
by the experimental nature of the workflow, which may fail if a UCX
catalog has not been created by the customer. To ensure successful
execution, users are advised to unpause the workflow after running the
`create-ucx-catalog` command, allowing them to control when the workflow
runs and verify that necessary prerequisites are in place.
* Skip integration tests with legacy dashboard creation as it is
deprecated ([#4009](#4009)).
The configuration has been updated with a new section for
`mypy_extensions` containing an empty list, in addition to the existing
`mypy-extensions` section, introducing redundancy. Furthermore, several
integration tests related to legacy dashboard creation have been skipped
due to deprecation, as indicated by Databricks no longer supporting this
feature. The skipped tests include those for dashboard creation,
migration, query linter functionality, and job progress encoder
failures, all of which have been marked with a skip reason citing the
deprecation of legacy dashboard creation. These changes are temporary
and will be revisited in collaboration with the product team to
potentially enable these tests only for Labs workspaces, with the
ultimate goal of accommodating the deprecation of legacy dashboard
creation and ensuring compatibility with future developments.
* Warns instead of an error while finding an acc group in workspace
([#4016](#4016)). The
behavior of the account group reflection functionality has been updated
to handle duplicate groups more robustly. When encountering a group that
already exists in the workspace, the function now logs a warning instead
of an error, allowing it to continue executing uninterrupted. This
change accommodates the introduction of nested account groups from
workspace local groups, which can lead to groups being present in the
workspace that are also being migrated. The warning message clearly
indicates that the group is being skipped due to its existing presence
in the workspace, providing transparency into the reflection process.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants