Auth issue with create-account-groups and sync-workspace-info

[earmada9099]

I am seeing several problems to execute create-account-groups.

Versions:

$ databricks -v
Databricks CLI v0.215.0

$ databricks labs installed
Name  Description                            Version
ucx   Unity Catalog Migration Toolkit (UCX)  v0.21.0

1st.

$ databricks labs ucx create-account-groups
10:49:50  INFO [d.labs.ucx] Account ID: 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd
10:49:50 ERROR [d.l.ucx.create-account-groups] ValueError: invalid literal for int() with base 10: ''

the docu said: "If --workspace-ids flag is not specified, UCX will create account groups for all workspaces configured in the account."

So in this point We have a problem. I solve it, modifing the line 120 in file cli.py

/home/emedina/.databricks/labs/ucx/lib/src/databricks/labs/ucx/cli.py


- if workspace_ids is not None:
+ if workspace_ids is not None and workspace_ids !="":

Now:

$ databricks labs ucx create-account-groups
10:55:29  INFO [d.labs.ucx] Account ID: 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd
10:55:29 ERROR [d.l.ucx.create-account-groups] PermissionDenied: Invalid access token. Config: host=https://adb-8111111111111111.9.azuredatabricks.net, account_id=1aa1111a-bbb1-1111-11c1-d1dd1d1111dd, token=***, auth_type=pat. Env: DATABRICKS_HOST, DATABRICKS_ACCOUNT_ID, DATABRICKS_TOKEN

2do.
I have configure in my proflie (.databrickcfg) with account_id and token. And the docu use account_id when I auth with username and password.
Also, if I use my personal token I get a problem with the API. I need to use account scim token (I think that it is logical because SCIM is used to perform user and group management operation ). There is no reference to this detail in the ucx documentation.

Example with my personal token (I am admin):

$ cat .databrickscfg
[pue2]
host  = https://adb-8111111111111111.9.azuredatabricks.net
token = dapiyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy90616-2 #personal token
#token =  dsapxxxxxxxxxxxxxxxxxxxxxxxxxxxx002d5122    #account scim tokenl
account_id = 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd

$ databricks labs ucx create-account-groups
11:04:10  INFO [d.labs.ucx] Account ID: 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd
11:04:15 ERROR [d.l.ucx.create-account-groups] NotFound: Endpoint not found for /2.0/accounts/1aa1111a-bbb1-1111-11c1-d1dd1d1111dd/scim/v2/Groups

Example with account scim token:

$ cat .databrickscfg
[pue2]
host  = https://adb-8111111111111111.9.azuredatabricks.net
#token = dapiyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy90616-2 #personal token
token =  dsapxxxxxxxxxxxxxxxxxxxxxxxxxxxx002d5122    #account scim tokenl
account_id = 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd

$ databricks labs ucx create-account-groups
11:05:43  INFO [d.labs.ucx] Account ID: 1aa1111a-bbb1-1111-11c1-d1dd1d1111dd
11:05:43 ERROR [d.l.ucx.create-account-groups] PermissionDenied: Invalid access token. Config: host=https://adb-8156323614688069.9.azuredatabricks.net, account_id=1aa1111a-bbb1-1111-11c1-d1dd1d1111dd, token=***, auth_type=pat. Env: DATABRICKS_HOST, DATABRICKS_ACCOUNT_ID, DATABRICKS_TOKEN

and in this point I think that there are bad config on the script because if I use the API to call the endpoint (using the SCIM token): /2.0/accounts/1aa1111a-bbb1-1111-11c1-d1dd1d1111dd/scim/v2/Groups I get a 200
Example:

$ curl -s --request GET -H "Authorization: Bearer dsapxxxxxxxxxxxxxxxxxxxxxxxxxxxx002d5122" "https://accounts.azuredatabricks.net/api/2.0/accounts/1aa1111a-bbb1-1111-11c1-d1dd1d1111dd/scim/v2/Groups?attributes=id&startIndex=1&count=100"  |jq .
{
  "totalResults": 18,
  "startIndex": 1,
  "itemsPerPage": 18,
  "schemas": [
    "urn:ietf:params:scim:api:messages:2.0:ListResponse"
  ],
  "Resources": [
    {
      "id": "137467708333641"
    },
    ....
    }
  ]
}

And the last question.
In $HOME/.databricks there is a file call token-cache.json

How can I update ?
If I edit it, I see that the token is expired.

$ cat token-cache.json
{
  "version": 1,
  "tokens": {
    "https://adb-8111111111111111.9.azuredatabricks.net": {
      "access_token": "eyXXXx....w",
      "token_type": "Bearer",
      "refresh_token": "doauecf110d1b11d111111e11e11b1111e33-2",
      **"expiry": "2024-03-26T13:07:39.060444328+01:00"**
    }
  }
}

[nfx]

It's quite difficult to answer three different issues in one discussion, but let's try. First one is a bug and file an issue for it. Second one: use u2m authentication, so that you get account-level token in a profile:

databricks auth login --account-id ... --host accounts.azuredatabricks.net
(This command "owns" token cache json and it's internal to it)

You can also verify if other account-level profiles work:
databricks auth profiles

[rkkalluri-dbx]

Hi Serge, still getting the below after u2m authentication with databricks auth login --account-id ... --host accounts.azuredatabricks.net. Looks like it cannot find the token in the cache...how do I validate this.

(base) ruser@taH7YWWTVX ~ % databricks labs ucx sync-workspace-info
[UPGRADE ADVISED] Newer ucx version was released 6 hours ago. Please run databricks labs upgrade ucx to upgrade: v0.21.0 -> v0.22.0
16:23:13 INFO [databricks.sdk] Using Databricks CLI authentication
16:23:13 INFO [d.labs.ucx] Account ID: xxx----yyy---zzz
16:23:14 ERROR [d.l.ucx.sync-workspace-info] ValueError: databricks-cli auth: databricks-cli: cannot get access token: Error: token refresh: oauth2: "invalid_request" "Refresh token is invalid"