403 Access Issue while Running queries in Audit log analysis notebook #38
Description
HI Team,
We have configured auditing and we are able to see logs (unity catalog) in s3 folder that we created. but when we run below notebook https://docs.databricks.com/_static/notebooks/aws-audit-logs-etl-unity-catalog.html, when we run below cell we are seeing Amazon S3; Status Code: 403; Error Code: AccessDenied
streamDF = ( spark .readStream .format("cloudFiles") .option("cloudFiles.format", "json") .option("cloudFiles.inferColumnTypes", True) .option("cloudFiles.schemaHints", "workspaceId long") .option("cloudFiles.schemaLocation", f"{checkpoint}/audit_log_schema") .option("cloudFiles.schemaEvolutionMode", "rescue") .load(f"{log_bucket}") .where(col("date")>=to_date(lit(f"{start_date}"), format='yyyy-MM-dd')) )
I have added instance profile related to that bucket for Unity catalog cluster, but still, we are seeing that error and cluster sec mode is in single user. I have also tried to add role related to audit bucket in cross account role data bricks and added policy for audit bucket, still we are seeing same issue. unable to read data into streamDF.
please let me know if anything missing