added integration for VPC Flow logs (#98)

simranquirky · web-flow · commit 18cd40ae9668 · 2025-07-17T18:33:16.000+05:30
* added integration for VPC Flow logs

* modifying image dimensions

* made fixes for 404 errors

---------

Co-authored-by: simranquirky &lt;simranquirky&gt;
diff --git a/docs/data-management/storage.md b/docs/data-management/storage.md
@@ -88,7 +88,7 @@ To use GCS for storing stream data, first create the bucket in GCS.
 | ZO_S3_FEATURE_HTTP1_ONLY | true   | Required for compatibility                                                              |
 | ZO_S3_PROVIDER           | s3     | Enables S3-compatible API                                           |
 
-Refer to [GCS AWS migration documentation]((https://cloud.google.com/storage/docs/aws-simple-migration)) for more information.
+Refer to [GCS AWS migration documentation](https://cloud.google.com/storage/docs/aws-simple-migration) for more information.
 
 **Using GCS directly:**
 
diff --git a/docs/ingestion/traces/go.md b/docs/ingestion/traces/go.md
@@ -82,7 +82,7 @@ func InitTracerHTTP() *sdktrace.TracerProvider {
 ```
 ## Setup up credentials 
 
-- You will get `url` and `Authorization` key [here](cloud.openobserve.ai/web/ingestion/custom/traces/).
+- You will get `url` and `Authorization` key [here](https://cloud.openobserve.ai/web/ingestion/custom/traces/).
 - Replace the `url` and `Authorization` key in the `pkg/tel/otel_helper_http.go` file.
 
 ## Setup Service/Application 
diff --git a/docs/integration/aws/.pages b/docs/integration/aws/.pages
@@ -1,4 +1,5 @@
 nav:
 
     - Amazon EC2 : ec2.md
-    - Application Load Balancer(ALB) : alb.md
+    - Application Load Balancer(ALB) : alb.md
+    - Amazon Virtual Private Cloud : vpc-flow.md
diff --git a/docs/integration/aws/vpc-flow.md b/docs/integration/aws/vpc-flow.md
@@ -0,0 +1,66 @@
+---
+title: AWS VPC Flow Logs Integration Guide
+description: Stream AWS VPC Flow Logs to OpenObserve using Kinesis Firehose (no CloudWatch required). Direct setup for basic ingestion.
+---
+
+# Integration with AWS VPC Flow Logs
+This guide explains how to stream VPC Flow Logs directly to OpenObserve using Amazon Kinesis Firehose.
+
+## Overview
+Capture and forward VPC Flow Logs to OpenObserve via Firehose for real-time network visibility.
+
+## Steps to Integrate
+
+??? "Prerequisites"
+    - OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../../quickstart/#self-hosted-installation))
+    - AWS account with access to VPC and Firehose
+    - S3 bucket for failed log backup (recommended)
+
+??? "Step 1: Get OpenObserve Ingestion URL and Access Key"
+
+    1. In OpenObserve: go to **Data Sources → Recommended → AWS**
+    2. Copy the ingestion URL and Access Key
+
+    ![Get OpenObserve Ingestion URL and Access Key](../images/aws-integrations/vpc-flow/fetch-url.png)
+    
+    > Update the URL to have the stream name of your choice:
+        ```
+        https://<your-openobserve-domain>/aws/default/<stream_name>/_kinesis_firehose
+        ```
+
+??? "Step 2: Create Firehose Delivery Stream"
+    
+    1. In AWS Kinesis Firehose, Create delivery stream with Source: `Direct PUT` and Destination: `HTTP Endpoint`.
+    2. Provide OpenObserve's HTTP Endpoint URL and Access Key, and set an S3 backup bucket.
+    3. Give the stream a meaningful name and Create it.
+
+    ![Create Firehose Delivery Stream](../images/aws-integrations/vpc-flow/firehose-stream.png){: style="height:800px"}
+   
+??? "Step 3: Enable VPC Flow Logs"
+
+    1. Go to **VPC → Your VPC → Flow Logs → Create Flow Log**
+    2. Set:
+        - Filter: `All`
+        - Destination: `Kinesis Data Firehose`
+        - Delivery stream: Select the stream you created in step 2
+        - Log format: `All fields`
+    3. Create the flow log
+
+        ![Create Flow Log](../images/aws-integrations/vpc-flow/vpc-flowlog.png){: style="height:800px"}
+
+??? "Step 4: Verify Logs in OpenObserve"
+
+    1. Go to **Logs** → select your log stream → Set time range → Click **Run Query**
+
+        ![Verify Logs in OpenObserve](../images/aws-integrations/vpc-flow/logs-stream.png)
+
+
+??? "Troubleshooting"
+
+    **No logs?**
+
+    - Ensure Firehose is `ACTIVE` and logs are reaching it
+    - Check S3 bucket for failed deliveries
+    - Confirm URL and Access Key are correct
+
+
diff --git a/docs/integration/images/aws-integrations/vpc-flow/create-flow-log.png b/docs/integration/images/aws-integrations/vpc-flow/create-flow-log.png
diff --git a/docs/integration/images/aws-integrations/vpc-flow/fetch-url.png b/docs/integration/images/aws-integrations/vpc-flow/fetch-url.png
diff --git a/docs/integration/images/aws-integrations/vpc-flow/firehose-stream.png b/docs/integration/images/aws-integrations/vpc-flow/firehose-stream.png
diff --git a/docs/integration/images/aws-integrations/vpc-flow/logs-stream.png b/docs/integration/images/aws-integrations/vpc-flow/logs-stream.png
diff --git a/docs/integration/images/aws-integrations/vpc-flow/vpc-flowlog-v2.png b/docs/integration/images/aws-integrations/vpc-flow/vpc-flowlog-v2.png
diff --git a/docs/integration/images/aws-integrations/vpc-flow/vpc-flowlog.png b/docs/integration/images/aws-integrations/vpc-flow/vpc-flowlog.png
diff --git a/docs/sql-functions/aggregate.md b/docs/sql-functions/aggregate.md
@@ -32,7 +32,7 @@ Each row in the result shows:
 - **`key`**: The start time of the 30-second bucket.
 - **`num`**: The count of log records that fall within that time bucket.
 <br>
-![histogram](./images/sql-reference/histogram.png)
+![histogram](../images/sql-reference/histogram.png)
 
 !!! note
     - If you do not specify an interval, the backend automatically determines a suitable value.
diff --git a/docs/sql-functions/full-text-search.md b/docs/sql-functions/full-text-search.md
@@ -53,11 +53,11 @@ This query filters logs from the `default` stream where the `k8s_pod_name` field
 **Syntax**: `match_all('value')` <br>
 **Description**: <br>
 
-- Filters logs by searching for the keyword across all fields that have the Index Type set to Full Text Search in the [stream settings](../user-guide/streams/schema-settings/). 
+- Filters logs by searching for the keyword across all fields that have the Index Type set to Full Text Search in the [stream settings](../user-guide/streams/schema-settings.md). 
 - This function is case-insensitive and returns matches regardless of the keyword's casing.
 
 !!! Note 
-    To enable support for fields indexed using the Inverted Index method, set the environment variable `ZO_ENABLE_INVERTED_INDEX` to true. Once enabled, you can configure the fields to use the Inverted Index by updating the [stream settings](../user-guide/streams/schema-settings/) in the user interface or through the [setting API](../api/stream/setting/).
+    To enable support for fields indexed using the Inverted Index method, set the environment variable `ZO_ENABLE_INVERTED_INDEX` to true. Once enabled, you can configure the fields to use the Inverted Index by updating the [stream settings](../user-guide/streams/schema-settings.md) in the user interface or through the [setting API](../api/stream/setting.md).
 
     The `match_all` function searches through inverted indexed terms, which are internally converted to lowercase. Therefore, keyword searches using `match_all` are always case-insensitive.
 
diff --git a/docs/user-guide/actions/actions-in-openobserve.md b/docs/user-guide/actions/actions-in-openobserve.md
@@ -39,9 +39,9 @@ These are ideal for recurring tasks such as:
 ## Next Steps
 
 - [Create and Use Real-time Actions](../create-and-use-real-time-actions/) 
-- [Create and Use Scheduled Actions](../create-and-use-scheduled-actions.md)   
+- [Create and Use Scheduled Actions](../create-and-use-scheduled-actions/)   
 
 ## Related Links
 
-- [Declare Python Dependencies in Actions](../declare-python-dependencies-in-O2-actions.md)
-- [Environment Variables in Actions](../environment-variables-in-actions.md)
+- [Declare Python Dependencies in Actions](../declare-python-dependencies-in-O2-actions/)
+- [Environment Variables in Actions](../environment-variables-in-actions/)
diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md b/docs/user-guide/dashboards/custom-charts/custom-charts-event-handlers-and-custom-functions.md
@@ -12,7 +12,7 @@ Common event handlers:
 
 Before you begin, note the following: 
 
-- Charts in OpenObserve use an [`option` object](what-are-custom-charts.md/#the-option-object).  
+- Charts in OpenObserve use an [`option` object](what-are-custom-charts.md#the-option-object).  
 - Use the `o2_events` block to specify the event type, such as `click`.  
 - Associate the event with a function that will run when the event occurs.
 
diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md b/docs/user-guide/dashboards/custom-charts/custom-charts-flat-data.md
@@ -1,4 +1,4 @@
-The following step-by-step instructions can help you build a [custom chart that expects flat data](what-are-custom-charts.md/#how-to-check-the-data-structure-a-chart-expects). 
+The following step-by-step instructions can help you build a [custom chart that expects flat data](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). 
 
 ## Use Case
 
@@ -12,7 +12,7 @@ To build a custom chart, you need to bridge two things:
 - **What data you already have**: This is the structure of your ingested data, which is usually flat.
 - **What the chart expects**: Each chart type needs data in a specific format. Some charts expect flat data, while others require nested data.
 
-> **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md/#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md/#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart.
+> **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart.
 
 ## Step 1: Understand the Ingested Dataset
 
@@ -36,19 +36,19 @@ In OpenObserve, the data ingested into a stream is typically in a flat structure
 
 ## Step 2: Identify the Expected Data Structure
 
-Before moving ahead, [identify what structure the chart expects](what-are-custom-charts.md/#how-to-check-the-data-structure-a-chart-expects). The heatmap chart expects flat data.
+Before moving ahead, [identify what structure the chart expects](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). The heatmap chart expects flat data.
 
-In this example, each row in [data[0]](what-are-custom-charts.md/#the-data-object) must contain:
+In this example, each row in [data[0]](what-are-custom-charts.md#the-data-object) must contain:
 
 - `organization_id` (example: "test")  
 - `search_type` (example: "logs")  
 - A numeric value (`total_seconds`) representing total query time.
 
-**Note**: For charts that expect flat data, [reshaping is not needed](what-are-custom-charts.md/#build-the-chart). SQL alone is enough to prepare the data in required format.
+**Note**: For charts that expect flat data, [reshaping is not needed](what-are-custom-charts.md#build-the-chart). SQL alone is enough to prepare the data in required format.
 
 ## Step 3: Prepare the Data (via SQL)
 
-In the [Add Panel](what-are-custom-charts.md/#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. 
+In the [Add Panel](what-are-custom-charts.md#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. 
 ![custom-chart-flat-data-add-panel](../../../images/custom-chart-flat-data-add-panel.png)
 
 Build a SQL query in the **Query Editor** to fetch and prepare the data:
@@ -92,7 +92,7 @@ data=[[
 ]]
 ```
 
-**Note**: OpenObserve stores the result of the query in [the `data` object](what-are-custom-charts.md/#the-data-object) as an **array of an array**.
+**Note**: OpenObserve stores the result of the query in [the `data` object](what-are-custom-charts.md#the-data-object) as an **array of an array**.
 
 ## Step 4: Inspect the Queried Dataset
 
@@ -105,7 +105,7 @@ console.log(data[0]);
 
 ## Step 5: JavaScript Code to Render the Heatmap
 
-In the JavaScript editor, you must construct an [object named `option`](what-are-custom-charts.md/#the-option-object). 
+In the JavaScript editor, you must construct an [object named `option`](what-are-custom-charts.md#the-option-object). 
 This `option` object defines how the chart looks and behaves. To feed data into the chart, use the query result stored in `data[0]` 
 
 The following script:
diff --git a/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md b/docs/user-guide/dashboards/custom-charts/custom-charts-nested-data.md
@@ -1,6 +1,6 @@
-The following step-by-step instructions show how to build a [custom chart that expects nested data](what-are-custom-charts.md/#how-to-check-the-data-structure-a-chart-expects). 
+The following step-by-step instructions show how to build a [custom chart that expects nested data](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). 
 
-This example starts with flat data from the `default` stream, fetched and prepared using SQL, and reshaped and rendered using JavaScript. [Learn more about data preparation and reshaping for custom charts](what-are-custom-charts.md/#build-the-chart). 
+This example starts with flat data from the `default` stream, fetched and prepared using SQL, and reshaped and rendered using JavaScript. [Learn more about data preparation and reshaping for custom charts](what-are-custom-charts.md#build-the-chart). 
 
 ## Use Case
 Build a custom **Sunburst chart** to visualize how **query execution time** is distributed across different **organizations** and **search types**.
@@ -19,7 +19,7 @@ To build a custom chart, you need to bridge two things:
 - **What data you already have**: This is the structure of your ingested data, which is usually flat.
 - **What the chart expects**: Each chart type needs data in a specific format. Some charts expect flat data, while others require nested data.
 
-> **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md/#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md/#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart. 
+> **Note**: Understanding both is important because it helps you write the right SQL query, [prepare](what-are-custom-charts.md#build-the-chart) the data through grouping or aggregation, [reshape](what-are-custom-charts.md#build-the-chart) the results to match the chart’s structure, and map them correctly in the JavaScript code that renders the chart. 
 
 ## Step 1: Understand the Ingested Data
 OpenObserve stores ingested data in a flat structure.  
@@ -43,13 +43,13 @@ OpenObserve stores ingested data in a flat structure.
 
 ## Step 2: Identify the Expected Data Structure
 
-Before starting, [identify what structure the chart expects](what-are-custom-charts.md/#how-to-check-the-data-structure-a-chart-expects). 
+Before starting, [identify what structure the chart expects](what-are-custom-charts.md#how-to-check-the-data-structure-a-chart-expects). 
 
 Sunburst chart expects data to be in nested format or parent-child hierarchy. Each parent (`organization_id`) should contain its children (`search_types`), each with a value.
 
 ## Step 3: Fetch and Prepare the Data
 
-In the [**Add Panel**](what-are-custom-charts.md/#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. 
+In the [**Add Panel**](what-are-custom-charts.md#how-to-access-custom-charts) page, under **Fields**, select the desired stream type and stream name. 
 ![custom-charts-add-panel](../../../images/custom-chart-nested-data-add-panel.png) 
 
 Write a SQL query in the Query Editor to fetch and prepare the data: 
@@ -117,7 +117,7 @@ data = [[
 ]];
 ```
 
-**Note**: OpenObserve stores the result of the query in the [`data` object](what-are-custom-charts.md/#the-data-object) as an array of an array.
+**Note**: OpenObserve stores the result of the query in the [`data` object](what-are-custom-charts.md#the-data-object) as an array of an array.
 
 ## Step 4: Inspect the Queried Data
 
@@ -162,7 +162,7 @@ Note: In the `option` object, use the reshaped `treeData` array as the data fiel
 
 ## Step 6: Render the Chart
 
-Construct the [`option` object](what-are-custom-charts.md/#the-option-object) in the JavaScript code to define the reshaped dataset and configure how the chart should appear. 
+Construct the [`option` object](what-are-custom-charts.md#the-option-object) in the JavaScript code to define the reshaped dataset and configure how the chart should appear. 
 
 ```linenums="17"
 option = {
diff --git a/docs/user-guide/dashboards/index.md b/docs/user-guide/dashboards/index.md
@@ -6,8 +6,8 @@ In OpenObserve, Dashboards are used to visualize and monitor both real-time and
 
 - [Dashboards in OpenObserve](dashboards-in-openobserve.md)
 - [Manage Dashboards](manage-dashboards.md)
-- [Variables](variables.md)
-- [Variable Dependencies in Dashboards](variable-dependencies.md)
+- [Variables](variables/index.md)
+- [Variable Dependencies in Dashboards](variables/variable-dependencies.md)
 - [Comparison Against in Dashboards](comparison-against-in-dashboards.md)
 - [Custom Charts](custom-charts)
 
diff --git a/docs/user-guide/logs/logs.md b/docs/user-guide/logs/logs.md
@@ -1,6 +1,6 @@
 This page explains how to run your first log search in OpenObserve, set a time range, execute a query, apply VRL transformations, adjust display settings, and save or export the results. 
 
-> Before you begin, make sure you have the required access to use the **Logs** page. Learn more about [access control](index.md/#access).
+> Before you begin, make sure you have the required access to use the **Logs** page. Learn more about [access control](index.md#access).
 > Ensure that at least one stream with data is available in your organization. Learn more about [streams](../streams/streams-in-openobserve.md). 
 
 ## Get Started with Logs
diff --git a/docs/user-guide/performance/download-manager.md b/docs/user-guide/performance/download-manager.md
@@ -17,7 +17,7 @@ The Download Manager is a background service that runs on each querier node. Its
 
 === "Ingestion"
 
-    During ingestion, OpenObserve continuously collects and stores incoming data. This data is stored in the form of small Parquet files. Smaller files are periodically merged into larger ones by either the ingester or the compactor. When a larger Parquet file is created, the system uses a [hashing mechanism](../../performance.md/#pre-cache-files-for-faster-query-execution) to select one of the querier nodes to receive and cache the file.
+    During ingestion, OpenObserve continuously collects and stores incoming data. This data is stored in the form of small Parquet files. Smaller files are periodically merged into larger ones by either the ingester or the compactor. When a larger Parquet file is created, the system uses a [hashing mechanism](../../performance.md#pre-cache-files-for-faster-query-execution) to select one of the querier nodes to receive and cache the file.
 
     The selected querier receives a broadcast event notifying it about the new file. It then uses the Download Manager to:
 
diff --git a/docs/user-guide/performance/monitor-download-queue-size-and-disk-cache-metrics.md b/docs/user-guide/performance/monitor-download-queue-size-and-disk-cache-metrics.md
@@ -6,9 +6,9 @@ This guide helps you monitor internal metrics that measure how efficiently the [
 
 OpenObserve emits four key metrics that track how data files are downloaded and cached:
 
-- `zo_file_downloader_normal_queue_size`: Number of files currently waiting in the [**normal** queue](download-manager.md/#dual-queue-system).
+- `zo_file_downloader_normal_queue_size`: Number of files currently waiting in the [**normal** queue](download-manager.md#dual-queue-system).
 
-- `zo_file_downloader_priority_queue_size`: Number of files in the [**priority** queue](download-manager.md/#dual-queue-system). 
+- `zo_file_downloader_priority_queue_size`: Number of files in the [**priority** queue](download-manager.md#dual-queue-system). 
 
 - `zo_query_disk_cache_hit_count`: Number of files that were already on disk when the query was executed. 
 - `zo_query_disk_cache_miss_count`: Number of files that were not on disk and had to be downloaded before the query could run.
@@ -62,7 +62,7 @@ After plotting the chart, you will calculate the cache efficiency to evaluate ov
 2. Go to **Dashboards** from the left navigation menu.
 3. Open an existing dashboard or create a new one.
 4. Click **Add Panel** to start building a chart.
-Refer to [Panels in Dashboards](../dashboards/dashboards-in-openobserve.md/#panels) if needed. 
+Refer to [Panels in Dashboards](../dashboards/dashboards-in-openobserve.md#panels) if needed. 
 
 ### Step 2: Choose a Chart Type
 From the chart type panel, choose a visualization type suitable for time-series data. **Line Chart** is recommended for tracking values over time.
diff --git a/docs/user-guide/streams/extended-retention.md b/docs/user-guide/streams/extended-retention.md
@@ -10,7 +10,7 @@ For example, logs from a known incident that occurred last month, you can config
 ## How to Apply Extended Retention
 
 1. From the **Streams** page, select the **Explore** icon from the **Actions** column. 
-2. Navigate to [**Stream Details**](../../user-guide/streams/stream-details.md/#access-the-stream-details).  
+2. Navigate to [**Stream Details**](../../user-guide/streams/stream-details.md#access-the-stream-details).  
 3. Select the **Extended Retention** tab.  
 4. Use the date selector to specify a time range in UTC. Click **Apply**.  
 5. Click **Update Settings** to apply the retention policy.
diff --git a/docs/user-guide/streams/stream-details.md b/docs/user-guide/streams/stream-details.md
@@ -1,4 +1,4 @@
-After After you complete [data ingestion](streams-in-openobserve.md/#ingest-data-into-stream), use **Stream Details** to view configuration and usage information for a stream and update its settings.
+After After you complete [data ingestion](streams-in-openobserve.md#ingest-data-into-stream), use **Stream Details** to view configuration and usage information for a stream and update its settings.
 
 This guide explains how to open **Stream Details** and the information it displays. 
 
diff --git a/docs/user-guide/streams/streams-in-openobserve.md b/docs/user-guide/streams/streams-in-openobserve.md
diff --git a/mkdocs.yml b/mkdocs.yml
