Skip to content

Commit 01edc29

Browse files
Create Docs- SSO Domain Restriction and Streaming Search (#105)
* create the SSO Domain Restrictions page * create the streaming search doc * Document pause and resume behavior for scheduled pipelines
1 parent 41095cb commit 01edc29

File tree

7 files changed

+92
-2
lines changed

7 files changed

+92
-2
lines changed
Loading
67.6 KB
Loading
264 KB
Loading

docs/user-guide/management/.pages

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
nav:
22
- Management Overview: index.md
3+
- Streaming Search: streaming-search.md
34
- Query Management: query-management.md
45
- Alert Destinations: alert-destinations.md
56
- Templates: templates.md

docs/user-guide/management/index.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,12 @@ The Management section includes tools for maintaining and securing OpenObserve o
22

33
Learn more:
44

5+
- [Streaming Search](../management/streaming-search/)
56
- [Query Management](../management/query-management/)
67
- [Alert Destinations](../management/alert-destinations/)
78
- [Templates](../management/templates/)
89
- [Audit trail](../management/audit-trail/)
910
- [Cipher Keys](../management/cipher-keys/)
1011
- [Nodes in OpenObserve](../management/nodes/)
11-
- [SSO Domain Restrictions](../management/sso-domain-restrictions/)
12+
- [SSO Domain Restrictions](../management/sso-domain-restrictions/)
13+
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
---
2+
title: OpenObserve Streaming Search
3+
description: Learn how OpenObserve's Streaming Search delivers incremental query results using HTTP/2 partitioning for faster log analysis and real-time data processing.
4+
---
5+
This user guide provides details on how to configure, and use OpenObserve's **Streaming Search** feature to improve query performance and responsiveness.
6+
7+
## What is Streaming Search?
8+
9+
Streaming Search is a feature that delivers query results incrementally to the user interface using an **HTTP/2** streaming connection. Instead of waiting for the entire query to complete, OpenObserve breaks large queries into smaller partitions and streams the results as they are processed. This allows matching log entries to appear more quickly, even while the query continues to run in the background.
10+
11+
!!! note "Where to Find"
12+
The **Streaming Search** toggle is located under **Management > General Settings**.
13+
14+
!!! note "Who Can Access"
15+
The `Root` user and any other user with permission to **update** the **Settings** module can modify the **Streaming Search** setting. Access is controlled through role-based access control (RBAC).
16+
![User Access](../../images/streaming-search-access.png)
17+
18+
19+
## Enable or Disable Streaming Search
20+
21+
1. Go to **Management**.
22+
2. Select **General Settings**.
23+
![Enable or Disable Streaming Search](../../images/enable-disable-streaming-search.png)
24+
3. Locate the **Enable Streaming Search** option.
25+
4. Toggle this switch to **On** to enable streaming mode, or **Off** to disable it.
26+
5. Click **Save** to save the changes.
27+
28+
## How does it work?
29+
30+
### 1. Partitioning
31+
32+
OpenObserve splits the total time range into partitions, each covering a segment of time or data volume, called **partitions**. For example, an 8-hour query with 8 GB of data may be split into 8 partitions, each processing 1 GB. Each partition is executed independently and contributes partial results to the overall output.
33+
34+
Streaming begins as soon as the first partition completes, improving responsiveness.
35+
36+
### 2. Mini-Partitioning
37+
38+
Mini-partitioning further breaks the first partition into small slices based strictly on time. Controlled by: `ZO_MINI_SEARCH_PARTITION_DURATION_SECS`=60 (default is 60 seconds)
39+
40+
These short-duration partitions are created only at the beginning of the query time range. Enables the result to appear within milliseconds, even if the full partition takes longer. Once mini-partitions complete, the system switches back to processing full-size partitions.
41+
42+
### 3. Streaming via HTTP/2
43+
44+
Results from each partition are sent incrementally to the browser using a single persistent HTTP/2 connection.
45+
46+
### 4. Real-time delivery
47+
48+
As each partition is processed, its results are immediately streamed to the user interface.
49+
50+
This mechanism avoids delays caused by waiting for the entire result set and is especially useful for long-running or high-volume searches.
51+
52+
53+
## Considerations
54+
55+
- Requires HTTP/2 support in the network stack.
56+
- Fallbacks to standard query mode if disabled.
57+
- Partitioning behavior is automatic. Mini-partitioning improves the time-to-first-result without affecting the accuracy of final results.

docs/user-guide/pipelines/manage-pipelines.md

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,36 @@ View all pipelines in a tabular format, including:
3030

3131
## Actions Tab
3232

33-
- **Pause/Start Pipelines**: Temporarily stop or restart pipelines as needed.
3433
- **Edit Pipeline**: Modify the configuration of an existing pipeline.
3534
- **Delete Pipeline**: Remove a pipeline permanently from your system.
35+
- **Pause/Start Pipelines**: Temporarily stop or restart pipelines as needed.
36+
37+
!!! Info "Pause and Resume a Scheduled Pipeline"
38+
**Pausing a Scheduled Pipeline:**
39+
40+
- When paused, the pipeline stops executing on its scheduled intervals.
41+
- The system preserves the exact timestamp when the pause occurred.
42+
- Pipeline configuration and state are maintained during the pause.
43+
44+
**Unpausing a Scheduled Pipeline:**
45+
46+
When resuming a paused scheduled pipeline, OpenObserve presents a **Resume Pipeline Ingestion** dialog with two options:
47+
<br>
48+
![Resume pipeline](../../images/resume-scheduled-pipeline.png)
49+
50+
- **Continue from where it paused:**
51+
52+
- Processes all data from the pause timestamp to the current time.
53+
- Maintains complete data continuity with no gaps.
54+
- May consume significant system resources for long pause periods.
55+
56+
> **Note:** Use the **Continue from where it paused** option, when data completeness is critical.
57+
58+
- **Start from now:**
59+
60+
- Begins processing from the current timestamp.
61+
- Creates a data gap between the pause and resume timestamps.
62+
- Provides immediate resumption with minimal resource usage.
63+
64+
> **Note:** Use the **Start from now** option, when data gaps are acceptable.
65+

0 commit comments

Comments
 (0)