Skip to content

Commit 41095cb

Browse files
simranquirkysimranquirky
authored
Cognito - Integration Page (#100)
* added integration for VPC Flow logs * modifying image dimensions * made fixes for 404 errors * added integration page for AWS Cognito * adding integration page for AWS Eventbridge * added Integration page for AWS Cloudwatch logs --------- Co-authored-by: simranquirky <simranquirky>
1 parent dd5b7eb commit 41095cb

File tree

10 files changed

+295
-1
lines changed

10 files changed

+295
-1
lines changed

‎docs/integration/aws/.pages

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,8 @@ nav:
22

33
- Amazon EC2 : ec2.md
44
- Application Load Balancer(ALB) : alb.md
5-
- Amazon Virtual Private Cloud : vpc-flow.md
5+
- Amazon Virtual Private Cloud : vpc-flow.md
6+
- Amazon Cognito : cognito.md
7+
- AWS Cloudwatch logs: cloudwatch-logs.md
8+
- Amazon EventBridge : eventbridge.md
9+

‎docs/integration/aws/cloudwatch-logs.md

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
---
2+
title: AWS CloudWatch Logs Integration Guide
3+
description: Stream CloudWatch Logs to OpenObserve using Kinesis Firehose and HTTP delivery.
4+
---
5+
6+
# Integration with AWS CloudWatch Logs
7+
8+
This guide explains how to *stream logs from AWS CloudWatch* to OpenObserve using Kinesis Firehose with an HTTP endpoint.
9+
10+
## Overview
11+
12+
Forward application or infrastructure logs from CloudWatch Logs to OpenObserve in near real-time.
13+
The integration uses **CloudWatch Logs subscriptions** to stream log data into **Kinesis Firehose**, which then delivers it to **OpenObserve over HTTP**.
14+
15+
This approach enables centralized log analysis, alerting, and visualization in OpenObserve without storing logs in S3 or managing Lambda transformations.
16+
17+
## Steps to Integrate
18+
19+
??? "Prerequisites"
20+
- OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../../quickstart/#self-hosted-installation))
21+
- AWS account with:
22+
- CloudWatch Log Group(s)
23+
- IAM permissions to create Firehose streams and set up log subscriptions
24+
25+
??? "Step 1: Get OpenObserve Ingestion URL and Access Key"
26+
27+
1. In OpenObserve: go to **Data Sources → Recommended → AWS**
28+
2. Copy the ingestion HTTP URL and Access Key
29+
30+
![Fetch OpenObserve Ingestion URL](../images/aws-integrations/vpc-flow/fetch-url.png)
31+
32+
> Example format:
33+
> ```
34+
> https://<your-openobserve-domain>/aws/default/cloudwatch-logs/_kinesis_firehose
35+
> ```
36+
37+
38+
??? "Step 2: Create a Kinesis Firehose Delivery Stream"
39+
40+
1. In AWS Kinesis Firehose, Create delivery stream.
41+
2. Set Source: `Direct PUT` and Destination: `HTTP Endpoint`.
42+
3. Provide OpenObserve's HTTP Endpoint URL and Access Key, and set an S3 backup bucket.
43+
4. Give the stream a meaningful name and Create it.
44+
45+
![Kinesis Firehose Delivery Stream](../images/aws-integrations/cloudwatch-logs/firehose-stream.png)
46+
47+
??? "Step 3: Attach CloudWatch Logs to Firehose"
48+
49+
1. Go to **CloudWatch → Log Groups**
50+
2. Select the log group you want to stream
51+
3. Click **Actions → Create subscription filter**
52+
4. Choose:
53+
- **Destination**: `Kinesis Firehose`
54+
- **Delivery stream**: the stream created in Step 2
55+
5. Set a filter pattern (e.g., `""` to send all logs)
56+
6. Click **Start streaming**
57+
58+
![Attach CloudWatch Logs to Firehose](../images/aws-integrations/cloudwatch-logs/subscription-filter.png)
59+
60+
??? "Step 4: Verify Logs in OpenObserve"
61+
62+
1. Go to **Logs** → select your log stream → Set time range → Click **Run Query**
63+
64+
![Verify Logs in OpenObserve](../images/aws-integrations/cloudwatch-logs/cloudwatch-logs.png)
65+
66+
??? "Troubleshooting"
67+
68+
**Not seeing logs in OpenObserve?**
69+
70+
Check the following:
71+
72+
- **CloudWatch Logs**
73+
- Ensure log events are being generated and the correct log group is selected.
74+
- Verify that the subscription filter is active.
75+
- **Firehose**
76+
- Confirm the delivery stream is active and not reporting errors.
77+
- Review the Monitoring tab for delivery status.
78+
- Check the backup S3 bucket (if configured) for failed records.
79+
- **OpenObserve**
80+
- Validate the HTTP ingestion URL and access key.
81+
- In the Logs view, select the correct stream and expand the time range.
82+
83+
**Start Streaming Fails after creating Subscription Filter**
84+
85+
Check the IAM role attached to the filter has full acess to Kinesis Firehose.

‎docs/integration/aws/cognito.md

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
3+
title: AWS Cognito Logs Integration Guide
4+
description: Stream AWS Cognito authentication events to OpenObserve using CloudTrail, EventBridge, and Kinesis Firehose.
5+
6+
---
7+
8+
# Integration with AWS Cognito
9+
10+
This guide explains how to *stream* AWS Cognito authentication events to OpenObserve using Amazon CloudTrail, EventBridge, and Kinesis Firehose.
11+
12+
## Overview
13+
14+
Monitor login attempts, user activity, and authentication behavior from Amazon Cognito by streaming events into OpenObserve for analysis. This integration uses CloudTrail to capture Cognito API calls, EventBridge to route events, and Firehose to deliver them to OpenObserve.
15+
16+
## Steps to Integrate
17+
18+
??? "Prerequisites"
19+
- OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../../quickstart/#self-hosted-installation))
20+
- AWS account with:
21+
- Cognito User Pool
22+
- Permissions to create EventBridge rules and Firehose streams
23+
24+
??? "Step 1: Enable CloudTrail Management Events"
25+
26+
1. Go to **AWS CloudTrail → Trails**
27+
2. Ensure you have a trail that logs **management events**
28+
3. If not:
29+
- Click **Create trail**
30+
- Name it (e.g., `CognitoTrail`)
31+
- Enable **Management events** (Read/Write or Write-only)
32+
- Choose a destination S3 bucket (new or existing)
33+
- Complete trail creation
34+
35+
??? "Step 2: Get OpenObserve Ingestion URL and Access Key"
36+
37+
1. In OpenObserve: go to **Data Sources → Recommended → AWS**
38+
2. Copy the ingestion URL and Access Key
39+
40+
![Get OpenObserve Ingestion URL and Access Key](../images/aws-integrations/vpc-flow/fetch-url.png)
41+
42+
> Update the URL to have the stream name of your choice:
43+
> ```
44+
> https://<your-openobserve-domain>/aws/default/<stream_name>/_kinesis_firehose
45+
> ```
46+
47+
48+
??? "Step 3: Create Firehose Delivery Stream"
49+
50+
1. In AWS Kinesis Firehose, Create delivery stream with Source: `Direct PUT` and Destination: `HTTP Endpoint`.
51+
2. Provide OpenObserve's HTTP Endpoint URL and Access Key, and set an S3 backup bucket.
52+
3. Give the stream a meaningful name and Create it.
53+
54+
??? "Step 4: Create EventBridge Rule to Capture Cognito Events"
55+
56+
1. Go to **EventBridge > Rules** and click **Create rule**
57+
2. Name the rule (e.g., `CognitoToOpenObserve`)
58+
3. Set event pattern:
59+
```json
60+
{
61+
"source": ["aws.cognito-idp"]
62+
}
63+
```
64+
4. Set the target to **Kinesis Firehose** and choose the delivery stream
65+
5. Click **Create rule**
66+
67+
![Create EventBridge Rule to Capture Cognito Events](../images/aws-integrations/cognito/event-rule.png)
68+
69+
70+
??? "Step 5: Verify Logs in OpenObserve"
71+
72+
1. Go to **Logs** → select your log stream → Set time range → Click **Run Query**
73+
74+
![Verify Cognito Logs](../images/aws-integrations/cognito/log-stream.png)
75+
76+
??? "Troubleshooting"
77+
78+
79+
**No logs appearing?**
80+
81+
- Ensure your CloudTrail is capturing **management events**
82+
- Confirm your EventBridge rule has **matched events** recently
83+
- Check if Firehose is delivering logs (see monitoring tab in Firehose)
84+
- Verify that the OpenObserve URL and Access Key are valid
85+
- Look for failed logs in your backup S3 bucket
86+

‎docs/integration/aws/eventbridge.md

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
---
2+
title: AWS EventBridge Logs Integration Guide
3+
description: Stream AWS API activity logs to OpenObserve using CloudTrail, EventBridge, and Kinesis Firehose.
4+
---
5+
6+
# Integration with AWS EventBridge
7+
8+
This guide explains how to *stream AWS management activity events* to OpenObserve using AWS CloudTrail, Amazon EventBridge, and Kinesis Firehose.
9+
10+
## Overview
11+
12+
Monitor and analyze AWS API activity (e.g., EC2 starts, IAM changes, S3 access) by routing events from **CloudTrail → EventBridge → Firehose → OpenObserve**.
13+
14+
- **CloudTrail** captures management-level API calls across AWS services
15+
- **EventBridge** filters and routes specific CloudTrail events
16+
- **Kinesis Firehose** delivers those events reliably to OpenObserve over HTTP
17+
18+
> **Note:** EventBridge *can* send events directly to OpenObserve using its HTTP target support.
19+
> However, we recommend using **Kinesis Firehose** in production for:
20+
>
21+
> - Built-in retry logic and failure handling
22+
> - Optional S3 backup for undelivered events
23+
24+
25+
## Steps to Integrate
26+
27+
??? "Prerequisites"
28+
- OpenObserve account ([Cloud](https://cloud.openobserve.ai/web/) or [Self-Hosted](../../../quickstart/#self-hosted-installation))
29+
- AWS account with:
30+
- CloudTrail enabled for management events
31+
- IAM permissions to create EventBridge rules and Firehose streams
32+
33+
34+
??? "Step 1: Ensure CloudTrail Management Events Are Enabled"
35+
36+
1. Go to **AWS Console → CloudTrail → Trails**
37+
2. Check if a trail exists that logs **management events**
38+
3. If not:
39+
- Click **Create trail**
40+
- Name the trail (e.g., `DefaultManagementTrail`)
41+
- Enable **Management events** (Read/Write or Write-only)
42+
- Choose an S3 bucket for storage (existing or new)
43+
- Complete the trail creation
44+
45+
> CloudTrail acts as the *source* of API activity events across AWS.
46+
47+
48+
??? "Step 2: Get OpenObserve Ingestion URL and Access Key"
49+
50+
1. In OpenObserve: go to **Data Sources → Recommended → AWS**
51+
2. Copy the HTTP ingestion URL and Access Key
52+
53+
![Get OpenObserve Ingestion URL and Access Key](../images/aws-integrations/vpc-flow/fetch-url.png)
54+
55+
> Example ingestion URL:
56+
> ```
57+
> https://<your-openobserve-domain>/aws/default/cloudtrail-events/_kinesis_firehose
58+
> ```
59+
60+
61+
??? "Step 3: Create Firehose Delivery Stream to OpenObserve"
62+
63+
1. In AWS Kinesis Firehose, Create delivery stream with Source: `Direct PUT` and Destination: `HTTP Endpoint`.
64+
2. Provide OpenObserve's HTTP Endpoint URL and Access Key, and set an S3 backup bucket.
65+
3. Give the stream a meaningful name and Create it.
66+
67+
68+
??? "Step 4: Create EventBridge Rule to Forward CloudTrail Events"
69+
70+
1. Go to **EventBridge > Rules** and click **Create rule**
71+
2. Name it (e.g., `ForwardCloudTrailEvents`)
72+
3. Choose:
73+
- **Event Source**: `AWS events or EventBridge partner events`
74+
- **Event Pattern**:
75+
```json
76+
{
77+
"source": ["aws.cloudtrail"],
78+
"detail-type": ["AWS API Call via CloudTrail"]
79+
}
80+
```
81+
4. Set the target to **Kinesis Firehose** and choose the delivery stream
82+
5. Click **Create rule**
83+
84+
85+
??? "Step 5: Verify Logs in OpenObserve"
86+
87+
1. Go to **Logs** → select your log stream → Set time range → Click **Run Query**
88+
2. You should see logs like:
89+
```json
90+
{
91+
"eventSource": "ec2.amazonaws.com",
92+
"eventName": "StartInstances",
93+
"userIdentity": { ... },
94+
"awsRegion": "us-east-1",
95+
...
96+
}
97+
```
98+
99+
100+
??? "Troubleshooting"
101+
102+
**No logs appearing in OpenObserve?**
103+
104+
- CloudTrail:
105+
- Ensure management events are enabled and recent API activity has occurred.
106+
107+
- EventBridge:
108+
- Confirm the rule is matching events.
109+
- Check the Monitoring tab for recent invocations or matched event counts.
110+
111+
- Firehose:
112+
- Verify the delivery stream is active and error-free.
113+
- Review the Monitoring tab and look for failed deliveries.
114+
115+
- OpenObserve:
116+
- Confirm the HTTP ingestion URL and access key are correct.
117+
- In the Logs view, select the correct stream and expand the time range to view recent data.
118+
119+

‎docs/integration/images/aws-integrations/cloudwatch-logs/cloudwatch-logs.png

722 KB
Loading

‎docs/integration/images/aws-integrations/cloudwatch-logs/firehose-stream.png

315 KB
Loading

‎docs/integration/images/aws-integrations/cloudwatch-logs/subscription-filter.png

418 KB
Loading

‎docs/integration/images/aws-integrations/cognito/event-rule.png

779 KB
Loading

‎docs/integration/images/aws-integrations/cognito/log-stream.png

1.05 MB
Loading

‎docs/integration/images/aws-integrations/vpc-flow/fetch-url.png

97.9 KB
Loading

0 commit comments

Comments
 (0)