Fluentd suddenly stopped sending logs to LA #10
Description
Hi Team,
We are running https://github.com/oracle-quickstart/oci-kubernetes-monitoring in our kubernetes cluster. We are noticing a very strange thing that some nodes stopped sending logs to LA. oci-onm-logan pods are generating normal logs that they are tailing the log files but oci-logging-analytics.log inside the pod /var/log log file stopped generating any logs and since then there are no logs uploaded to LA.
We have enabled DEBUG logs also but there is nothing useful. Can you please help us with this situation? This is happening very randomly. The solution for now is to restart the pod and then it again starts sending logs normally but that is manual. We would like to know what will be permanent fix for it or how to debug this?
Regards,
Sourabh