Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions config/_default/menus/main.en.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3418,6 +3418,11 @@ menu:
parent: containers_troubleshooting
identifier: containers_troubleshooting_admission_controller
weight: 905
- name: Log Collection
url: containers/troubleshooting/log-collection
parent: containers_troubleshooting
identifier: containers_troubleshooting_log_collection
weight: 906
- name: Guides
url: containers/guide
parent: containers
Expand Down
8 changes: 8 additions & 0 deletions content/en/containers/docker/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ further_reading:
- link: "/agent/docker/tag/"
tag: "Documentation"
text: "Assign tags to all data emitted by a container"
- link: "/containers/troubleshooting/log-collection"
tag: "Documentation"
text: "Container Log Collection Troubleshooting"
---

## Overview
Expand Down Expand Up @@ -349,6 +352,10 @@ Since Agent v6.14+, the Agent collects logs for all containers (running or stopp

For Kubernetes environments, see the [Kubernetes short lived container documentation][12].

## Troubleshooting

For troubleshooting steps, see [Container Log Collection Troubleshooting][13].

## Further Reading

{{< partial name="whats-next/whats-next.html" >}}
Expand All @@ -365,3 +372,4 @@ For Kubernetes environments, see the [Kubernetes short lived container documenta
[10]: /logs/guide/docker-logs-collection-troubleshooting-guide/
[11]: /agent/guide/autodiscovery-management/
[12]: /agent/kubernetes/log/?tab=daemonset#short-lived-containers
[13]: /containers/troubleshooting/log-collection/
67 changes: 5 additions & 62 deletions content/en/containers/kubernetes/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ further_reading:
- link: "/agent/kubernetes/tag/"
tag: "Documentation"
text: "Assign tags to all data emitted by a container"
- link: "/containers/troubleshooting/log-collection"
tag: "Documentation"
text: "Container Log Collection Troubleshooting"
---

This page discusses collecting logs from Kubernetes log files.
Expand Down Expand Up @@ -462,68 +465,7 @@ The equivalent volume and volumeMount path need to be set in the Agent container

## Troubleshooting

#### Short lived containers

By default the Agent looks every 5 seconds for new containers.

For Agent v6.12+, short lived container logs (stopped or crashed) are automatically collected when using the K8s file log collection method (through `/var/log/pods`). This also includes the collection init container logs.

#### Missing tags on new containers or Pods

When sending logs to Datadog from newly created containers or Pods, the Datadog Agent's internal tagger may not yet have the related container/pod tags. As a result, tags may be missing from these logs.

To remediate this issue, you can use the environment variable `DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION` to configure a duration (in seconds) for the Datadog Agent to wait before it begins to send logs from newly created containers and Pods. The default value is `0`.

{{< tabs >}}
{{% tab "Datadog Operator" %}}

```yaml
spec:
override:
nodeAgent:
env:
- name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
value: "5"
```
{{% /tab %}}
{{% tab "Helm" %}}
```yaml
datadog:
env:
- name: DD_LOGS_CONFIG_TAGGER_WARMUP_DURATION
value: "5"
```
{{% /tab %}}
{{< /tabs >}}

#### Missing host-level tags on new hosts or nodes

Host-level tags are those seen in the infrastructure list for a given host, and are sourced from either a cloud provider or the Datadog Agent. Common host-level tags include `kube_cluster_name`, `region`, `instance-type`, and `autoscaling-group`.

When sending logs to Datadog from a newly created host or node, it can take a few minutes for host-level tags to be [inherited][12]. As a result, host-level tags may be missing from these logs.

To remediate this issue, you can use the environment variable `DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION` to configure a duration (in minutes). For this duration, the Datadog Agent manually attaches the host-level tags that it knows about to each sent log. After this duration, the Agent reverts to relying on tag inheritance at intake.

{{< tabs >}}
{{% tab "Datadog Operator" %}}
```yaml
spec:
override:
nodeAgent:
env:
- name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
value: "10m"
```
{{% /tab %}}
{{% tab "Helm" %}}
```yaml
datadog:
env:
- name: DD_LOGS_CONFIG_EXPECTED_TAGS_DURATION
value: "10m"
```
{{% /tab %}}
{{< /tabs >}}
For troubleshooting steps, see [Container Log Collection Troubleshooting][21].

## Further Reading

Expand All @@ -549,3 +491,4 @@ datadog:
[18]: https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
[19]: /containers/kubernetes/log/?tab=helm#autodiscovery-annotations
[20]: /containers/kubernetes/log/?tab=helm#autodiscovery-configuration-files
[21]: /containers/troubleshooting/log-collection/?tab=datadogoperator
Loading
Loading