Monitor docker with datadog

To monitor Docker containers with Datadog, you can use the Datadog Docker integration. Here are the general steps:

1. Install the Datadog Agent on your Docker host(s), as described in the previous answer.
2. Enable the Docker integration in the Datadog Agent configuration file by adding the following lines to /etc/datadog-agent/datadog.yaml:

logs_enabled: true
logs_config:
  - type: docker
    image: "*"

This configuration will enable Docker container logging and collect logs from all containers.

3. Restart the Datadog Agent service:

sudo systemctl restart datadog-agent

4. Install the Datadog Agent on each Docker container you want to monitor by adding the following lines to your Dockerfile:

RUN DD_API_KEY=<your_api_key> DD_APM_ENABLED=true DD_LOGS_ENABLED=true DD_TRACE_ENABLED=true DD_SITE="datadoghq.com" bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/datadog-agent/master/cmd/agent/install_script.sh)"

This will install the Datadog Agent and enable APM, logs, and tracing on the container.

5. Start your Docker container(s) and ensure that they are sending data to your Datadog account.

You can check the Datadog Agent logs for any errors by running the following command on the Docker host:

sudo tail -f /var/log/datadog/agent.log

You can also use the Datadog Agent Status page to verify that the Docker containers are being monitored.

6. Configure your Datadog dashboards and alerts to visualize and alert on the Docker container metrics and logs.

Here’s an example dashboard that displays Docker container CPU usage and memory usage:

{
    "dash": {
        "description": "",
        "layout_type": "ordered",
        "title": "Docker Containers",
        "graphs": [
            {
                "definition": {
                    "requests": [
                        {
                            "q": "avg:docker.cpu.usage{*}"
                        }
                    ],
                    "visualization_type": "timeseries"
                },
                "title": "CPU Usage"
            },
            {
                "definition": {
                    "requests": [
                        {
                            "q": "avg:docker.mem.usage{*} by {container_name}"
                        }
                    ],
                    "visualization_type": "timeseries"
                },
                "title": "Memory Usage"
            }
        ]
    }
}

This dashboard will display the average CPU usage for all Docker containers, as well as the average memory usage for each container.

You can also create alerts that trigger when Docker container metrics exceed certain thresholds. For example, you can create an alert that triggers when a container’s CPU usage exceeds 90%:

{
    "name": "Docker CPU Usage Alert",
    "query": "avg:docker.cpu.usage{*} by {container_name} > 90",
    "message": "CPU usage for container {{container_name}} is above 90%",
    "options": {
        "notify_no_data": true,
        "no_data_timeframe": 10,
        "escalation_message": "The CPU usage for container {{container_name}} is still above 90%",
        "escalation_message_priority": "warning"
    }
}

This alert will send a notification when a container’s CPU usage exceeds 90%, and escalate the notification if the issue persists.