[Nvidia/GPU] Introduce Nvidia GPU Integration

[strawgate]

Proposed commit message

Introduce NVIDIA GPU Monitoring Integration

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

How to test this PR locally

Deploy NVIDIA DGCM on a device with an NVIDIA GPU to get a prometheus metrics endpoint that you can provide to the integration.

If you have docker this just requires:

docker run -d --gpus all --rm -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:3.3.9-3.6.1-ubuntu22.04
curl localhost:9400/metrics

Configure the integration to point at the host running the container and GPU http://nvidiahost:9400/metrics

Some metrics are not enabled by default with the container, enabling all metrics requires some extra steps.

Related issues

Fixes #11930

Screenshots

WIP:

[strawgate]

Todo: Add k8s container, pod, and namespace info from labels

Should also include labels / mapping for kubernetes container, pod, and namespace

container="dcgmproftester11",namespace="default",pod="dcgmproftester"

container => kubernetes.container.name
namespace => kubernetes.namespace
pod => kubernetes.pod.name

- name: kubernetes
  type: group
  fields:
    - name: pod.name
      type: keyword
      description: >
        Kubernetes pod name
    - name: container.name
      type: keyword
      description: >
        Kubernetes container name
    - name: namespace
      type: keyword
      description: >
        Kubernetes namespace

- rename:
    field: prometheus.labels.container
    target_field: kubernetes.container.name
    ignore_missing: true
- rename:
    field: prometheus.labels.namespace
    target_field: kubernetes.namespace
    ignore_missing: true
- rename:
    field: prometheus.labels.pod
    target_field: kubernetes.pod.name
    ignore_missing: true

Perhaps with corresponding dashboard elements

[ishleenk17]

Thanks for the contribution.
Shared inital set of comments from the 1st review.

[ishleenk17]

We don't want these options to be configurable ?

[strawgate]

the ingest pipeline and mappings expect a certain format coming from the prometheus input of metricbeat, i believe changing these will change the format and the ingest pipeline/mappings will not be valid

[agithomas]

While going through this issue, there is a mention about

• high resource utilisation
• deprecation of a few metrics including DCGM_FI_DEV_GPU_UTIL.

Is this scenario observed while integration testing? If there are metrics that are deprecated or result in high resource intensive, it would be best to not consider this metric for creating dashboard visualisation.

[strawgate]

Will remove this metric and consider replacing with the referenced ones

[agithomas]

Added @daniela-elastic as the reviewer for the dashboard.

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 11d5263

Failed CI Steps

• Check integrations nvidia_gpu

History

• 💔 Build #22526 failed 58dae3b
• 💔 Build #22483 failed 4aaef17
• 💔 Build #22255 failed ec8bb4a
• 💔 Build #22241 failed 0bce5f9