---

build: list: never publishResources: false render: never sitemap: disable: true

VictoriaMetrics datasource for Grafana

The VictoriaMetrics datasource plugin allows to query and visualize data from VictoriaMetrics in Grafana.

• Motivation
• Capabilities
• Installation
• How to use WITH templates
• How to make new release
• Frequently Asked Questions
• License

Motivation

Thanks to VictoriaMetrics compatibility with Prometheus API most users can use Prometheus datasource for Grafana to query data from VictoriaMetrics. But with time, Prometheus and VictoriaMetrics diverge more and more. The benefits of using VictoriaMetrics plugin are the following:

• MetricsQL functions support;
• Supports query tracing in Explore mode or right in panel's expressions;
• Supports WITH expressions;
• Plugin fixes label names validation issue;
• Integration with vmui.

Capabilities

1. Use MetricsQL to query metrics in Grafana.
2. Use Explore mode with Grafana.
3. Build dashboards and setup alerts.
4. Use Ad Hoc filters.
5. Template queries and expressions.
6. Get insights about query execution bottlenecks via tracing.
7. Automatically format queries via Prettify button.

Try it at VictoriaMetrics playground!

Installation

For detailed instructions on how to install the plugin on Grafana Cloud or locally, please checkout the Plugin installation docs.

Install via Docker

To install plugin via Grafana docker container specify GF_INSTALL_PLUGINS=victoriametrics-metrics-datasource environment variable. It will pre-install the plugin on container start. See more details at Configure a Grafana Docker image.

Grafana Provisioning

Provision of Grafana plugin requires to create datasource config file.

Example of config file for provisioning VictoriaMetrics datasource is the following:

apiVersion: 1

# List of data sources to insert/update depending on what's
# available in the database.
datasources:
  # <string, required> Name of the VictoriaMetrics datasource
  # displayed in Grafana panels and queries.
  - name: VictoriaMetrics
    # <string, required> Sets the data source type.
    type: victoriametrics-metrics-datasource
      # <string, required> Sets the access mode, either
      # proxy or direct (Server or Browser in the UI).
      # Some data sources are incompatible with any setting
    # but proxy (Server).
    access: proxy
    # <string> Sets default URL of the single node version of VictoriaMetrics
    url: http://victoriametrics:8428
    # <string> Sets the pre-selected datasource for new panels.
    # You can set only one default data source per organization.
    isDefault: true

    # <string, required> Name of the VictoriaMetrics datasource
    # displayed in Grafana panels and queries.
  - name: VictoriaMetrics - cluster
    # <string, required> Sets the data source type.
    type: victoriametrics-metrics-datasource
    # <string, required> Sets the access mode, either
    # proxy or direct (Server or Browser in the UI).
    # Some data sources are incompatible with any setting
    # but proxy (Server).
    access: proxy
    # <string> Sets default URL of the cluster version of VictoriaMetrics
    url: http://vmselect:8481/select/0/prometheus
    # <string> Sets the pre-selected datasource for new panels.
    # You can set only one default data source per organization.
    isDefault: false

Please find the example of provisioning Grafana instance with VictoriaMetrics datasource below:

1. Create a file at ./provisioning/datasources/vm.yml with datasource example file.

2. Define Grafana installation via docker-compose:

     version: '3.0'
     services:
       grafana:
         image: grafana/grafana:11.0.0
         environment:
         - GF_INSTALL_PLUGINS=victoriametrics-metrics-datasource
         ports:
         - 3000:3000/tcp
         volumes:
         - ./provisioning:/etc/grafana/provisioning
   

3. Run docker-compose file:

docker-compose -f docker-compose.yaml up

When Grafana starts successfully datasources should be present on the datasources tab:

Install in Kubernetes

Grafana helm chart

Example with Grafana helm chart:

Option 1. Using Grafana provisioning:

env:
  GF_INSTALL_PLUGINS: victoriametrics-metrics-datasource

Option 2. Using Grafana plugins section in values.yaml:

plugins:
  - victoriametrics-metrics-datasource

Option 3. Using init container:

extraInitContainers:
  - name: "load-vm-ds-plugin"
    image: "curlimages/curl:7.85.0"
    command: [ "/bin/sh" ]
    workingDir: "/var/lib/grafana"
    securityContext:
      runAsUser: 472
      runAsNonRoot: true
      runAsGroup: 472
    args:
     - "-c"
     - |
       set -ex
       mkdir -p /var/lib/grafana/plugins/
       ver=$(curl -s -L https://api.github.com/repos/VictoriaMetrics/victoriametrics-datasource/releases/latest | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1)
       curl -L https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/$ver/victoriametrics-metrics-datasource-$ver.tar.gz -o /var/lib/grafana/plugins/vm-plugin.tar.gz
       tar -xf /var/lib/grafana/plugins/vm-plugin.tar.gz -C /var/lib/grafana/plugins/
       rm /var/lib/grafana/plugins/vm-plugin.tar.gz
    volumeMounts:
      # For grafana-operator users, change `name: storage` to `name: grafana-data`
      - name: storage
        mountPath: /var/lib/grafana

For grafana-operator users, the above configuration should be done for the part /spec/deployment/spec/template/spec/initContainers of your kind=Grafana resource.

This example uses init container to download and install plugin. To allow Grafana using this container as a sidecar set the following config:

sidecar:
  datasources:
    initDatasources: true
    enabled: true

See more about chart settings here

Grafana operator

Example with Grafana operator:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-vm
spec:
  persistentVolumeClaim:
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 200Mi
  deployment:
    spec:
      template:
        spec:
          initContainers:
            - name: "load-vm-ds-plugin"
              image: "curlimages/curl:7.85.0"
              command: [ "/bin/sh" ]
              workingDir: "/var/lib/grafana"
              securityContext:
                runAsUser: 472
                runAsNonRoot: true
                runAsGroup: 472
              args:
                - "-c"
                - |
                  set -ex
                  mkdir -p /var/lib/grafana/plugins/
                  ver=$(curl -s https://api.github.com/repos/VictoriaMetrics/victoriametrics-datasource/releases/latest | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1)
                  curl -L https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/$ver/victoriametrics-metrics-datasource-$ver.tar.gz -o /var/lib/grafana/plugins/vm-plugin.tar.gz
                  tar -xf /var/lib/grafana/plugins/vm-plugin.tar.gz -C /var/lib/grafana/plugins/
                  rm /var/lib/grafana/plugins/vm-plugin.tar.gz
              volumeMounts:
                - name: grafana-data
                  mountPath: /var/lib/grafana

See Grafana operator reference to find more about Grafana operator. This example uses init container to download and install plugin.

It is also possible to request plugin at GrafanaDatasource or GrafanaDashboard CRDs. For example:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: vm-datasource
spec:
  datasource:
    access: proxy
    type: victoriametrics-metrics-datasource
    name: VM
    url: http://vmsingle-vm-stack-victoria-metrics-k8s-stack.monitoring.svc.cluster.local:8429
  instanceSelector:
    matchLabels:
      dashboards: grafana
  plugins:
    - name: victoriametrics-metrics-datasource
      version: "0.14.0"
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: vm-dashboard
spec:
  resyncPeriod: 30s
  plugins:
    - name: victoriametrics-metrics-datasource
      version: "0.14.0"
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/VictoriaMetrics/VictoriaMetrics/refs/heads/master/dashboards/vm/victoriametrics.json"

Dev release installation

1. To download plugin build and move contents into Grafana plugins directory:

   ver=$(curl -s https://api.github.com/repos/VictoriaMetrics/victoriametrics-datasource/releases/latest | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1)
   curl -L https://github.com/VictoriaMetrics/victoriametrics-datasource/releases/download/$ver/victoriametrics-metrics-datasource-$ver.tar.gz -o /var/lib/grafana/plugins/vm-plugin.tar.gz
   tar -xf /var/lib/grafana/plugins/vm-plugin.tar.gz -C /var/lib/grafana/plugins/
   rm /var/lib/grafana/plugins/vm-plugin.tar.gz
   

2. Restart Grafana

Getting started development

1. Install Grafana

Tip for Apple(arm64)

To download the arm64 build of grafana for macOS from the grafana download page, need to change amd64 to arm64 in the wget url.

More details about debugging grafana plugin on Apple Silicon can be found in this article.

2. Configure Grafana

Installing dev version of Grafana plugin requires to change grafana.ini config to allow loading unsigned plugins:

# Directory where Grafana will automatically scan and look for plugins
plugins = {{path to directory with plugin}}

[plugins]
allow_loading_unsigned_plugins = victoriametrics-metrics-datasource

3. Run the plugin

1. How to run a frontend plugin in development mode:

In the project directory, you can run:

# install dependencies
yarn install

# run the app in the development mode
yarn dev

# build the plugin for production to the `dist` folder and zip build
yarn build:zip

2. How to run a debugger for a backend plugin:

1. install delve

  go install github.com/go-delve/delve/cmd/dlv@latest

2. install mage

  go install github.com/magefile/mage@latest

3. install dependencies

  go mod download

4. build backend plugin

  mage build

5. build frontend plugin:

  make vl-frontend-plugin-build

6. run grafana
7. run debugger for backend plugin

  mage debugger

8. run debugger in IDE with the following configuration - port:3222.

4. How to build backend plugin

From the root folder of the project run the following command:

make vm-backend-plugin-build

This command will build executable multi-platform files to the dist folder for the following platforms:

• linux/amd64
• linux/arm64
• linux/arm
• linux/386
• amd64
• arm64
• windows

5.How to build frontend plugin

From the root folder of the project run the following command:

make vm-frontend-plugin-build

This command will build all frontend app into dist folder.

6. How to build frontend and backend parts of the plugin:

When frontend and backend parts of the plugin is required, run the following command from the root folder of the project:

make vm-plugin-build

This command will build frontend part and backend part or the plugin and locate both parts into dist folder.

7. How to build and run plugin via docker compose:

From the root folder of the project run the following command:

make vm-plugin-build

Then you can run grafana with the installed plugin by running the following command:

docker compose up [-d]

The -d flag is optional and will run grafana in detached mode. The grafana data is stored in ./grafana folder, so you can stop and start grafana without losing data.

How to use WITH templates

The WITH templates feature simplifies the construction and management of complex queries. You can try this feature in the WITH templates playground.

WITH templates are stored as a hidden dashboard variable and are automatically included when exporting or importing dashboards.

You can use Grafana variables inside WITH template expressions (e.g. $instance, $job). Panels will automatically refresh when variable values change.

Defining WITH Expressions

1. Navigate to the dashboard where you want to add a template.
   

2. Click the WITH templates button next to any query.

3. Enter the expression in the input field. Once done, press the Save button to apply the changes. For example:

   commonFilters = {instance=~"$node:$port",job=~"$job"},
   
   # `cpuCount` is the number of CPUs on the node
   cpuCount = count(count(node_cpu_seconds_total{commonFilters}) by (cpu)),
   
   # `cpuIdle` is the sum of idle CPU cores
   cpuIdle = sum(rate(node_cpu_seconds_total{mode='idle',commonFilters}[5m]))
   

   You can specify a comment before the variable and use markdown in it. The comment will be displayed as a hint during auto-completion. The comment can span multiple lines.

Using WITH Expressions

After saving the template, you can enter it into the query editor field:

((cpuCount - cpuIdle) * 100) / cpuCount

Thus, the entire query will look as follows:

WITH (
 commonFilters = {instance=~"$node:$port",job=~"$job"},
 cpuCount = count(count(node_cpu_seconds_total{commonFilters}) by (cpu)),
 cpuIdle = sum(rate(node_cpu_seconds_total{mode='idle',commonFilters}[5m]))
)
((cpuCount - cpuIdle) * 100) / cpuCount

To view the raw query in the interface, enable the Raw toggle.

How to make new release

1. Make sure there are no open security issues.
2. Change version in package.json in a main branch
3. Push changes to the github repository and be sure that the main branch is up to date.
4. Trigger release pipeline.
5. Go to releases page once pipeline is finished and verify release with the name TAG has been created and has all the needed binaries and checksums attached.

FAQ

How to convert dashboard from Prometheus to VictoriaMetrics datasource?

Make sure that VictoriaMetrics datasource plugin is installed, and a new datasource is created from the plugin.

Each panel in Grafana dashboard has a datasource dropdown when in Edit mode. Just choose the VictoriaMetrics datasource instead of Prometheus datasource in dropdown.

If datasource is configured via Grafana variable, then change variable to VictoriaMetrics datasource type.

Why VictoriaMetrics datasource doesn't support alerting?

Grafana doesn't allow forwarding Alert requests to alerting API /api/v1/rules for plugins which are not of Prometheus or Loki type. See more details here.

License

This project is licensed under the AGPL-3.0-only.