Question regarding proxying vmauth/victoriametrics with TLS and basic auth

[corporate-gadfly]

Please definitely label this as a question.

I'm in the initial stages of designing my monitoring setup. I'm planning to have 4 VMs:

• VM for dashboarding (will have grafana and promxy)
• VM for alerting (will have karma, Prometheus alertmanager and vmalert)
• VM for victoriametrics (will have vmauth for TLS as well)
• Another VM for victoriametrics for HA (will have vmauth for TLS as well)

Here's a logical diagram:

What would be a simple config in promxy which would allow TLS to vmauth/victoriametrics?

Here is my config.yaml:

global:
  evaluation_interval: 5s
  external_labels:
    source: promxy

rule_files: []

alerting:
  alertmanagers: []

remote_write: []

promxy:
  server_groups:
    - static_configs:
      - targets:
        - victoriametrics-1.xxx:8427
      anti_affinity: 10s
      remote_read: false
      remote_read_path: ''
      path_prefix: ''
      query_params:
        nocache: 1
      http_client:
       dial_timeout: 1s
       tls_config:
         insecure_skip_verify: false

And trace output shows:

TRAC[2021-03-28T17:27:47-04:00] Potential target pre-relabel: {__address__="victoriametrics-1.xxx:8427", __path_prefix__="", __scheme__="http"} 
TRAC[2021-03-28T17:27:47-04:00] Potential target post-relabel: {__address__="victoriametrics-1.xxx:8427", __path_prefix__="", __scheme__="http"}

How do I specify the targets so that scheme is https? Thanks in advance for any pointers.

[jacksontj]

the target configuration is the same as in prometheus (https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config); so if you add scheme: https to the servergroup config it should work.

[corporate-gadfly]

TY, that makes a lot of sense. If you don't mind, I'll keep this open until I can successfully reach the vmauth backend.

Currently, after adding scheme: https, I'm getting:

client_error: client error: 401

But definitely making progress in the right direction.

[corporate-gadfly]

If I issue the following curl command, directly to victoriametrics:

curl -i -u vm1:PASSWORD_HERE \
    -d 'query=sum(vm_available_cpu_cores{job="victoria-metrics", instance=~"self"})' \
    -X POST https://victoriametrics-1.xxx:8427/api/v1/query_range

I get the following JSON back:

{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
        "metric": {},
        "values": [
          [
            1617286981,
            "4"
          ],
          [
            1617287281,
            "4"
          ]
        ]
      }
    ]
  }
}

Are you able to advise if I got the equivalent config correct in the promxy config, which as follows?

global:
  evaluation_interval: 5s
  external_labels:
    source: promxy

rule_files: []

alerting:
  alertmanagers: []

remote_write: []

promxy:
  server_groups:
    - static_configs:
      - targets:
        - victoriametrics-1.xxx:8427
      scheme: https
      basic_auth:
        username: vm1
        password: PASSWORD_HERE
      anti_affinity: 10s
      remote_read: false
      remote_read_path: ''
      path_prefix: ''
      query_params:
        nocache: 1
      http_client:
       dial_timeout: 1s

Trace output from promxy is as follows (indicating 401 unauthorized):

127.0.0.1 - - [01/Apr/2021 14:47:38] "GET /api/v1/rules HTTP/1.1 200 70" 0.001521 
DEBU[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=LabelValues label=__name__
DEBU[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=LabelValues label=__name__
DEBU[2021-04-01T10:47:38-04:00] AggregateExpr sum(vm_available_cpu_cores{instance=~"self",job="victoria-metrics"}) sum 
DEBU[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=QueryRange query="sum(vm_available_cpu_cores{instance=~\"self\",job=\"victoria-metrics\"})" r="{2021-04-01 14:17:30 +0000 UTC 2021-04-01 14:47:30 +0000 UTC 15s}"
TRAC[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=QueryRange error="client_error: client error: 401" query="sum(vm_available_cpu_cores{instance=~\"self\",job=\"victoria-metrics\"})" r="{2021-04-01 14:17:30 +0000 UTC 2021-04-01 14:47:30 +0000 UTC 15s}" took=79.239443ms value="<nil>" warnings="[]"
127.0.0.1 - - [01/Apr/2021 14:47:38] "POST /api/v1/query_range HTTP/1.1 422 88" 0.080252 end=1617288450&step=15&query=sum%28vm_available_cpu_cores%7Bjob%3D%22victoria-metrics%22%2C+instance%3D~%22self%22%7D%29&start=1617286650
TRAC[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=LabelValues error="client_error: client error: 401" label=__name__ took=82.886623ms value="[]" warnings="[]"
DEBU[2021-04-01T10:47:38-04:00] LabelValues                                   name=__name__ took=83.097738ms
127.0.0.1 - - [01/Apr/2021 14:47:38] "GET /api/v1/label/__name__/values HTTP/1.1 422 88" 0.084891 start=1617286658&end=1617288458
TRAC[2021-04-01T10:47:38-04:00] https://victoriametrics-1.xxx:8427  api=LabelValues error="client_error: client error: 401" label=__name__ took=83.409056ms value="[]" warnings="[]"
DEBU[2021-04-01T10:47:38-04:00] LabelValues                                   name=__name__ took=83.545527ms
127.0.0.1 - - [01/Apr/2021 14:47:38] "GET /api/v1/label/__name__/values HTTP/1.1 422 88" 0.084164 start=1617286659&end=1617288459
127.0.0.1 - - [01/Apr/2021 14:47:38] "GET /api/v1/metadata HTTP/1.1 200 59" 0.000321 
127.0.0.1 - - [01/Apr/2021 14:47:38] "GET /api/v1/metadata HTTP/1.1 200 59" 0.000190

[corporate-gadfly]

I'm also confused, if I should be specifying basic auth in the datasource config in grafana or only in promxy config or both places:

[jacksontj]

This depends entirely on your setup. Grafana is talking to promxy -- so credentials there are a question if you require auth on the promxy endpoint. If VM requires auth then you'd need to configure it on promxy (since promxy talks to the downstream VM). There is a request to have promxy propagate auth context (#322) -- so if that is desired feel free to subscribe to that case :)

[corporate-gadfly]

TY for the response @jacksontj . Much appreciated.

Unfortunately still struggling a bit. With the following running on a single machine:

Software	Port
VictoriaMetrics	8428
vmauth	8427
promxy	8082

I can get promxy to return values from VictoriaMetrics on port 8428 with a minimal config:

promxy:
  server_groups:
    - static_configs:
        - targets:
          - victoriametrics-1:8428

What is the minimal config that I would need to get it to talk to vmauth on port 8427? So, far, I have tried following with no success (I get client_error: client error: 401):

promxy:
  server_groups:
    - static_configs:
        - targets:
          - victoriametrics-1:8427
      scheme: https
      basic_auth:
        username: vm1
        password: abc123
      http_client:
        tls_config:
          insecure_skip_verify: true

[corporate-gadfly]

I moved basic_auth under http_client and now it works out of the box. Also looked at #70 for hints. Final config with vmauth looks like as follows:

promxy:
  server_groups:
    - static_configs:
        - targets:
          - victoriametrics-1:8427
      scheme: https
      http_client:
        basic_auth:
          username: vm1
          password: abc123
        tls_config:
          insecure_skip_verify: true       # use only in non-production

Also since promxy has the basic auth configuration, data source in grafana should have Basic auth unchecked.