Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes Resource examples don't work #2434

Closed
bjan-sag opened this issue Feb 7, 2025 · 2 comments
Closed

Kubernetes Resource examples don't work #2434

bjan-sag opened this issue Feb 7, 2025 · 2 comments
Assignees
@yashmehrotra

Comments

@bjan-sag
Copy link

bjan-sag commented Feb 7, 2025

Hi,

Any time I try running an official example using the kubernetesResource check it fails. I tested on on-prem Kubernetes (v1.20.5) and on EKS (v1.29 up to v1.31). They always fail with the same error.

Example below. The httpbin pod did get created, but then something in Canary Checker crapped out. When run inside the cluster, these checks always fail with no logs nor messages.

Image

Example 
~/canaries> cat k8s-service-pass.yaml
---
apiVersion: canaries.flanksource.com/v1
kind: Canary
metadata:
  name: pod-svc-test
  namespace: default
  annotations:
    trace: "true"
  labels:
    "Expected-Fail": "false"
spec:
  schedule: "@every 5m"
  kubernetesResource:
    - name: service accessibility test
      namespace: default
      description: "deploy httpbin & check that it's accessible via its service"
      waitFor:
        expr: 'dyn(resources).all(r, k8s.isReady(r))'
        interval: 2s
        timeout: 2m
      resources:
        - apiVersion: v1
          kind: Pod
          metadata:
            name: httpbin-pod-1
            namespace: default
            labels:
              app: httpbin-pod-1
          spec:
            containers:
              - name: httpbin
                image: "kennethreitz/httpbin:latest"
                ports:
                  - containerPort: 80
        - apiVersion: v1
          kind: Service
          metadata:
            name: httpbin-svc
            namespace: default
          spec:
            selector:
              app: httpbin-pod-1
            ports:
              - port: 80
                targetPort: 80
      checks:
        - http:
            - name: Call httpbin service
              url: "http://httpbin-svc.default.svc"
      checkRetries:
        delay: 2s
        interval: 3s
        timeout: 2m
~/canaries> ./canary-checker run k8s-service-pass.yaml
14:26:25.021 WRN /home/bjan/canaries/canary-checker.properties does not exist
14:26:25.022 WRN --db not configured
14:26:25.022 INF (k8s) Using kubeconfig /home/bjan/.kube/config
14:26:25.028 INF Checking k8s-service-pass.yaml, 1 checks found
14:26:25.028 DBG (canary[default/pod-svc-test]) [pod-svc-test] checking 1 checks
W0207 14:26:25.773762  112350 warnings.go:70] unknown field "creationTimestamp"
14:26:25.773 INF Pod/default/httpbin-pod-1 (created) +kustomized
14:26:25.875 INF Service/default/httpbin-svc (created) +kustomized
14:28:32.216 INF (pod-svc-test.service accessibility test) FAIL duration=127187  Oops: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | ^
ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | ^
ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | .................................^
Time: 2025-02-07 13:28:32.216631804 +0000 UTC
Trace: 01JKG9J16RHJFFRW7E34SFHGZB
Context:
  * template: display.keys().map(k, k + ': ' + display[k]).join('
')
  * details: <nil>
  * metrics: []
  * data: map[results:map[]]
  * environment: map[canary:map[id: labels:map[Expected-Fail:false] name:pod-svc-test namespace:default] check:map[description:deploy httpbin & check that it's accessible via its service endpoint:service accessibility test id: labels:map[] name:service accessibility test] duration:0 results:map[]]
  * labels: map[]
  * namespace: default
  * name: pod-svc-test
Stacktrace:
  Oops: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/go/pkg/mod/github.com/flanksource/gomplate/v3@v3.24.35/template.go:190 RunExpressionContext()
    --- at /home/runner/go/pkg/mod/github.com/flanksource/gomplate/v3@v3.24.35/template.go:264 RunTemplateContext()
    --- at /home/runner/go/pkg/mod/github.com/flanksource/duty@v1.0.706/context/template.go:22 Context.RunTemplate()
    --- at /home/runner/work/canary-checker/canary-checker/checks/common.go:72 template()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:269 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()
  Thrown: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/go/pkg/mod/github.com/flanksource/duty@v1.0.706/context/template.go:24 Context.RunTemplate()
    --- at /home/runner/work/canary-checker/canary-checker/checks/common.go:72 template()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:269 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()
  Thrown: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:271 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()
 Oops: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | ^
ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | ^
ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
 | display.keys().map(k, k + ': ' + display[k]).join(' ')
 | .................................^
Time: 2025-02-07 13:28:32.216631804 +0000 UTC
Trace: 01JKG9J16RHJFFRW7E38F8B74R
Context:
  * environment: map[canary:map[id: labels:map[Expected-Fail:false] name:pod-svc-test namespace:default] check:map[description:deploy httpbin & check that it's accessible via its service endpoint:service accessibility test id: labels:map[] name:service accessibility test] duration:0 results:map[]]
  * details: <nil>
  * labels: map[]
  * metrics: []
  * namespace: default
  * name: pod-svc-test
  * data: map[results:map[]]
  * template: display.keys().map(k, k + ': ' + display[k]).join('
')
Stacktrace:
  Oops: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/go/pkg/mod/github.com/flanksource/gomplate/v3@v3.24.35/template.go:190 RunExpressionContext()
    --- at /home/runner/go/pkg/mod/github.com/flanksource/gomplate/v3@v3.24.35/template.go:264 RunTemplateContext()
    --- at /home/runner/go/pkg/mod/github.com/flanksource/duty@v1.0.706/context/template.go:22 Context.RunTemplate()
    --- at /home/runner/work/canary-checker/canary-checker/checks/common.go:72 template()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:269 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()
  Thrown: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/go/pkg/mod/github.com/flanksource/duty@v1.0.706/context/template.go:24 Context.RunTemplate()
    --- at /home/runner/work/canary-checker/canary-checker/checks/common.go:72 template()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:269 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()
  Thrown: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^: ERROR: <input>:1:1: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | ^
  ERROR: <input>:1:34: undeclared reference to 'display' (in container '')
   | display.keys().map(k, k + ': ' + display[k]).join(' ')
   | .................................^
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:271 processTemplates()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:213 TransformResults()
    --- at /home/runner/work/canary-checker/canary-checker/checks/runchecks.go:127 RunChecks()
    --- at /home/runner/work/canary-checker/canary-checker/cmd/run.go:69 init.func5.1()
    --- at /home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.0.linux-amd64/src/runtime/asm_amd64.s:1700 goexit()

14:28:32.217 INF 0 passed, 1 failed in 2m
@moshloop
Copy link
Member

@bjan-sag can you retest with v1.1.0 ?

@bjan-sag
Copy link
Author

Works now - thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yashmehrotra yashmehrotra

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@moshloop @yashmehrotra @bjan-sag