A question about 'Out of Memory' #159

yangyongzhi7 · 2019-04-12T06:41:37Z

Hello . you guys:

I have deploy a tiller and swift into my k8s cluster. It seem everything is OK.
But the swift pod will been killed about every 2 hours.
The kernel log located in path "/var/log" :
Apr 12 08:43:14 VM_133_12_centos kernel: Out of memory: Kill process 6593 (swift) score 1470 or sacrifice child Apr 12 08:43:14 VM_133_12_centos kernel: Out of memory: Kill process 6611 (swift) score 1470 or sacrifice child Apr 12 10:10:49 VM_133_12_centos kernel: Out of memory: Kill process 20516 (swift) score 1455 or sacrifice child Apr 12 10:56:59 VM_133_12_centos kernel: Out of memory: Kill process 11716 (swift) score 1457 or sacrifice child Apr 12 10:56:59 VM_133_12_centos kernel: Out of memory: Kill process 11737 (swift) score 1457 or sacrifice child Apr 12 11:43:03 VM_133_12_centos kernel: Out of memory: Kill process 27510 (swift) score 1457 or sacrifice child Apr 12 13:14:57 VM_133_12_centos kernel: Out of memory: Kill process 11044 (swift) score 1457 or sacrifice child Apr 12 13:14:57 VM_133_12_centos kernel: Out of memory: Kill process 11075 (swift) score 1458 or sacrifice child

Tiller version:
Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}

Swift version
appscode/swift:0.11.0

Kubernetes version
Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-tke.3", GitCommit:"53e244be925234190938376fe8637189b6caf125", GitTreeState:"clean", BuildDate:"2018-12-04T04:10:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-tke.3", GitCommit:"53e244be925234190938376fe8637189b6caf125", GitTreeState:"clean", BuildDate:"2018-12-04T04:10:15Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

I will post the more clues as soon as I find them.

Thanks.

The text was updated successfully, but these errors were encountered:

yangyongzhi7 · 2019-04-12T06:47:45Z

The memory utilization metric of swift pod:
`sum(container_memory_usage_bytes{namespace="kube-system", pod_name="swift-swift-c5d5c7bf7-65k9v", container_name!="POD", container_name!=""}) by (container_name)

{container_name="swift"} | 6474194944
it is about 6.4G.
`

Out of memory: Kill process 11075 (swift) score 1458 or sacrifice child messages-Apr 12 13:14:57 VM_133_12_centos kernel: Killed process 11075 (swift) total-vm:10307388kB, anon-rss:3658748kB, file-rss:4kB, shmem-rss:0kB

At every time the swift has been killed, almost it's memory utilization is about 10307388kB.

tamalsaha · 2019-04-12T06:55:34Z

can you run the command netstat | wc -l ?

yangyongzhi7 · 2019-04-12T07:03:02Z

can you run the command netstat | wc -l ?
Should I execute this command into the container? Or just on the host?

yangyongzhi7 · 2019-04-12T07:06:14Z

can you run the command netstat | wc -l ?

For host:
[root@VM_133_12_centos log]# netstat | wc -l 447

For container:
[root@VM_133_12_centos log]# ks exec -it swift-swift-c5d5c7bf7-65k9v sh ~ $ netstat Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 swift-swift-c5d5c7bf7-65k9v:33574 kubernetes.default.svc.cluster.local:https ESTABLISHED tcp 0 9 localhost:57582 localhost:9855 ESTABLISHED tcp 0 9 localhost:9855 localhost:57582 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path

yangyongzhi7 · 2019-04-12T09:52:01Z

the output of pstack:
[root@VM_133_11_centos ~]# pstack 5629 Thread 1 (process 5629): #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:532 #1 0x0000000000429ddb in runtime.futexsleep (addr=0x207d6a0 <runtime.m0+320>, ns=-1, val=0) at /usr/local/go/src/runtime/os_linux.go:46 #2 0x000000000040a962 in runtime.notesleep (n=0x207d6a0 <runtime.m0+320>) at /usr/local/go/src/runtime/lock_futex.go:151 #3 0x00000000004328ea in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2165 #4 0x0000000000434019 in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2565 #5 0x000000000043419e in runtime.park_m (gp=0xc000255500) at /usr/local/go/src/runtime/proc.go:2676 #6 0x0000000000459f3b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 #7 0x0000000000459e59 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201 #8 0x0000000000000000 in ?? ()

the output of top
`PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

5629 nfsnobo+ 20 0 2869436 1.7g 9888 S 106.4 22.3 35:57.19 swift `

yangyongzhi7 · 2019-04-12T11:08:39Z

The progress:
The OOM problem has disappeared after I use version 0.10 instead of 0.11.

So I think this problem arises from upgrading the version of swift?

yangyongzhi7 · 2019-04-12T11:42:25Z

$ pstack 11008
Thread 17 (Thread 0x7febd9ffb700 (LWP 11874)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf82 in runtime.futexsleep (addr=0x22488a0 <runtime.timers+416>, val=0, ns=4999996575) at /usr/local/go/src/runtime/os_linux.go:63
#2  0x000000000040c562 in runtime.notetsleep_internal (n=0x22488a0 <runtime.timers+416>, ns=4999996575, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3  0x000000000040c6bc in runtime.notetsleepg (n=0x22488a0 <runtime.timers+416>, ns=4999996575, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4  0x000000000044d86a in runtime.timerproc (tb=0x2248880 <runtime.timers+384>) at /usr/local/go/src/runtime/time.go:311
#5  0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6  0x0000000002248880 in runtime.timers ()
#7  0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7febda7fc700 (LWP 11031)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0000a39c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0000a39c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc00005b400, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc00056cc00) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c0000a5800 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febda7fc700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 15 (Thread 0x7febdaffd700 (LWP 11026)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0000a2f48, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0000a2f48) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc00056cf00) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c0000a5380 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdaffd700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 14 (Thread 0x7febdb7fe700 (LWP 11025)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0003c84c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0003c84c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc00005d900, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc000245e00) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c000244180 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdb7fe700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 13 (Thread 0x7febdbfff700 (LWP 11020)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0003c92c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0003c92c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000056a00, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc00056cc00) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c0003c0900 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdbfff700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 12 (Thread 0x7febfcde7700 (LWP 11019)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0002f24c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0002f24c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc00056c780) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c0000fc600 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfcde7700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 11 (Thread 0x7febfd5e8700 (LWP 11018)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0001d5d48, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0001d5d48) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000054500, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc000245800) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c000245680 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfd5e8700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 10 (Thread 0x7febfdde9700 (LWP 11017)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc0001d59c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc0001d59c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000058f00, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc000245b00) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c000245200 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfdde9700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 9 (Thread 0x7febfe5ea700 (LWP 11016)):
#0  runtime.epollwait () at /usr/local/go/src/runtime/sys_linux_amd64.s:675
#1  0x000000000042cdc6 in runtime.netpoll (block=true, ~r1=...) at /usr/local/go/src/runtime/netpoll_epoll.go:71
#2  0x0000000000435ca5 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2380
#3  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#4  0x0000000000436ba1 in runtime.park_m (gp=0xc00056cf00) at /usr/local/go/src/runtime/proc.go:2605
#5  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#6  0x000000c0000fc300 in ?? ()
#7  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#8  github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#9  0x00007fec10dcd5aa in start_thread (arg=0x7febfe5ea700) at pthread_create.c:463
#10 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 8 (Thread 0x7febfedeb700 (LWP 11015)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf82 in runtime.futexsleep (addr=0x22487a0 <runtime.timers+160>, val=0, ns=3215267039472) at /usr/local/go/src/runtime/os_linux.go:63
#2  0x000000000040c562 in runtime.notetsleep_internal (n=0x22487a0 <runtime.timers+160>, ns=3215267039472, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3  0x000000000040c6bc in runtime.notetsleepg (n=0x22487a0 <runtime.timers+160>, ns=3215267039472, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4  0x000000000044d86a in runtime.timerproc (tb=0x2248780 <runtime.timers+128>) at /usr/local/go/src/runtime/time.go:311
#5  0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6  0x0000000002248780 in runtime.timers ()
#7  0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7febff5ec700 (LWP 11014)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0xc00003f2c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0xc00003f2c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4  0x0000000000435cd0 in runtime.findrunnable (gp=0xc000062500, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5  0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6  0x0000000000436ba1 in runtime.park_m (gp=0xc0004bc180) at /usr/local/go/src/runtime/proc.go:2605
#7  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8  0x000000c000001500 in ?? ()
#9  0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febff5ec700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 6 (Thread 0x7febfffff700 (LWP 11013)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0x22675d8 <runtime.newmHandoff+24>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0x22675d8 <runtime.newmHandoff+24>) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x0000000000434ae3 in runtime.templateThread () at /usr/local/go/src/runtime/proc.go:1914
#4  0x00000000004337d4 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1206
#5  0x00000000004336ef in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1172
#6  0x0000000000401a83 in runtime/cgo(.text) ()
#7  0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fec0cff7700 (LWP 11012)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf82 in runtime.futexsleep (addr=0x2248820 <runtime.timers+288>, val=0, ns=29999939229) at /usr/local/go/src/runtime/os_linux.go:63
#2  0x000000000040c562 in runtime.notetsleep_internal (n=0x2248820 <runtime.timers+288>, ns=29999939229, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3  0x000000000040c6bc in runtime.notetsleepg (n=0x2248820 <runtime.timers+288>, ns=29999939229, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4  0x000000000044d86a in runtime.timerproc (tb=0x2248800 <runtime.timers+256>) at /usr/local/go/src/runtime/time.go:311
#5  0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6  0x0000000002248800 in runtime.timers ()
#7  0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fec0d7f8700 (LWP 11011)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0x2267740 <runtime.sig>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c4b8 in runtime.notetsleep_internal (n=0x2267740 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:174
#3  0x000000000040c6bc in runtime.notetsleepg (n=0x2267740 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4  0x000000000044529c in os/signal.signal_recv (~r0=<optimized out>) at /usr/local/go/src/runtime/sigqueue.go:139
#5  0x00000000005b57b2 in os/signal.loop () at /usr/local/go/src/os/signal/signal_unix.go:23
#6  0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#7  0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fec0dff9700 (LWP 11010)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf82 in runtime.futexsleep (addr=0x2248720 <runtime.timers+32>, val=0, ns=14867423858) at /usr/local/go/src/runtime/os_linux.go:63
#2  0x000000000040c562 in runtime.notetsleep_internal (n=0x2248720 <runtime.timers+32>, ns=14867423858, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3  0x000000000040c6bc in runtime.notetsleepg (n=0x2248720 <runtime.timers+32>, ns=14867423858, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4  0x000000000044d86a in runtime.timerproc (tb=0x2248700 <runtime.timers>) at /usr/local/go/src/runtime/time.go:311
#5  0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6  0x0000000002248700 in ?? ()
#7  0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fec0e7fa700 (LWP 11009)):
#0  runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:131
#1  0x000000000043af9b in runtime.sysmon () at /usr/local/go/src/runtime/proc.go:4282
#2  0x00000000004337d4 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1206
#3  0x00000000004336ef in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1172
#4  0x0000000000401a83 in runtime/cgo(.text) ()
#5  0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fec111e1740 (LWP 11008)):
#0  runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1  0x000000000042cf0b in runtime.futexsleep (addr=0x2244b08 <runtime.m0+328>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2  0x000000000040c3d1 in runtime.notesleep (n=0x2244b08 <runtime.m0+328>) at /usr/local/go/src/runtime/lock_futex.go:151
#3  0x000000000043521c in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4  0x0000000000436a2c in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5  0x0000000000436ba1 in runtime.park_m (gp=0xc0003c0480) at /usr/local/go/src/runtime/proc.go:2605
#6  0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#7  0x000000000045be49 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201
#8  0x0000000000000000 in ?? ()

yangyongzhi7 · 2019-04-15T07:34:53Z

I had tried to use 0.11.1 version to debug on my local PC.
and it looks work very well.

These are about heap-top output :

      flat  flat%   sum%        cum   cum%
     513kB 50.04% 50.04%      513kB 50.04%  

github.com/appscode/swift/vendor/golang.org/x/net/http2/hpack.newInternalNode

  512.25kB 49.96%   100%   512.25kB 49.96%  

github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseProperties

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/pkg/apis/swift/v2.init.2

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parse

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseReference

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseSchema

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*SchemaLoader).Compile

         0     0%   100%   512.25kB 49.96%  github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.NewSchema

         0     0%   100%      513kB 50.04%  github.com/appscode/swift/vendor/golang.org/x/net/http2/hpack.init.ializers

         0     0%   100%  1025.25kB   100%  runtime.main

If I switch to 0.11.0 again, the output display that there is a high utilization:

Showing top 10 nodes out of 25
      flat  flat%   sum%        cum   cum%

   37376kB 97.31% 97.31%    37376kB 97.31%  bytes.makeSlice

  520.04kB  1.35% 98.67%   520.04kB  1.35%  net/http.(*http2ClientConn).frameScratchBuffer

  512.06kB  1.33%   100%   512.06kB  1.33%  net/url.parse

         0     0%   100%    37376kB 97.31%  bytes.(*Buffer).Write

         0     0%   100%    37376kB 97.31%  bytes.(*Buffer).grow

         0     0%   100%   512.06kB  1.33%  github.com/appscode/swift/pkg/apis/swift/v2.init.2

         0     0%   100%    37376kB 97.31%  github.com/appscode/swift/vendor/github.com/soheilhy/cmux.(*bufferedReader).Read

         0     0%   100%    37376kB 97.31%  github.com/appscode/swift/vendor/github.com/soheilhy/cmux.(*cMux).serve

         0     0%   100%    37376kB 97.31%  github.com/appscode/swift/vendor/github.com/soheilhy/cmux.HTTP2MatchHeaderFieldSendSettings.func1

         0     0%   100%    37376kB 97.31%  github.com/appscode/swift/vendor/github.com/soheilhy/cmux.matchHTTP2Field

So I think the issue has been fixed for a while, and I will observe for a while.

tamalsaha mentioned this issue Apr 12, 2019

Pin github.com/soheilhy/cmux dependency #160

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A question about 'Out of Memory' #159

A question about 'Out of Memory' #159

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

tamalsaha commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited by tamalsaha

Loading

yangyongzhi7 commented Apr 15, 2019 •

edited

Loading

A question about 'Out of Memory' #159

A question about 'Out of Memory' #159

Comments

yangyongzhi7 commented Apr 12, 2019 • edited Loading

yangyongzhi7 commented Apr 12, 2019 • edited Loading

tamalsaha commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019

yangyongzhi7 commented Apr 12, 2019 • edited Loading

yangyongzhi7 commented Apr 12, 2019 • edited Loading

yangyongzhi7 commented Apr 12, 2019 • edited by tamalsaha Loading

yangyongzhi7 commented Apr 15, 2019 • edited Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited

Loading

yangyongzhi7 commented Apr 12, 2019 •

edited by tamalsaha

Loading

yangyongzhi7 commented Apr 15, 2019 •

edited

Loading