问题描述
在执行 docker ps 命令时（运行容器、其他命令），产生如下错误：
# docker ps
runtime/cgo: pthread_create failed: No space left on device
SIGABRT: abort
PC=0x7f1a69cace97 m=0 sigcode=18446744073709551610
goroutine 0 [idle]:
runtime: unknown pc 0x7f1a69cace97
stack: frame={sp:0x7ffd418bd5d0, fp:0x0} stack=[0x7ffd410bec10,0x7ffd418bdc50)
00007ffd418bd4d0: 0000000000000000 0000000000000000
...
00007ffd418bd540: 000000006e43a318 00005601a65818ac
00007ffd418bd550: 0000000000000000 00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd560: 00007ffd418bd568 0000000000000000
00007ffd418bd570: 0000000000000000 00007ffd418bd5b8
00007ffd418bd580: 00005601a50855ba <runtime.mmap.func1+90> 0000000000000000
00007ffd418bd590: 0000000000210800 0000002200000003
00007ffd418bd5a0: 00000000ffffffff 00007f1a67a5d000
00007ffd418bd5b0: 00007ffd418bd5f8 00007ffd418bd630
00007ffd418bd5c0: 00005601a5032113 <runtime.mmap+179> 00007ffd418bd600
00007ffd418bd5d0: <0000000000000000 00007ffd418bd628
...
00007ffd418bd600: 000000006e43a318 00005601a65818ac
00007ffd418bd610: 0000000000210800 00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd620: 00007ffd418bd628 00007ffd418bd5f8
00007ffd418bd630: 00007ffd418bd680 00007ffd418bd678
00007ffd418bd640: 00005601a50855ba <runtime.mmap.func1+90> 0000000000000000
00007ffd418bd650: fffffffe7fffffff ffffffffffffffff
...
00007ffd418bd6c0: ffffffffffffffff ffffffffffffffff
runtime: unknown pc 0x7f1a69cace97
stack: frame={sp:0x7ffd418bd5d0, fp:0x0} stack=[0x7ffd410bec10,0x7ffd418bdc50)
00007ffd418bd4d0: 0000000000000000 0000000000000000
...
00007ffd418bd540: 000000006e43a318 00005601a65818ac
00007ffd418bd550: 0000000000000000 00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd560: 00007ffd418bd568 0000000000000000
00007ffd418bd570: 0000000000000000 00007ffd418bd5b8
00007ffd418bd580: 00005601a50855ba <runtime.mmap.func1+90> 0000000000000000
00007ffd418bd590: 0000000000210800 0000002200000003
00007ffd418bd5a0: 00000000ffffffff 00007f1a67a5d000
00007ffd418bd5b0: 00007ffd418bd5f8 00007ffd418bd630
00007ffd418bd5c0: 00005601a5032113 <runtime.mmap+179> 00007ffd418bd600
00007ffd418bd5d0: <0000000000000000 00007ffd418bd628
...
00007ffd418bd600: 000000006e43a318 00005601a65818ac
00007ffd418bd610: 0000000000210800 00005601a508d93e <runtime.callCgoMmap+62>
00007ffd418bd620: 00007ffd418bd628 00007ffd418bd5f8
00007ffd418bd630: 00007ffd418bd680 00007ffd418bd678
00007ffd418bd640: 00005601a50855ba <runtime.mmap.func1+90> 0000000000000000
00007ffd418bd650: fffffffe7fffffff ffffffffffffffff
...
00007ffd418bd690: ffffffffffffffff ffffffffffffffff
goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
/usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc0000986c8 sp=0xc0000986c0 pc=0x5601a50899b0
runtime.newproc(0x560100000000, 0x5601a7220820)
/usr/local/go/src/runtime/proc.go:3255 +0x70 fp=0xc000098710 sp=0xc0000986c8 pc=0x5601a5067500
runtime.init.5()
/usr/local/go/src/runtime/proc.go:242 +0x37 fp=0xc000098730 sp=0xc000098710 pc=0x5601a505faa7
runtime.doInit(0x5601a8134ac0)
/usr/local/go/src/runtime/proc.go:5222 +0x8c fp=0xc000098760 sp=0xc000098730 pc=0x5601a506beac
runtime.main()
/usr/local/go/src/runtime/proc.go:150 +0xcb fp=0xc0000987e0 sp=0xc000098760 pc=0x5601a505f78b
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0xc0000987e8 sp=0xc0000987e0 pc=0x5601a508bab1
rax 0x0
rbx 0x7f1a6a05a840
rcx 0x7f1a69cace97
rdx 0x0
rdi 0x2
rsi 0x7ffd418bd5d0
rbp 0x5601a686fb54
rsp 0x7ffd418bd5d0
r8 0x0
r9 0x7ffd418bd5d0
r10 0x8
r11 0x246
r12 0x5601a8e5c3f0
r13 0x0
r14 0x5601a68173ac
r15 0x0
rip 0x7f1a69cace97
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
问题原因
Kernel not freeing memory cgroup causing no space left on device · Issue #29638 · moby/moby
解决方案
我们未尝试过以下解决方案，这里仅根据 GitHub Issue 记录：
1）升级到 4.18 内核（This never happened again after I upgrade Linux to 4.18）
2）定期清理缓存：echo 3 > /proc/sys/vm/drop_caches（参考 Kernel not freeing memory cgroup causing no space left on device 解释）
参考文献
Kernel not freeing memory cgroup causing no space left on device · Issue #29638 · moby/moby