「Harbor」- HTTP status 504 Gateway Time-out

  CREATED BY JENKINSBOT

# 07/20/2019 # 我刚想处理这个问题,好了个屁的。这种最可怕了。

问题背景

使用docker push命令推送镜像。

在某个镜像层推送将要结束的时候,收到如下错误信息:

# docker push docker-registry.example.com/project/image-foo:v2.5.0
The push refers to repository [docker-registry.example.com/project/image-foo:v2.5.0]
3d10bce3ce31: Layer already exists
1b70b131f01d: Layer already exists
2c5ef295a217: Pushing [==================================================>]  375.4MB
2ce4f2676a16: Layer already exists
35bbe47ffc77: Layer already exists
05b61dc1031f: Layer already exists
d6d37cde236f: Layer already exists
ecbc53aebc27: Layer already exists
1585039add0a: Layer already exists
692d855fb28e: Layer already exists
717b092b8c86: Layer already exists
received unexpected HTTP status: 504 Gateway Time-out

查看/var/log/registry.log文件,日志中有如下信息(经过了简单的处理):

# Jul 22 18:26:55 172.18.0.1 proxy[1114]: 2019/07/22 10:26:55 [error] 8#0: *11392 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.50.63, server: , request: "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1", upstream: "http://172.18.0.6:8080/registryproxy/v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9", host: "docker-registry.cluster.dfwsgroup.cn"
# Jul 22 18:26:55 172.18.0.1 proxy[1114]: 192.168.50.63 - "PATCH /v2/veryeast/ve-m-ssr/blobs/uploads/a8008eea-5412-40d2-a566-4efc71b72cb3?_state=gvZ4sODnz4enCabFx1TGEpNdkLaLAjkitundRI3az197Ik5hbWUiOiJ2ZXJ5ZWFzdC92ZS1tLXNzciIsIlVVSUQiOiJhODAwOGVlYS01NDEyLTQwZDItYTU2Ni00ZWZjNzFiNzJjYjMiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTktMDctMjJUMTA6MjU6MTkuMzU4NzkwNzk1WiJ9 HTTP/1.1" 504 176 "-" "docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.18.12-1.el7.elrepo.x86_64 os/linux arch/amd64 UpstreamClient(Docker-Client/18.06.1-ce \x5C(linux\x5C))" 95.187 95.186 .

环境信息

操作系统:CentOS Linux release 7.4.1708 (Core) 这种问题和操作的关系不大。

软件版本:Harbor v1.4.0

所需资料

仓库地址:https://github.com/goharbor/harbor/tree/v1.4.0
安装配置:Installation & Configuration Guide
用户手册:User Guide
软件架构:Architecture Overview of Harbor

问题排查

暂缓问题

replication for big images failed with Gateway Time-out intermittently #3446

修改common/templates/nginx/nginx.http(s).conf文件中的location /v2/部分,添加proxy_send_timeout 900;proxy_read_timeout 900;配置行。因为从Nginx的日志看是访问/v2/时产生的超时。

可能原因

系统磁盘IO存在问题:

#1 在进行镜像传输时候,系统的「平均负载」比较高,四核,处于3.5以上。而iostat命令显示系统IO并不是很高。

因此推测是IO较低,导致镜像写入磁盘的时间比较长,由于超过了Nginx的等待时间,而返回了504错误。

另外,系统中并没有相关Harbor错误日志。在时间点附近,大部分日志都是正常的。

参考文献

docker registry keep showing manifest unknown error for image pull after somedays of push image #17031
Delete tag failed #6515