腾讯云harbor私有仓库部署实践

Harbor是由VMware公司开源的企业级的Docker Registry管理项目,相比docker官方拥有更丰富的权限权利和完善的架构设计,适用大规模docker集群部署提供仓库服务,并且提供UI界面。
一般harbor可以通过helm或者docker-compose安装,本文以compose安装为例,介绍harbor如何配置腾讯云对象存储COS作为私有镜像仓库存储地址。

一 部署过程

1 准备:

安装docker-compose

curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

2 重启docker

systemctl daemon-reload

3 下载habor装包解压 harbor.v2.1.0.tar.gz

3.1 配置项注意:

  • hostname 修改主机地址,即访问域名
  • https 增加HTTPS证书配置,注意如果使用了CLB,需要在CLB同时配置证书
  • storage_service 中配置COS信息,注意harbor支持aws S3,可以在s3中配置COS桶信息,需要在对象存储提前建好对应桶信息
  • 注意harbor日志路径:默认 /var/log/harbor/registryctl.log
# harbor.v2.1.0.tar.gz
vi harbor.yml
-----------harbor.yml---------
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.yourset.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harborxxx

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 1024 for postgres of harbor.
  max_open_conns: 1000

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
storage_service:
  s3:
     accesskey: xxxxxx
     secretkey: xxxxxxx
     regionendpoint: cos.ap-guangzhou.myqcloud.com
     region: ap-guangzhou
     bucket: harborgz-xxxxx
     secure: true
   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.

#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

###....

4 启动,进入安装目录

$ docker-compose down -v

# 修改配置后,需要重新检查运行环境
$ ./prepare

# 启动harbor服务
$ docker-compose up -d

5 测试

5.1 网页测试,如果需要https访问,需要申请证书并配置在CLB及harbor服务器上

配置在harbor.yml的https中:

-----------------
  https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/key/harbor.yourset.com.crt
  private_key: /data/key/harbor.yourset.xyz.key

5.2 本地测试,输入密码

5.3 测试推镜像

注意事项:需要在harbor中先建立项目,否则会推送失败

[root@centos ~/tmp]# docker tag hello-world 127.0.0.1/s3/hello-world:v1.0.0 
[root@centos ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Preparing 
unauthorized: project not found, name: s3: project not found, name: s3
[root@centos ~/tmp]# docker push 127.0.0.1/s3/hello-world:v1.0.0
The push refers to repository [127.0.0.1/s3/hello-world]
f22b99068db9: Pushed 
v1.0.0: digest: sha256:1b26826f602946860c279fce65829b57792 size: 525

5.4 对象存储中也生成了相关文件:

二 踩坑记录

2.1 坑一:S3配置踩坑

网上搜的文章及其他用户反馈S3部分的配置如下:

s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      endpoint: cos.ap-singapore.myqcloud.com
      secure: true

使用该配置后,启动harbor后总会有harbor-registryclt等容器不断重启,造成无法推拉镜像:

查看错误日志:

tail -f /var/log/harbor/registryctl.log 

Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [ERROR] [/registryctl/config/config.go:63]: failed to load storage driver, err:No region parameter provided
Aug  3 15:32:31 172.30.0.1 registryctl[28778]: 2021-08-03T07:32:31Z [FATAL] [/registryctl/main.go:78]: Failed to load configurations with error: No region parameter provided

关键信息:

询问几个同事都没有结论,在google查了半天也没找到相关的文档,只好决定从源码入手,先去查看registryctl/main.go源码:

继续查看config.go:63

https://github.com/goharbor/harbor/blob/9e117539492b9e54658b8c4dd240af231c351cb5/src/registryctl/config/config.go

查看setStorageDriver()

找出storagedriver中s3的相关代码

查看s3部分:

https://github.com/distribution/distribution/blob/01f589cf8726565aa3c5c053be12873bafedbedc/registry/storage/driver/s3-aws/s3.go

发现一段特别的地方:

当"regionendpoint"为空时,程序会去aws的官方的 validRegins列表中查询可用区,而本次是要配置腾讯云COS地址,当然在aws的region列表里面没有,所以会提示 err:No region parameter provided。

因此需要传入"regionendpoint"的key才可避免查询aws自己的region list(网上的文章误导人啊),而不是传入"endpoint"

,所以需要在harbor.yml中把配置改为:

s3:
      region: ap-xxx
      bucket: xx-sigp-xxxxxxx
      accesskey: xxxxxxx
      secretkey: xxxxx
      regionendpoint: cos.ap-singapore.myqcloud.com
      secure: true

修改后重载harbor启动成功,推拉镜像正常。

2.2 坑二:COS 强一致配置踩坑

有用户根据上面部署后,发现新的报错,现象是通过docker push文件成功,但是harbor总返回500报错:

经过与COS团队沟通,主要是list强一致问题,即put 文件后直接list不一定能list到文件,因为list是最终一致性的,需要COS运维同事下发强一致的配置后,会保障能list出来。

发配置需提供客户账号appid、可用区、桶名称信息(可向腾讯云提工单),发布配置后该报错可解决。

三 总结

1 部署过程及时记录自己操作过程,关注日志

2 如果网上没有现成的答案,请教身边的专家

3 实在解决不了,去查源码,所有的逻辑都已经写在代码里了

本站文章资源均来源自网络,除非特别声明,否则均不代表站方观点,并仅供查阅,不作为任何参考依据!
如有侵权请及时跟我们联系,本站将及时删除!
如遇版权问题,请查看 本站版权声明
THE END
分享
二维码
海报
腾讯云harbor私有仓库部署实践
5.1 网页测试,如果需要https访问,需要申请证书并配置在CLB及harbor服务器上
<<上一篇
下一篇>>