Operator3-设计一个operator

对你无可奈何 • 2023-01-02 • 云技术社区 • 282 阅读

背景：

前置知识Operator-1初识Operator，Operator-2从pod开始简单operator。

先拿一个个人的工作环境来设计吧，应用有十多个微服务，恩各种类型job deployment statefulset service ingress pv pvc configmap这些资源准备模仿eck:https://github.com/elastic/cloud-on-k8s来设计！

image.png

创建一个自己的operator

goland 创建项目

image.png

命名规则

关于应用命名一下吧：

就拿月份来作应用名称吧：

应用1：Jan

应用2：feb

应用3：mar

应用4：apr

应用5：may

也没有想好具体的代表什么，下面了边作边看......

kubebuilder init

[zhangpeng@zhangpeng develop-operator]$ kubebuilder init --plugins go/v3 --domain zhangpeng.com --owner "zhang peng"

image.png

开启支持多接口组

模仿目录结构：

image.png

设置multigroup=true，忘了这是在哪个地方搜到的了，应该是思否一篇文章

[zhangpeng@zhangpeng develop-operator]$ kubebuilder edit --multigroup=true

[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group jan --version v1 --kind Jan

image.png

[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group feb --version v1 --kind Feb

image.png

[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group mar --version v1 --kind Mar
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group apr --version v1 --kind Apr
[zhangpeng@zhangpeng develop-operator]$ kubebuilder create api --group may --version v1 --kind May

image.png

从Jan开始

jan应用为一个deployment应用，参照https://www.qikqiak.com/post/k8s-operator-101/,创建一个deployment并与Operator-2从pod开始简单operator中对比一下pod 与deployment的区别！

注意：以下代码都是抄写自阳明大佬，些许修改......，比如有个& 还有关于service的修改

定义jan_type

apis/jan/v1/jan_type.go

type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	Size      *int32                      `json:"size"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
}

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	appsv1.DeploymentStatus `json:",inline"`
}

image.png

make install

make install 失败，继续拆解命令：

[zhangpeng@zhangpeng develop-operator]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases

继续发布到集群：

[zhangpeng@zhangpeng develop-operator]$ kustomize build config/crd | kubectl apply -f -

不知道有没有单独发布的办法，一整都一起发布了......

image.png

[zhangpeng@zhangpeng develop-operator]$ kubectl describe crd jans.jan.zhangpeng.com

image.png

describe crd的内容与config/crd/bases/jan.zhangpeng.com_jans.yaml 内容是一样的，强调一下......

创建deployment pod service的方法

对应文件都偷懒了，直接放在controllers/jan目录下了：

image.png

cat jan_helper.go

package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Size,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
	}

}
func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}
func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  corev1.ServiceTypeNodePort,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

jan_controller.go Reconcile

基本阳明大佬的博客抄来的,Reconcile调谐函数：

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"reflect"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	"k8s.io/apimachinery/pkg/runtime"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=jan.zhangpeng.com,resources=jans/finalizers,verbs=update

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}

	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}
	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}
		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)
		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}

		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}

	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}

	if !reflect.DeepEqual(instance.Spec, oldspec) {
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}

		return reconcile.Result{}, nil
	}

	return reconcile.Result{}, nil

}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

强调一下：

json.Unmarshal

image.png

https://www.qikqiak.com/post/k8s-operator-101/#%E9%A1%B9%E7%9B%AE%E7%BB%93%E6%9E%84是这样写的，but maker run就报错了，看了一下别人有人写了&就加了一下

image.png

make run and test

image.png

测试yaml就用config/samples/jan_v1_jan.yaml去测试了：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  size: 2
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30002

[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created

image.png

修改副本数为3：

image.png

继续改造

修改Service Type

en 对比前一节的pod operator我想输出更多的内容，get Jan也想输出数量：继续改造(还有Jan服务可以输入Type我不喜欢nodePort的方式)

image.png

[zhangpeng@zhangpeng develop-operator]$ make install
[zhangpeng@zhangpeng develop-operator]$ ./bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
[zhangpeng@zhangpeng develop-operator]$  kustomize build config/crd | kubectl apply -f -

image.png

kubecelt delete -f config/samples/jan_v1_jan.yaml

重新编辑文件如下：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  size: 3
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
//     nodePort: 30002
    type: ClusterIP

controllers/jan/jan_helper.go NewService修改Type如下：Type: app.Spec.Type,

image.png

make run and kubectl apply

[zhangpeng@zhangpeng develop-operator]$ kubectl delete -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com "jan-sample" deleted
[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created
[zhangpeng@zhangpeng develop-operator]$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
jan-sample   ClusterIP   10.99.27.123   <none>        80/TCP    1s
kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP   19d
[zhangpeng@zhangpeng develop-operator]$ kubectl get jan
NAME         AGE
jan-sample   5m21s

image.png

继续模仿

eck 有更多的输出阿咱们的输出现在只有AGE......想输出更多

image.png

注意：comon我还是没有用到，创建了就创建了吧，后面看看还是否用的到！

人家eck有个公用的comon?咱也搞一个

image.png

kubebuilder create api --group common --version v1 --kind Common

image.png

继续狗一下

image.png

这里懒得写了网上搜到一个博客：https://qingwave.github.io/how-to-write-a-k8s-operator就按照他写的改一下了：

apis/jan/v1/jan_type.go

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// JanSpec defines the desired state of Jan
type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	//+kubebuilder:default:=1
	//+kubebuilder:validation:Minimum:=1
	Replicas  *int32                      `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
	Type      corev1.ServiceType          `json:"type,omitempty"`
}

const (
	Running  = "Running"
	Pending  = "Pending"
	NotReady = "NotReady"
	Failed   = "Failed"
)

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	// Phase is the phase of guestbook
	Phase string `json:"phase,omitempty"`
	// replicas is the number of Pods created by the StatefulSet controller.
	Replicas int32 `json:"replicas"`

	// readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.
	ReadyReplicas int32 `json:"readyReplicas"`

	// LabelSelector is label selectors for query over pods that should match the replica count used by HPA.
	LabelSelector string `json:"labelSelector,omitempty"`
}

//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.labelSelector
//+kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase",description="The phase of game."
//+kubebuilder:printcolumn:name="DESIRED",type="integer",JSONPath=".spec.replicas",description="The desired number of pods."
//+kubebuilder:printcolumn:name="CURRENT",type="integer",JSONPath=".status.replicas",description="The number of currently all pods."
//+kubebuilder:printcolumn:name="READY",type="integer",JSONPath=".status.readyReplicas",description="The number of pods ready."
//+kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp",description="CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC."
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Jan is the Schema for the jans API
type Jan struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   JanSpec   `json:"spec,omitempty"`
	Status JanStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// JanList contains a list of Jan
type JanList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Jan `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Jan{}, &JanList{})
}

make install

image.png

controllers/jan/jan_helper.go

其实我就修改了一下type，我可不想一直写nodeport......一般都司clasterip

package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Replicas,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
		Status: appv1.DeploymentStatus{},
	}

}

func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}

func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  app.Spec.Type,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

jan_controller.go

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"reflect"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps,resources=deployments/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=networking,resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	defer utilruntime.HandleCrash()
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}
	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}

	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}

		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}
	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}
	if !reflect.DeepEqual(instance.Spec, oldspec) {
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}
		return reconcile.Result{}, nil
	}
	newStatus := janv1.JanStatus{
		Replicas:      *instance.Spec.Replicas,
		ReadyReplicas: instance.Status.Replicas,
	}

	if newStatus.Replicas == newStatus.ReadyReplicas {
		newStatus.Phase = janv1.Running
	} else {
		newStatus.Phase = janv1.NotReady
	}
	if !reflect.DeepEqual(instance.Status, newStatus) {
		instance.Status = newStatus
		log.FromContext(ctx).Info("update game status", "name", instance.Name)
		err := r.Client.Status().Update(ctx, instance)
		return reconcile.Result{}, err

	}
	return reconcile.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

偷懒抄来的：

image.png

make run :

清空原来的Jan应用

[zhangpeng@zhangpeng develop-operator]$ kubectl delete jan jan-sample
jan.jan.zhangpeng.com "jan-sample" deleted

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 2
  image: nginx:1.7.9
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png

[zhangpeng@zhangpeng develop-operator]$ kubectl apply -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com/jan-sample created
[zhangpeng@zhangpeng develop-operator]$ kubectl get jan
NAME         PHASE     DESIRED   CURRENT   READY   AGE
jan-sample   Running   2         2         2       1s

image.png

问题又来了：

image.png

这样还是不能更新状态status阿？没有什么实际意义。所以这个哥们写的也是一个纯看的demo....还是找个成熟的应用去看吧！

image.png

偷懒秘籍：

更新资源这里我是不是可以重新生成一下Annotations？那旧的数据不就变成新的了？

image.png

尝试一下：

image.png

自己骗自己算是成功了

image.png

但是还是有问题：

修改image

image.png

pod更新中这样是否能接受呢？

image.png

看了一眼deployment也是这样，我接受了.......

最终代码

上面感觉还是缺少点东西，什么呢？ingress我是否也可以封过来？

增加ingress相关字段

apis/jan/v1/jan_type.go

image.png

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1

import (
	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EDIT THIS FILE!  THIS IS SCAFFOLDING FOR YOU TO OWN!
// NOTE: json tags are required.  Any new fields you add must have json tags for the fields to be serialized.

// JanSpec defines the desired state of Jan
type JanSpec struct {
	// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	//+kubebuilder:default:=1
	//+kubebuilder:validation:Minimum:=1
	Replicas  *int32                      `json:"replicas,omitempty" protobuf:"varint,1,opt,name=replicas"`
	Image     string                      `json:"image"`
	Resources corev1.ResourceRequirements `json:"resources,omitempty"`
	Envs      []corev1.EnvVar             `json:"envs,omitempty"`
	Ports     []corev1.ServicePort        `json:"ports,omitempty"`
	Type      corev1.ServiceType          `json:"type,omitempty"`
	Host      string                      `json:"host,omitempty"`
}

const (
	Running  = "Running"
	Pending  = "Pending"
	NotReady = "NotReady"
	Failed   = "Failed"
)

// JanStatus defines the observed state of Jan
type JanStatus struct {
	// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
	// Important: Run "make" to regenerate code after modifying this file
	// Phase is the phase of guestbook
	Phase string `json:"phase,omitempty"`
	// replicas is the number of Pods created by the StatefulSet controller.
	Replicas int32 `json:"replicas"`

	// readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.
	ReadyReplicas int32 `json:"readyReplicas"`

	// LabelSelector is label selectors for query over pods that should match the replica count used by HPA.
	LabelSelector string `json:"labelSelector,omitempty"`
}

//+kubebuilder:printcolumn:name="Host",type="string",JSONPath=".spec.host",description="The host address."
//+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.labelSelector
//+kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase",description="The phase of game."
//+kubebuilder:printcolumn:name="DESIRED",type="integer",JSONPath=".spec.replicas",description="The desired number of pods."
//+kubebuilder:printcolumn:name="CURRENT",type="integer",JSONPath=".status.replicas",description="The number of currently all pods."
//+kubebuilder:printcolumn:name="READY",type="integer",JSONPath=".status.readyReplicas",description="The number of pods ready."
//+kubebuilder:printcolumn:name="AGE",type="date",JSONPath=".metadata.creationTimestamp",description="CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC."
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// Jan is the Schema for the jans API
type Jan struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   JanSpec   `json:"spec,omitempty"`
	Status JanStatus `json:"status,omitempty"`
}

//+kubebuilder:object:root=true

// JanList contains a list of Jan
type JanList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Jan `json:"items"`
}

func init() {
	SchemeBuilder.Register(&Jan{}, &JanList{})
}

创建NewIngress方法

jan_helper.go 中，先写死这个了port了

image.png

package jan

import (
	janv1 "develop-operator/apis/jan/v1"
	appv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	v1 "k8s.io/api/networking/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

func NewJan(app *janv1.Jan) *appv1.Deployment {
	labels := map[string]string{"app": app.Name}
	selector := &metav1.LabelSelector{MatchLabels: labels}
	return &appv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			Kind:       "apps/v1",
			APIVersion: "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: appv1.DeploymentSpec{
			Replicas: app.Spec.Replicas,
			Selector: selector,
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{Labels: labels},
				Spec:       corev1.PodSpec{Containers: newContainers(app)},
			},
			MinReadySeconds: 0,
		},
		Status: appv1.DeploymentStatus{},
	}

}

func newContainers(app *janv1.Jan) []corev1.Container {
	containerPorts := []corev1.ContainerPort{}
	for _, svcPort := range app.Spec.Ports {
		cport := corev1.ContainerPort{}
		cport.ContainerPort = svcPort.TargetPort.IntVal
		containerPorts = append(containerPorts, cport)
	}
	return []corev1.Container{
		{
			Name:            app.Name,
			Image:           app.Spec.Image,
			Resources:       app.Spec.Resources,
			Ports:           containerPorts,
			ImagePullPolicy: corev1.PullIfNotPresent,
			Env:             app.Spec.Envs,
		},
	}
}

func NewService(app *janv1.Jan) *corev1.Service {
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Service",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
			OwnerReferences: []metav1.OwnerReference{
				*metav1.NewControllerRef(app, schema.GroupVersionKind{
					Group:   janv1.GroupVersion.Group,
					Version: janv1.GroupVersion.Version,
					Kind:    "Jan",
				}),
			},
		},
		Spec: corev1.ServiceSpec{
			Type:  app.Spec.Type,
			Ports: app.Spec.Ports,
			Selector: map[string]string{
				"app": app.Name,
			},
		},
	}
}

const (
	port = 80
)

func NewIngress(app *janv1.Jan) *v1.Ingress {
	pathType := v1.PathTypePrefix
	return &v1.Ingress{
		TypeMeta: metav1.TypeMeta{
			Kind:       "Ingress",
			APIVersion: "v1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      app.Name,
			Namespace: app.Namespace,
		},
		Spec: v1.IngressSpec{
			IngressClassName: nil,
			Rules: []v1.IngressRule{
				{
					Host: app.Spec.Host,
					IngressRuleValue: v1.IngressRuleValue{
						HTTP: &v1.HTTPIngressRuleValue{
							Paths: []v1.HTTPIngressPath{{
								Path:     "/",
								PathType: &pathType,
								Backend: v1.IngressBackend{
									Service: &v1.IngressServiceBackend{
										Name: app.Name,
										Port: v1.ServiceBackendPort{
											Number: int32(port),
										},
									},
									Resource: nil,
								},
							},
							}}},
				},
			},
		},
	}
}

jan_controller.go中增加ingress相关

image.png

/*
Copyright 2022 zhang peng.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package jan

import (
	"context"
	"encoding/json"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	"reflect"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/reconcile"

	janv1 "develop-operator/apis/jan/v1"
)

// JanReconciler reconciles a Jan object
type JanReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=mar.zhangpeng.com,resources=jan/finalizers,verbs=update
//+kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=apps,resources=deployments/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=networking,resources=ingresses,verbs=get;list;watch;create;update;patch;delete

// Reconcile is part of the main kubernetes reconciliation loop which aims to
// move the current state of the cluster closer to the desired state.
// TODO(user): Modify the Reconcile function to compare the state specified by
// the Jan object against the actual cluster state, and then
// perform operations to make the cluster state reflect the state specified by
// the user.
//
// For more details, check Reconcile and its Result here:
// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.11.2/pkg/reconcile
func (r *JanReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	defer utilruntime.HandleCrash()
	_ = log.FromContext(ctx)
	instance := &janv1.Jan{}
	err := r.Client.Get(context.TODO(), req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			return reconcile.Result{}, nil
		}
		// Error reading the object - requeue the request.
		return reconcile.Result{}, err
	}
	if instance.DeletionTimestamp != nil {
		return reconcile.Result{}, err
	}

	// 如果不存在，则创建关联资源
	// 如果存在，判断是否需要更新
	//   如果需要更新，则直接更新
	//   如果不需要更新，则正常返回

	deploy := &appsv1.Deployment{}

	if err := r.Client.Get(context.TODO(), req.NamespacedName, deploy); err != nil && errors.IsNotFound(err) {
		// 创建关联资源
		// 1. 创建 Deploy
		deploy := NewJan(instance)
		if err := r.Client.Create(context.TODO(), deploy); err != nil {
			return reconcile.Result{}, err
		}

		// 2. 创建 Service
		service := NewService(instance)
		if err := r.Client.Create(context.TODO(), service); err != nil {
			return reconcile.Result{}, err
		}
		// 3. 创建 Ingress
		ingress := NewIngress(instance)
		if err := r.Client.Create(context.TODO(), ingress); err != nil {
			return reconcile.Result{}, err
		}
		// 4. 关联 Annotations
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		return reconcile.Result{}, nil
	}
	oldspec := janv1.JanSpec{}
	if err := json.Unmarshal([]byte(instance.Annotations["spec"]), &oldspec); err != nil {
		return reconcile.Result{}, err
	}

	if !reflect.DeepEqual(instance.Spec, oldspec) {
		data, _ := json.Marshal(instance.Spec)

		if instance.Annotations != nil {
			instance.Annotations["spec"] = string(data)
		} else {
			instance.Annotations = map[string]string{"spec": string(data)}
		}
		if err := r.Client.Update(context.TODO(), instance); err != nil {
			return reconcile.Result{}, nil
		}
		// 更新关联资源
		newDeploy := NewJan(instance)
		oldDeploy := &appsv1.Deployment{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldDeploy); err != nil {
			return reconcile.Result{}, err
		}
		oldDeploy.Spec = newDeploy.Spec
		if err := r.Client.Update(context.TODO(), oldDeploy); err != nil {
			return reconcile.Result{}, err
		}

		newService := NewService(instance)
		oldService := &corev1.Service{}
		if err := r.Client.Get(context.TODO(), req.NamespacedName, oldService); err != nil {
			return reconcile.Result{}, err
		}
		oldService.Spec = newService.Spec
		if err := r.Client.Update(context.TODO(), oldService); err != nil {
			return reconcile.Result{}, err
		}
		return reconcile.Result{}, nil
	}
	newStatus := janv1.JanStatus{
		Replicas:      *instance.Spec.Replicas,
		ReadyReplicas: instance.Status.Replicas,
	}

	if newStatus.Replicas == newStatus.ReadyReplicas {
		newStatus.Phase = janv1.Running
	} else {
		newStatus.Phase = janv1.NotReady
	}
	if !reflect.DeepEqual(instance.Status, newStatus) {
		instance.Status = newStatus
		log.FromContext(ctx).Info("update game status", "name", instance.Name)
		err = r.Client.Status().Update(ctx, instance)
		if err != nil {
			return reconcile.Result{}, err
		}
	}
	return reconcile.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *JanReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&janv1.Jan{}).
		Complete(r)
}

实验一下：

make install make run

image.png

删除清空jan应用

[zhangpeng@zhangpeng develop-operator]$ kubectl delete -f config/samples/jan_v1_jan.yaml 
jan.jan.zhangpeng.com "jan-sample" deleted

修改config/samples/jan_v1_jan.yaml 如下：

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 3
  image: nginx:1.17.6
  host: www.zhangpeng.com
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png

接着修改config/samples/jan_v1_jan.yaml 副本数与host

apiVersion: jan.zhangpeng.com/v1
kind: Jan
metadata:
  name: jan-sample
spec:
  replicas: 2
  image: nginx:1.17.6
  host: www1.zhangpeng.com
  ports:
    - port: 80
      targetPort: 80
  type: ClusterIP

image.png

基本实现了第一步的需求了！