AKS 上的 Kubernetes NiFi 集群设置

Question

想到 post 这是因为它可能对某人有所帮助。我找不到没有 helm 包的 Kubernetes NiFi 设置，所以我为 Kubernetes NiFi 集群设置准备了以下配置 YAML。

这是 link

Answer 1

如果您在配置中的任何地方发现任何问题或者您想提供任何建议，请发表评论。根据您的使用情况增加磁盘存储配置。

apiVersion: v1
kind: Service
metadata:
  name: nifi-hs
  labels:
    app: nifi
spec:
  ports:
  - port: 1025
    name: nodeport
  - port: 8080
    name: client
  clusterIP: None
  selector:
    app: nifi
---
apiVersion: v1
kind: Service
metadata:
  name: nifi-cs
  labels:
    app: nifi
  annotations:
      service.beta.kubernetes.io/azure-dns-label-name: nifi
spec:
  ports:
  - port: 80
    targetPort: 8080
    name: client
  selector:
    app: nifi
  type: LoadBalancer
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: nifi-pdb
spec:
  selector:
    matchLabels:
      app: nifi
  maxUnavailable: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nifi-sc
  selfLink: /apis/storage.k8s.io/v1/storageclasses/nifi-sc
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: 'true'
provisioner: kubernetes.io/azure-disk
parameters:
  cachingmode: ReadOnly
  kind: Managed
  storageaccounttype: StandardSSD_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
spec:
  selector:
    matchLabels:
      app: nifi
  serviceName: nifi-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: nifi
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - k: "app"
                    operator: In
                    values:
                    - nifi
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nifi        
        image: "apache/nifi:1.13.0"
        env:           
        - name: NIFI_CLUSTER_IS_NODE
          value: "true"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NIFI_CLUSTER_ADDRESS
          value: $(HOSTNAME).nifi-hs
        - name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
          value: "1025"
        - name: NIFI_WEB_HTTP_HOST
          value: $(HOSTNAME).nifi-hs.ns1.svc.cluster.local
        #- name: NIFI_WEB_HTTP_PORT
        #  value: "80"
        - name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
          value: "100" 
        - name: NIFI_ZK_CONNECT_STRING
          value: "zk-cs:2181" 
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "3" 
        ports:
        - containerPort: 8080
          name: client
        - containerPort: 1025
          name: nodeport
        volumeMounts:        
        - name: nifi-database
          mountPath: "/opt/nifi/nifi-current/database_repository"
        - name: nifi-flowfile
          mountPath: "/opt/nifi/nifi-current/flowfile_repository"
        - name: nifi-content
          mountPath: "/opt/nifi/nifi-current/content_repository"
        - name: nifi-provenance
          mountPath: "/opt/nifi/nifi-current/provenance_repository"
        - name: nifi-state
          mountPath: "/opt/nifi/nifi-current/state"
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: nifi-database
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-flowfile
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-content
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-provenance
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-state
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

Answer 2

亲和块中的标签选择器缺少几个词。以下是 statefulset 的更新工作 yaml 块。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
spec:
  selector:
    matchLabels:
      app: nifi
  serviceName: nifi-hs
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: OrderedReady
  template:
    metadata:
      labels:
        app: nifi
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - nifi
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nifi        
        image: "apache/nifi:1.13.0"
        env:           
        - name: NIFI_CLUSTER_IS_NODE
          value: "true"
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: NIFI_CLUSTER_ADDRESS
          value: $(HOSTNAME).nifi-hs
        - name: NIFI_CLUSTER_NODE_PROTOCOL_PORT
          value: "1025"
        - name: NIFI_WEB_HTTP_HOST
          value: $(HOSTNAME).nifi-hs.ns1.svc.cluster.local
        #- name: NIFI_WEB_HTTP_PORT
        #  value: "80"
        - name: NIFI_CLUSTER_NODE_PROTOCOL_MAX_THREADS
          value: "100" 
        - name: NIFI_ZK_CONNECT_STRING
          value: "zk-cs:2181" 
        - name: NIFI_ELECTION_MAX_CANDIDATES
          value: "3" 
        ports:
        - containerPort: 8080
          name: client
        - containerPort: 1025
          name: nodeport
        volumeMounts:        
        - name: nifi-database
          mountPath: "/opt/nifi/nifi-current/database_repository"
        - name: nifi-flowfile
          mountPath: "/opt/nifi/nifi-current/flowfile_repository"
        - name: nifi-content
          mountPath: "/opt/nifi/nifi-current/content_repository"
        - name: nifi-provenance
          mountPath: "/opt/nifi/nifi-current/provenance_repository"
        - name: nifi-state
          mountPath: "/opt/nifi/nifi-current/state"
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: nifi-database
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-flowfile
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-content
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-provenance
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi
  - metadata:
      name: nifi-state
    spec:
      storageClassName: "nifi-sc"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 5Gi

AKS 上的 Kubernetes NiFi 集群设置

Kubernetes NiFi Cluster setup in AKS

kubernetes

apache-nifi

azure-aks