月別アーカイブ: 2021年6月

helmでインストールしたwordpressのphpスクリプトの場所を特定

/var/snap/microk8s/common/ 以下に置かれているようだ。

$ sudo find / -name wp-login.php 2>/dev/null
/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/179/fs/opt/bitnami/wordpress/wp-login.php
/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/170/fs/opt/bitnami/wordpress/wp-login.php
/var/snap/microk8s/common/run/containerd/io.containerd.runtime.v2.task/k8s.io/6f704f341ec11315ce29fc36f1a4d4c4e7c88b55d460fa0249c7e268b20be2ed/rootfs/opt/bitnami/wordpress/wp-login.php

k8sのサービスへ別マシンから接続

k8sのサービスへ別マシンから接続するには、ポートフォワーディングを設定します。例えばk8sのdashbordをアクセスする場合には、設定に必要な情報を次の手順で取得

$ microk8s kubectl get all --all-namespaces | grep dashboard | grep TCP
kube-system          service/kubernetes-dashboard        ClusterIP      10.152.183.19    <none>        443/TCP

必要な部分は太字の箇所

kube-system service/kubernetes-dashboard ClusterIP 10.152.183.19 443/TCP

外部からアクセスする場合のポートを10443とした場合のportforward設定の例;10443が既に使われていた場合はエラーとなるので、別のポートを指定する。

microk8s.kubectl port-forward --address 0.0.0.0 -n kube-system service/kubernetes-dashboard 10443:443 &

wordpressをアクセスする場合;外部マシンからhttp://xxx.xxx.xxx.xxx:20443/をアクセス

(xxx.xxx.xxx.xxxはk8sマシンのIPアドレス)

microk8s kubectl get all --all-namespaces | grep wordpress | grep TCP
helm-test            service/test-wordpress              LoadBalancer   10.152.183.226   <pending>     80:30034/TCP,443:30311/TC
microk8s.kubectl port-forward --address 0.0.0.0 -n helm-test service/test-wordpress 20443:443 &
Forwarding from 0.0.0.0:20443 -> 8443

helmでwordpressをデプロイ

これまでの失敗の一因:

(1)microk8s config >.kube/configの未実施。

(2)クリーンでない環境へインストールしようとした。(例えば、80/tcpが既に使われていた環境)

$ microk8s config >.kube/config  を実行しておかないと、次のようなエラーとなる。
The connection to the server localhost:8080 was refused - did you specify the right host or port?

#helmをmake
git clone https://github.com/helm/helm.git
cd helm
#ここでmakeすると go が欠落していてエラー snapでgoをインストール
sudo snap install go --classic
make
sudo make install

kubectl create namespace helm-test
namespace/helm-test created

mars@mars-VirtualBox:~$ helm install test bitnami/wordpress --namespace helm-test 
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/mars/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/mars/.kube/config
NAME: test
LAST DEPLOYED: Tue Jun 22 19:46:16 2021
NAMESPACE: helm-test
STATUS: deployed
REVISION: 1
NOTES:
** Please be patient while the chart is being deployed **
Your WordPress site can be accessed through the following DNS name from within your cluster:
    test-wordpress.helm-test.svc.cluster.local (port 80)
To access your WordPress site from outside the cluster follow the steps below:
1. Get the WordPress URL by running these commands:
  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace helm-test -w test-wordpress'
   export SERVICE_IP=$(kubectl get svc --namespace helm-test test-wordpress --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
   echo "WordPress URL: http://$SERVICE_IP/"
   echo "WordPress Admin URL: http://$SERVICE_IP/admin"

2. Open a browser and access WordPress using the obtained URL.
3. Login with the following credentials below to see your blog:
  echo Username: user
  echo Password: $(kubectl get secret --namespace helm-test test-wordpress -o jsonpath="{.data.wordpress-password}" | base64 --decode)

次のように、wordpressとmariadbが起動している。

helm list -n helm-test

WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/mars/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/mars/.kube/config
NAME	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART            	APP VERSION
test	helm-test	1       	2021-06-22 19:46:16.598835081 +0900 JST	deployed	wordpress-11.0.16	5.7.2      

$  kubectl get po -n helm-test
NAME                             READY   STATUS    RESTARTS   AGE
test-wordpress-8d8bb84b6-m5b5j   0/1     Running   0          87s
test-mariadb-0                   1/1     Running   0          87s

Adminでログインしたwordpressの画面

microk8sを試す(helm編)

WordPress Helm Chartのデプロイを参考にhelmをインストールしてみました。

Helm v3のすゝめ がより実践的?

helmのインストール方法が異なるが、その後の手順はほぼ同じ(以下、enabel helm3でインストールした場合)?

$ helm search hub prometheus のコマンドは;
$ microk8s.helm3 search hub prometheus のように読み替え

リポジトリを追加する

$ micro8ks.helm3 repo add stable https://charts.helm.sh/stable
$ microk8s.helm3 repo add bitnami https://charts.bitnami.com/bitnami

追加したレポジトリのリスト

$ microk8s helm3 repo list
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2265/credentials/client.config
NAME    URL
stable  https://charts.helm.sh/stable
bitnami https://charts.bitnami.com/bitnami

リポジトリ内のChartを検索する

$ microk8s helm3 search repo wordpress
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2265/credentials/client.config
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
bitnami/wordpress       11.0.16         5.7.2           Web publishing platform for building blogs and ...
stable/wordpress        9.0.3           5.3.2           DEPRECATED Web publishing platform for building...
  • helm search hubHelm HubのChartを検索できます。
  • helm install コマンドの--version引数にChartのバージョンを指定できますので、任意のバージョンのChartをデプロイすることも可能です。helm pullコマンドでChartをローカルにダウンロードできます。
    ダウンロードしたChartはお好きに書き換えてデプロイできるので、Chartに用意されているパラメタで変更できないような設定も変更できます。

アプリケーションをデプロイする

# namespaceを作成
$ kubectl create namespace helm-test
# dry-run
$ helm install test stable/prometheus --namespace helm-test --dry-run
# デプロイ
$ helm install stable/prometheus --name test --namespace helm-test
# 確認
$ helm list -n helm-test
$ kubectl get po -n helm-test

dry-runで表示された情報

$ microk8s helm3 install test bitnami/wordpress --namespace helm-test --dry-run
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2265/credentials/client.config
NAME: test
LAST DEPLOYED: Tue Jun 22 10:03:29 2021
NAMESPACE: helm-test
STATUS: pending-install
REVISION: 1
HOOKS:
---
# Source: wordpress/templates/tests/test-mariadb-connection.yaml
apiVersion: v1
kind: Pod
metadata:
  name: "test-credentials-test"
  annotations:
    "helm.sh/hook": test-success
spec:
  securityContext:
    fsGroup: 1001
  containers:
    - name: test-credentials-test
      image: docker.io/bitnami/wordpress:5.7.2-debian-10-r25
      imagePullPolicy: "IfNotPresent"
      securityContext:
        runAsNonRoot: true
        runAsUser: 1001
      env:
        - name: MARIADB_HOST
          value: "test-mariadb"
        - name: MARIADB_PORT
          value: "3306"
        - name: WORDPRESS_DATABASE_NAME
          value: "bitnami_wordpress"
        - name: WORDPRESS_DATABASE_USER
          value: "bn_wordpress"
        - name: WORDPRESS_DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: test-mariadb
              key: mariadb-password
      command:
        - /bin/bash
        - -ec
        - |
          mysql --host=$MARIADB_HOST --port=$MARIADB_PORT --user=$WORDPRESS_DATABASE_USER --password=$WORDPRESS_DATABASE_PASSWORD
  restartPolicy: Never
MANIFEST:
---
# Source: wordpress/charts/mariadb/templates/serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-mariadb
  namespace: helm-test
  labels:
    app.kubernetes.io/name: mariadb
    helm.sh/chart: mariadb-9.3.14
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
  annotations:
---
# Source: wordpress/charts/mariadb/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: test-mariadb
  namespace: helm-test
  labels:
    app.kubernetes.io/name: mariadb
    helm.sh/chart: mariadb-9.3.14
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  mariadb-root-password: "eGMyb0NNWXZVUg=="
  mariadb-password: "SFc1WlkwNWpsMw=="
---
# Source: wordpress/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: test-wordpress
  namespace: "helm-test"
  labels:
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-11.0.16
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  wordpress-password: "TVJteHlENlFtQQ=="
---
# Source: wordpress/charts/mariadb/templates/primary/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: test-mariadb
  namespace: helm-test
  labels:
    app.kubernetes.io/name: mariadb
    helm.sh/chart: mariadb-9.3.14
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary
data:
  my.cnf: |-
    [mysqld]
    skip-name-resolve
    explicit_defaults_for_timestamp
    basedir=/opt/bitnami/mariadb
    plugin_dir=/opt/bitnami/mariadb/plugin
    port=3306
    socket=/opt/bitnami/mariadb/tmp/mysql.sock
    tmpdir=/opt/bitnami/mariadb/tmp
    max_allowed_packet=16M
    bind-address=0.0.0.0
    pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
    log-error=/opt/bitnami/mariadb/logs/mysqld.log
    character-set-server=UTF8
    collation-server=utf8_general_ci

    [client]
    port=3306
    socket=/opt/bitnami/mariadb/tmp/mysql.sock
    default-character-set=UTF8
    plugin_dir=/opt/bitnami/mariadb/plugin

    [manager]
    port=3306
    socket=/opt/bitnami/mariadb/tmp/mysql.sock
    pid-file=/opt/bitnami/mariadb/tmp/mysqld.pid
---
# Source: wordpress/templates/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-wordpress
  namespace: "helm-test"
  labels:
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-11.0.16
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "10Gi"
---
# Source: wordpress/charts/mariadb/templates/primary/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: test-mariadb
  namespace: helm-test
  labels:
    app.kubernetes.io/name: mariadb
    helm.sh/chart: mariadb-9.3.14
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary
  annotations:
spec:
  type: ClusterIP
  ports:
    - name: mysql
      port: 3306
      protocol: TCP
      targetPort: mysql
      nodePort: null
  selector:
    app.kubernetes.io/name: mariadb
    app.kubernetes.io/instance: test
    app.kubernetes.io/component: primary
---
# Source: wordpress/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: test-wordpress
  namespace: "helm-test"
  labels:
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-11.0.16
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
spec:
  type: LoadBalancer
  externalTrafficPolicy: "Cluster"
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      protocol: TCP
      targetPort: https
  selector:
    app.kubernetes.io/name: wordpress
    app.kubernetes.io/instance: test
---
# Source: wordpress/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-wordpress
  namespace: "helm-test"
  labels:
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-11.0.16
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: wordpress
      app.kubernetes.io/instance: test
  strategy:
    rollingUpdate: {}
    type: RollingUpdate
  replicas: 1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: wordpress
        helm.sh/chart: wordpress-11.0.16
        app.kubernetes.io/instance: test
        app.kubernetes.io/managed-by: Helm
    spec:

      serviceAccountName: default
      # yamllint disable rule:indentation
      hostAliases:
        - hostnames:
          - status.localhost
          ip: 127.0.0.1
      # yamllint enable rule:indentation
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: wordpress
                    app.kubernetes.io/instance: test
                namespaces:
                  - "helm-test"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      securityContext:
        fsGroup: 1001
      containers:
        - name: wordpress
          image: docker.io/bitnami/wordpress:5.7.2-debian-10-r25
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          env:
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
            - name: MARIADB_HOST
              value: "test-mariadb"
            - name: MARIADB_PORT_NUMBER
              value: "3306"
            - name: WORDPRESS_DATABASE_NAME
              value: "bitnami_wordpress"
            - name: WORDPRESS_DATABASE_USER
              value: "bn_wordpress"
            - name: WORDPRESS_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: test-mariadb
                  key: mariadb-password
            - name: WORDPRESS_USERNAME
              value: "user"
            - name: WORDPRESS_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: test-wordpress
                  key: wordpress-password
            - name: WORDPRESS_EMAIL
              value: "user@example.com"
            - name: WORDPRESS_FIRST_NAME
              value: "FirstName"
            - name: WORDPRESS_LAST_NAME
              value: "LastName"
            - name: WORDPRESS_HTACCESS_OVERRIDE_NONE
              value: "no"
            - name: WORDPRESS_ENABLE_HTACCESS_PERSISTENCE
              value: "no"
            - name: WORDPRESS_BLOG_NAME
              value: "User's Blog!"
            - name: WORDPRESS_SKIP_BOOTSTRAP
              value: "no"
            - name: WORDPRESS_TABLE_PREFIX
              value: "wp_"
            - name: WORDPRESS_SCHEME
              value: "http"
            - name: WORDPRESS_EXTRA_WP_CONFIG_CONTENT
              value:
            - name: WORDPRESS_AUTO_UPDATE_LEVEL
              value: "none"
            - name: WORDPRESS_PLUGINS
              value: "none"
          envFrom:
          ports:
            - name: http
              containerPort: 8080
            - name: https
              containerPort: 8443
          livenessProbe:
            failureThreshold: 6
            httpGet:
              httpHeaders: []
              path: /wp-admin/install.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 120
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          readinessProbe:
            failureThreshold: 6
            httpGet:
              httpHeaders: []
              path: /wp-login.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            limits: {}
            requests:
              cpu: 300m
              memory: 512Mi
          volumeMounts:
            - mountPath: /bitnami/wordpress
              name: wordpress-data
              subPath: wordpress
      volumes:
        - name: wordpress-data
          persistentVolumeClaim:
            claimName: test-wordpress
---
# Source: wordpress/charts/mariadb/templates/primary/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: test-mariadb
  namespace: helm-test
  labels:
    app.kubernetes.io/name: mariadb
    helm.sh/chart: mariadb-9.3.14
    app.kubernetes.io/instance: test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: mariadb
      app.kubernetes.io/instance: test
      app.kubernetes.io/component: primary
  serviceName: test-mariadb
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        checksum/configuration: ba8296f4257f44a12c500b7f1720b6f3c44eb6b885a21e83bc3175cf4859939f
      labels:
        app.kubernetes.io/name: mariadb
        helm.sh/chart: mariadb-9.3.14
        app.kubernetes.io/instance: test
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/component: primary
    spec:

      serviceAccountName: test-mariadb
      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: mariadb
                    app.kubernetes.io/instance: test
                    app.kubernetes.io/component: primary
                namespaces:
                  - "helm-test"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:

      securityContext:
        fsGroup: 1001
      containers:
        - name: mariadb
          image: docker.io/bitnami/mariadb:10.5.10-debian-10-r18
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsUser: 1001
          env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: MARIADB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: test-mariadb
                  key: mariadb-root-password
            - name: MARIADB_USER
              value: "bn_wordpress"
            - name: MARIADB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: test-mariadb
                  key: mariadb-password
            - name: MARIADB_DATABASE
              value: "bitnami_wordpress"
          ports:
            - name: mysql
              containerPort: 3306
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 120
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
            exec:
              command:
                - /bin/bash
                - -ec
                - |
                  password_aux="${MARIADB_ROOT_PASSWORD:-}"
                  if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
                      password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
                  fi
                  mysqladmin status -uroot -p"${password_aux}"
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
            exec:
              command:
                - /bin/bash
                - -ec
                - |
                  password_aux="${MARIADB_ROOT_PASSWORD:-}"
                  if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
                      password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
                  fi
                  mysqladmin status -uroot -p"${password_aux}"
          resources:
            limits: {}
            requests: {}
          volumeMounts:
            - name: data
              mountPath: /bitnami/mariadb
            - name: config
              mountPath: /opt/bitnami/mariadb/conf/my.cnf
              subPath: my.cnf
      volumes:
        - name: config
          configMap:
            name: test-mariadb
  volumeClaimTemplates:
    - metadata:
        name: data
        labels:
          app.kubernetes.io/name: mariadb
          app.kubernetes.io/instance: test
          app.kubernetes.io/component: primary
      spec:
        accessModes:
          - "ReadWriteOnce"
        resources:
          requests:
            storage: "8Gi"

NOTES:
** Please be patient while the chart is being deployed **

Your WordPress site can be accessed through the following DNS name from within your cluster:

    test-wordpress.helm-test.svc.cluster.local (port 80)

To access your WordPress site from outside the cluster follow the steps below:

1. Get the WordPress URL by running these commands:

  NOTE: It may take a few minutes for the LoadBalancer IP to be available.
        Watch the status with: 'kubectl get svc --namespace helm-test -w test-wordpress'

   export SERVICE_IP=$(kubectl get svc --namespace helm-test test-wordpress --template "{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}")
   echo "WordPress URL: http://$SERVICE_IP/"
   echo "WordPress Admin URL: http://$SERVICE_IP/admin"

2. Open a browser and access WordPress using the obtained URL.

3. Login with the following credentials below to see your blog:

  echo Username: user
  echo Password: $(kubectl get secret --namespace helm-test test-wordpress -o jsonpath="{.data.wordpress-password}" | base64 --decode)

ストレージの有効化

 microk8s enable storage

Enabling default storage class
[sudo] mars のパスワード: 

deployment.apps/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon

Kubernetes IDEであるLensをMicroK8sで使う を参考にlensをインストール

sudo snap install kontena-lens --classic

kontena-lensを起動

kontena-lens 

info: 📟 Setting Lens as protocol client for lens://
info: 📟 failed ❗
info: 🚀 Starting Lens from "/home/mars/snap/kontena-lens/179/.config/Lens"
info: 🐚 Syncing shell environment
info: 💾 Loading stores
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.0.0-beta.2
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.4.1
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.6.0-beta.2
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.6.0-beta.3
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.7.0-beta.0
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 2.7.0-beta.1
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 3.6.0-beta.1
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json): 4.2.2

Migrating embedded kubeconfig paths
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/lens-cluster-store.json
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/lens-extensions.json
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/lens-filesystem-provisioner-store.json
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-workspace-store.json): 4.2.0-beta.1
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/lens-workspace-store.json
STORE MIGRATION (/home/mars/snap/kontena-lens/179/.config/Lens/lens-user-store.json): 2.1.0-beta.4
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/lens-user-store.json
info: 🔑 Getting free port for LensProxy server
info: 🔌 Starting LensProxy
info: [LENS-PROXY]: Proxy server has started at http://localhost:45293
info: 🔎 Testing LensProxy connection ...
error: ENOENT: no such file or directory, open '/home/mars/.kube/config' {"errno":-2,"code":"ENOENT","syscall":"open","path":"/home/mars/.kube/config"}
info: ⚡ LensProxy connection OK
info: 🖥️  Starting WindowManager
info: 🧩 Initializing extensions
info: [EXTENSION-DISCOVERY] loading extensions from /home/mars/snap/kontena-lens/179/.config/Lens

(kontena-lens:1373066): libappindicator-WARNING **: 08:48:20.134: Using '/tmp' paths in SNAP environment will lead to unreadable resources
info: [EXTENSION-INSTALLER] installing dependencies at /home/mars/snap/kontena-lens/179/.config/Lens
info: [WINDOW-MANAGER]: Loading Main window from url: http://localhost:45293 ...
info: [EXTENSION-INSTALLER] dependencies installed at /home/mars/snap/kontena-lens/179/.config/Lens
info: [EXTENSION-DISCOVERY] watching extension add/remove in /home/mars/.k8slens/extensions
info: [EXTENSION]: enabled lens-license@0.1.0
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/extension-store/lens-survey/preferences-store.json
info: [EXTENSION]: enabled lens-survey@0.1.0
telemetry main extension activated
info: [STORE]: LOADED from /home/mars/snap/kontena-lens/179/.config/Lens/extension-store/lens-telemetry/preferences-store.json
info: [EXTENSION]: enabled lens-telemetry@0.1.0
info: [WINDOW-MANAGER]: Main window loaded
info: 📡 Checking for app updates
info: Checking for update
error: Error: Error: ENOENT: no such file or directory, open '/snap/kontena-lens/179/resources/app-update.yml'
error: [UPDATE-CHECKER]: failed with an error {"error":"Error: ENOENT: no such file or directory, open '/snap/kontena-lens/179/resources/app-update.yml'"}

Microk8sを試す

microk8sを用いると簡単に単ノード構成のkubernetesを構築することができ、さらにkubernetesクラスタも構築することができるようなので試してみることにしました。

インストールの手順は、microk8sを踏襲することにします。

Install MicroK8s on Linux

sudo snap install microk8s --classic

ユーザを micro8ksのグループへ追加(次のログイン時に反映される)

sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube

microk8s起動状況の表示

$ microk8s status

microk8s is running
high-availability: yes
  datastore master nodes: 192.168.68.132:19001 192.168.68.129:19001 192.168.68.111:19001
  datastore standby nodes: 192.168.68.130:19001

addons:
  enabled:
    dashboard            # The Kubernetes dashboard
    dns                  # CoreDNS
    ha-cluster           # Configure high availability on the current node
    ingress              # Ingress controller for external access
    metallb              # Loadbalancer for your Kubernetes cluster
    metrics-server       # K8s Metrics Server for API access to service metrics
    storage              # Storage class; allocates storage from host directory
  disabled:
    ambassador           # Ambassador API Gateway and Ingress
    cilium               # SDN, fast with full network policy
    fluentd              # Elasticsearch-Fluentd-Kibana logging and monitoring
    gpu                  # Automatic enablement of Nvidia CUDA
    helm                 # Helm 2 - the package manager for Kubernetes
    helm3                # Helm 3 - Kubernetes package manager
    host-access          # Allow Pods connecting to Host services smoothly
    istio                # Core Istio service mesh services
    jaeger               # Kubernetes Jaeger operator with its simple config
    keda                 # Kubernetes-based Event Driven Autoscaling
    knative              # The Knative framework on Kubernetes.
    kubeflow             # Kubeflow for easy ML deployments
    linkerd              # Linkerd is a service mesh for Kubernetes and other frameworks
    multus               # Multus CNI enables attaching multiple network interfaces to pods
    portainer            # Portainer UI for your Kubernetes cluster
    prometheus           # Prometheus operator for monitoring and logging
    rbac                 # Role-Based Access Control for authorisation
    registry             # Private image registry exposed on localhost:32000
    traefik              # traefik Ingress controller for external access

Check the status while Kubernetes starts

microk8s status --wait-ready

disableされている中から必要なservices を有効化

microk8s enable dashboard dns registry istio

ここで microk8s enable –help を実行すると利用可能なサービスを一覧できます。

起動中の利用可能なサービスを表示

microk8s kubectl get all --all-namespaces
$ microk8s kubectl get all --all-namespaces

NAMESPACE        NAME                                             READY   STATUS        RESTARTS   AGE
ingress          pod/nginx-ingress-microk8s-controller-rj2vj      1/1     Running       0          22h
metallb-system   pod/speaker-bwpl7                                1/1     Running       0          22h
metallb-system   pod/speaker-g2kcl                                1/1     Running       1          22h
ingress          pod/nginx-ingress-microk8s-controller-dvb29      1/1     Running       1          22h
kube-system      pod/calico-node-jdpsl                            1/1     Running       1          23h
kube-system      pod/hostpath-provisioner-5c65fbdb4f-wlg45        1/1     Terminating   1          22h
kube-system      pod/coredns-86f78bb79c-r94k8                     1/1     Terminating   1          23h
kube-system      pod/hostpath-provisioner-5c65fbdb4f-6hz6n        1/1     Running       0          10h
kube-system      pod/calico-node-8hqxv                            1/1     Running       0          24h
ingress          pod/nginx-ingress-microk8s-controller-w5qv9      1/1     Running       0          22h
metallb-system   pod/speaker-wpg4z                                1/1     Running       0          22h
kube-system      pod/coredns-86f78bb79c-hl56s                     1/1     Terminating   0          10h
metallb-system   pod/controller-559b68bfd8-6nw4k                  1/1     Running       0          10h
metallb-system   pod/speaker-nrw9w                                1/1     Running       17         22h
kube-system      pod/coredns-86f78bb79c-72zp4                     1/1     Running       3          10h
kube-system      pod/calico-node-nk45x                            1/1     Running       18         24h
ingress          pod/nginx-ingress-microk8s-controller-cnpzq      1/1     Running       15         22h
kube-system      pod/metrics-server-8bbfb4bdb-9kh97               1/1     Running       0          38m
kube-system      pod/dashboard-metrics-scraper-6c4568dc68-vrmzl   1/1     Running       0          36m
kube-system      pod/calico-kube-controllers-847c8c99d-z867h      1/1     Running       0          33h
kube-system      pod/calico-node-ppczw                            1/1     Running       0          24h
kube-system      pod/kubernetes-dashboard-7ffd448895-q7zkf        1/1     Running       7          36m


NAMESPACE     NAME                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes                  ClusterIP   10.152.183.1     <none>        443/TCP                  33h
kube-system   service/kube-dns                    ClusterIP   10.152.183.10    <none>        53/UDP,53/TCP,9153/TCP   23h
kube-system   service/metrics-server              ClusterIP   10.152.183.231   <none>        443/TCP                  38m
kube-system   service/kubernetes-dashboard        ClusterIP   10.152.183.118   <none>        443/TCP                  36m
kube-system   service/dashboard-metrics-scraper   ClusterIP   10.152.183.83    <none>        8000/TCP                 36m

NAMESPACE        NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/calico-node                         4         4         2       4            2           kubernetes.io/os=linux        33h
metallb-system   daemonset.apps/speaker                             2         2         2       2            2           beta.kubernetes.io/os=linux   22h
ingress          daemonset.apps/nginx-ingress-microk8s-controller   2         2         2       2            2           <none>                        22h

NAMESPACE        NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system      deployment.apps/hostpath-provisioner        1/1     1            1           22h
metallb-system   deployment.apps/controller                  1/1     1            1           22h
kube-system      deployment.apps/calico-kube-controllers     1/1     1            1           33h
kube-system      deployment.apps/coredns                     1/1     1            1           23h
kube-system      deployment.apps/metrics-server              1/1     1            1           38m
kube-system      deployment.apps/dashboard-metrics-scraper   1/1     1            1           36m
kube-system      deployment.apps/kubernetes-dashboard        1/1     1            1           36m

NAMESPACE        NAME                                                   DESIRED   CURRENT   READY   AGE
kube-system      replicaset.apps/hostpath-provisioner-5c65fbdb4f        1         1         1       22h
metallb-system   replicaset.apps/controller-559b68bfd8                  1         1         1       22h
kube-system      replicaset.apps/calico-kube-controllers-847c8c99d      1         1         1       33h
kube-system      replicaset.apps/coredns-86f78bb79c                     1         1         1       23h
kube-system      replicaset.apps/metrics-server-8bbfb4bdb               1         1         1       38m
kube-system      replicaset.apps/dashboard-metrics-scraper-6c4568dc68   1         1         1       36m
kube-system      replicaset.apps/kubernetes-dashboard-7ffd448895        1         1         1       36m

Kubernetesの起動と停止

microk8s start
or
microk8s stop

ノードをクラスターへ追加;microk8s joink~ の行をCopyし、追加するノードの端末へPasteして実行する。

microk8s add-node

/* ノード参加に必要な以下のようなメッセージ が表示されるので、参加するノードで、microk8s joinjoinを実行 */

Join node with:
microk8s join ip-172-31-20-243:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf

If the node you are adding is not reachable through the default
interface you can use one of the following:

microk8s join 10.1.84.0:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf
microk8s join 10.22.254.77:25000/DDOkUupkmaBezNnMheTBqFYHLWINGDbf
  • 参加しているノードの表示
microk8s kubectl get no

frirewallの設定

sudo ufw allow in on cni0 && sudo ufw allow out on cni0
sudo ufw default allow routed

dashboardの利用するには、アクセスのためのトークンを作成します。

token=$(microk8s kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s kubectl -n kube-system describe secret $token

トークンの一例(dashboardをアクセスした際に要求されるtoken。Copy&Pasteでログイン)

Name:         default-token-57rfp
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: default
              kubernetes.io/service-account.uid: 2f782a0f-3d04-43aa-88fe-a6d67364b297

Type:  kubernetes.io/service-account-token

Data
====
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6Im56Vm1vTVJyaXVJQzBaSnM4SS1PTWNrZTkzMlJMdFBqS0NMeFgxWnIzdWMifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkZWZhdWx0LXRva2VuLTU3cmZwIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImRlZmF1bHQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiIyZjc4MmEwZi0zZDA0LTQzYWEtODhmZS1hNmQ2NzM2NGIyOTciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZGVmYXVsdCJ9.EGkufnRoGONpt14vmBLAG8vF2khLtHZLx8J0VNzJUxX28z8NtSNB5MhRKCamChqXdTRm1iwyaPQIlgWwaKTci7CA9TNf8o70EmJBOO1AvDJ_QcC2mRpQzrkxcl0wiuLbpkHzC-wvuzqwY4b32utYikbUxHNjiDsOSyVmqN9NyDZ84FKRZUGkaWrgJmUNakBGetqaOPSpTAIC8JncPznYIOt88nyx6kCIrOFJjjh_UhPGMfevVNFZcji617uNTencBVrkwaej6O09wyqzjPVK-jWXhHaigaIb5O2TmjfcQJCyiEkF_6LYFGr7ilOzzpbAqw-iICmBQUW1Mred3FsN9Q
ca.crt:     1103 bytes

dashboardへログインした画面

dashboardでノードの状態を表示

podの追加例:

インストールの過程で気になったキーワード

multipass  Ubuntu環境へ簡単にVMを構築

helm     Kubernetes 用パッケージマネージャー

WordPress  Helm Chartのデプロイ 

3ノード以上の構成では、データストアーがクラスター内で複製され、1ノード故障に対しては耐性がある。

From the 1.19 release of MicroK8s, HA is enabled by default. If your cluster consists of three or more nodes, the datastore will be replicated across the nodes and it will be resilient to a single failure (if one node develops a problem, workloads will continue to run without interruption).

ESP32/DCモータ/Websocket UIで倒立振子

PID制御のパラメータと自立のための目標角度(TARGET)をWebsocket UIでスマホから設定します。今のところ、パラメータの値が適切でないようで、硬い床面上では短時間で倒れます。やわらかい素材の上では倒れないので、この状態で適切なパラメータを試行錯誤で探索中、、、WebベースのUIのためか、細かなパラメータ設定の操作性がいまいちなので、改善が必要かもしれない。


倒立振子のコードは、https://qiita.com/Google_Homer/items/3897e7ffef9d247e2f56 を参考にしています。

Websocketのコードは、https://github.com/mgo-tec/ESP32_SPIFFS_EasyWebSocket を参考にしています。

#include <WiFi.h>
#include <WiFiMulti.h>
#include "ESP32_SPIFFS_EasyWebSocket.h" //beta ver 1.60

#include <MadgwickAHRS.h>
Madgwick MadgwickFilter;
#include "I2Cdev.h"
#define MPU6050_PWR_MGMT_1   0x6B
#define MPU_ADDRESS  0x68

#include "Wire.h"
#include <SPI.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>
#define SCREEN_WIDTH 128 // OLED display width, in pixels
#define SCREEN_HEIGHT 64 // OLED display height, in pixels
#define OLED_RESET     4 // Reset pin # (or -1 if sharing Arduino reset pin)
#define SCREEN_ADDRESS 0x3C ///< See datasheet for Address; 0x3D for 128x64, 0x3C for 128x32
Adafruit_SSD1306 display(SCREEN_WIDTH, SCREEN_HEIGHT, &Wire, OLED_RESET);

//PIDパラメータ調整
//PID係数
//#define TARGET            0.4
//#define KP                100.0
//#define KI                2.0
//#define KD                2.0

//Motor
#define LED_PIN            27                      // 内蔵LED
#define MOTOR_PIN_F        A4                      // to DC Motor Driver FIN
#define MOTOR_PIN_R        A18                      // to DC Motor Driver RIN
#define MOTOR_PWM_F        0                       // PWM CHANNEL
#define MOTOR_PWM_R        1                       // PWM CHANNEL
#define MOTOR_POWER_MIN    50
int MOTOR_POWER_MAX = 200;

#define DPS                1000                    // Gscale [deg/s]

const char* ssid = "*******"; //ご自分のルーターのSSIDに書き換えてください
const char* password = "******"; //ご自分のルーターのパスワードに書き換えてください

const char* HTM_head_file1 = "/EWS/LIPhead1.txt"; //HTMLヘッダファイル1
const char* HTM_head_file2 = "/EWS/LIPhead2.txt"; //HTMLヘッダファイル2
const char* HTML_body_file = "/EWS/dummy.txt"; //HTML body要素ファイル(ここではダミーファイルとしておく)
const char* dummy_file = "/EWS/dummy.txt"; //HTMLファイル連結のためのダミーファイル

ESP32_SPIFFS_EasyWebSocket ews;
WiFiMulti wifiMulti;

IPAddress LIP; //ローカルIPアドレス自動取得用

String ret_str; //ブラウザから送られてくる文字列格納用
String txt = "text send?"; //ブラウザから受信した文字列を ESP32から再送信する文字列

int PingSendTime = 10000; //ESP32からブラウザへPing送信する間隔(ms)

long ESP32_send_LastTime;
int ESP32_send_Rate = 300;
byte cnt = 0;

//PID
float power = 0, I = 0, preP = 0, preTime;
float now = 0, Duty = 0, pitch, roll, yaw;
float KP = 100.0, KI = 2.0, KD = 2.0, TARGET = 0.0;
boolean f_disp = false;

void setup() {
  Wire.begin();
  Serial.begin(38400);
  Wire.beginTransmission(MPU_ADDRESS);
  Wire.write(MPU6050_PWR_MGMT_1);  //MPU6050_PWR_MGMT_1レジスタの設定
  Wire.write(0x00);
  Wire.endTransmission();

  Serial.print(F("Connecting to "));
  Serial.println(ssid);

  wifiMulti.addAP(ssid, password);

  Serial.println(F("Connecting Wifi..."));
  if (wifiMulti.run() == WL_CONNECTED) {
    Serial.println("");
    Serial.println(F("WiFi connected"));
    Serial.println(F("IP address: "));
    LIP = WiFi.localIP(); //ESP32のローカルIPアドレスを自動取得
    Serial.println(WiFi.localIP());
  }
  ews.EWS_server_begin();
  Serial.println(); Serial.println("Initializing SPIFFS ...");

  if (!SPIFFS.begin()) {
    Serial.println("SPIFFS failed, or not present");
    return;
  }

  Serial.println("SPIFFS initialized. OK!");
  MadgwickFilter.begin(100); //100Hz
  // SSD1306_SWITCHCAPVCC = generate display voltage from 3.3V internally
  if (!display.begin(SSD1306_SWITCHCAPVCC, SCREEN_ADDRESS)) {
    Serial.println(F("SSD1306 allocation failed"));
    for (;;); // Don't proceed, loop forever
  }
  pinMode(LED_PIN, OUTPUT);
  digitalWrite(LED_PIN, HIGH);
  pinMode(MOTOR_PIN_F, OUTPUT);
  pinMode(MOTOR_PIN_R, OUTPUT);

  ledcSetup(MOTOR_PWM_F, 312500, 8); //CHANNEL, FREQ, BIT
  ledcSetup(MOTOR_PWM_R, 312500, 8);
  ledcAttachPin(MOTOR_PIN_F, MOTOR_PWM_F);
  ledcAttachPin(MOTOR_PIN_R, MOTOR_PWM_R);
  // 初期化

  delay(10);

  // Clear the buffer
  display.clearDisplay();
  // テキストサイズを設定
  display.setTextSize(1);
  // テキスト色を設定
  display.setTextColor(WHITE);
  display.setCursor(20, 5);
  display.println("Start...");
  display.setCursor(20, 25);
  display.println(WiFi.localIP());
  display.display();

  delay(10); // Pause for 2 seconds
  preTime = micros();
  //  TaskHandle_t th; //ESP32 マルチタスク ハンドル定義
  //  xTaskCreatePinnedToCore(Task1, "Task1", 4096, NULL, 5, &th, 0); //マルチタスク core 0 実行

  ESP32_send_LastTime = millis();
}

void loop() {
  websocket_handshake();

  if (ret_str != "_close") {
    if (millis() - ESP32_send_LastTime > ESP32_send_Rate) {
      if (cnt > 3) {
        cnt = 0;
      }
      websocket_send(cnt, txt);
      cnt++;
      ESP32_send_LastTime = millis();
    }
    ret_str = ews.EWS_ESP32CharReceive(PingSendTime);
    if (ret_str != "\0") {
      Serial.println(ret_str);
      if (ret_str != "Ping") {
        if (ret_str[0] != 't') {
          int ws_data = (ret_str[0] - 0x30) * 100 + (ret_str[1] - 0x30) * 10 + (ret_str[2] - 0x30);
          switch (ret_str[4]) {
            case '!':
              ESP32_send_Rate = ws_data;
              break;
            case 'B':
              TARGET = float(5.0 - ws_data / 20.0);
              break;
            case 'G':
              KP = float(ws_data / 1.8) ;
              break;
            case 'R':
              KI = float(ws_data / 50.0);
              break;
            case '_':
              KD = float(ws_data / 50.0);
              break;
            case 'A':
              f_disp = ! f_disp;
              break;
            case 'O':
              KP = 100, KI = 2.0, KD = 2.0, TARGET = 0.0;
              break;
          }
        } else if (ret_str[0] == 't') {
          txt = ret_str.substring(ret_str.indexOf('|') + 1, ret_str.length() - 1);
          Serial.println(txt);
        }
      }
    }
  } else if (ret_str == "_close") {
    ESP32_send_LastTime = millis();
    ret_str = "";
  }
  PID();
}

//************* 倒立振り子 ****************************************
void PID() {
  float  P, D, dt, Time;
  static float  powerI = 0;

  Wire.beginTransmission(0x68);
  Wire.write(0x3B);
  Wire.endTransmission(false);
  Wire.requestFrom(0x68, 14, true);
  while (Wire.available() < 14);
  int16_t axRaw, ayRaw, azRaw, gxRaw, gyRaw, gzRaw, Temperature;

  axRaw = Wire.read() << 8 | Wire.read();
  ayRaw = Wire.read() << 8 | Wire.read();
  azRaw = Wire.read() << 8 | Wire.read();
  Temperature = Wire.read() << 8 | Wire.read();
  gxRaw = Wire.read() << 8 | Wire.read();
  gyRaw = Wire.read() << 8 | Wire.read();
  gzRaw = Wire.read() << 8 | Wire.read();

  // 加速度値を分解能で割って加速度(G)に変換する
  float acc_x = axRaw / 16384.0;  //FS_SEL_0 16,384 LSB / g
  float acc_y = ayRaw / 16384.0;
  float acc_z = azRaw / 16384.0;

  // 角速度値を分解能で割って角速度(degrees per sec)に変換する
  float gyro_x = gxRaw / 131.0;  // (度/s)
  float gyro_y = gyRaw / 131.0;
  float gyro_z = gzRaw / 131.0;

  /*
    //c.f. Madgwickフィルターを使わずに、PRY(pitch, roll, yaw)を計算
    double roll  = atan2(acc_y, acc_z) * RAD_TO_DEG;
    double pitch = atan(-acc_x / sqrt(acc_y * acc_y + acc_z * acc_z)) * RAD_TO_DEG;
  */

  //Madgwickフィルターを用いて、PRY(pitch, roll, yaw)を計算
  MadgwickFilter.updateIMU(gyro_x, gyro_y, gyro_z, acc_x, acc_y, acc_z);

  //PRYの計算結果を取得する
  roll  = MadgwickFilter.getRoll();
  pitch = MadgwickFilter.getPitch();
  yaw   = MadgwickFilter.getYaw();
  now     = TARGET - roll                 ; // 目標角度から現在の角度を引いて偏差を求める
  if (f_disp) {
    display.clearDisplay();
    display.setCursor(20, 5);
    display.println( roll);
    display.setCursor(20, 25);
    display.println( now);
    display.display();
  }
  if (-20 < now && now < 20) {
    Time    = micros()                    ;
    dt      = (Time - preTime) / 1000000  ; // 処理時間を求める
    preTime = Time                        ; // 処理時間を記録
    P       = now / 90                    ; // -90~90→-1.0~1.0
    I      += P * dt                      ; // 偏差を積分する
    D       = (P - preP) / dt             ; // 偏差を微分する
    preP    = P                           ; // 偏差を記録する
    power  += KP * P + KI * I + KD * D    ; // 出力を計算する
    if (power < -1) power = -1            ; // →-1.0~1.0
    if (1 < power)  power =  1            ;
    //Motor駆動
    Duty = (int)((MOTOR_POWER_MAX - MOTOR_POWER_MIN) * abs(power) + MOTOR_POWER_MIN);
    ledcWrite( MOTOR_PWM_F, (power < 0 ?    0 : Duty) );
    ledcWrite( MOTOR_PWM_R, (power < 0 ? Duty :    0) );
    digitalWrite(LED_PIN, HIGH);
    if (f_disp) {
      display.clearDisplay();
      display.setCursor(20, 5);
      display.println( roll);
      display.setCursor(20, 25);
      display.println( Duty);
      display.setCursor(20, 45);
      display.println( power);
      display.display();
    }
  } else {                                                  // 転倒したら停止
    ledcWrite(MOTOR_PWM_F, 0);
    ledcWrite(MOTOR_PWM_R, 0);
    power = 0;
    I = 0;
    digitalWrite(LED_PIN, LOW);
  }

}

//**************************************************************
void LED_PWM(byte Led_gr, byte channel, int data_i) {
  Serial.println(data_i);
}
//*********************************************
void websocket_send(uint8_t count, String str_txt) {
  String str, tmp;
  //※WebSocketへのテキスト送信は110 byte 程度なので、全角35文字程度に抑えること
  tmp = "AGL=" + String(roll) + ",TGT=" + String(TARGET) + ",P=" + String(KP) +  ",I=" + String(KI) + ",D=" + String(KD);
  switch (cnt) {
    case 0:
      //str = str_txt;
      str = tmp;
      break;
    case 1:
      str = tmp;
      break;
    case 2:
      str = tmp;
      break;
    case 3:
      str = tmp;
      break;
  }

  ews.EWS_ESP32_Str_SEND(str, "wroomTXT"); //ブラウザに文字列を送信
}
//************************* Websocket handshake **************************************
void websocket_handshake() {
  if (ews.Get_Http_Req_Status()) { //ブラウザからGETリクエストがあったかどうかの判定
    String html_str1 = "", html_str2 = "", html_str3 = "", html_str4 = "", html_str5 = "", html_str6 = "", html_str7 = "";

    //※String変数一つにEWS_Canvas_Slider_T関数は2つまでしか入らない
    html_str1 += "<body style='background:#000; color:#fff;'>\r\n";
    html_str1 += "<font size=3>\r\n";
    html_str1 += "ESP-WROOM-32(ESP32)\r\n";
    html_str1 += "<br>\r\n";
    html_str1 += "ESP32_SPIFFS_EasyWebSocket Beta1.60 Sample\r\n";
    html_str1 += "</font><br>\r\n";
    html_str1 += ews.EWS_BrowserSendRate();
    html_str1 += "<br>\r\n";
    html_str1 += ews.EWS_ESP32_SendRate("!esp32t-Rate");
    html_str1 += "<br>\r\n";
    html_str1 += ews.EWS_BrowserReceiveTextTag2("wroomTXT", "from ESP32 DATA", "#555", 20, "#00FF00");
    html_str1 += "<br>\r\n";
    html_str1 += ews.EWS_Status_Text2("WebSocket Status", "#555", 20, "#FF00FF");
    html_str1 += "<br><br>\r\n";

    html_str2 += ews.EWS_TextBox_Send("txt1", "Hello Easy WebSocket Beta1.60", "送信");
    html_str2 += "<br><br>\r\n";
    html_str2 += "SETTING \r\n";
    html_str2 += ews.EWS_On_Momentary_Button("ALL", "ALL-ON", 80, 25, 15, "#000000", "#AAAAAA");
    html_str2 += ews.EWS_On_Momentary_Button("OUT", "DEFALT", 80, 25, 15, "#FFFFFF", "#555555");
    html_str2 += "<br>\r\n";

    html_str3 += "<br>TGT_ :\r\n";
    html_str3 += ews.EWS_Canvas_Slider_T("BLUE", 240, 40, "#777777", "#0000ff"); //CanvasスライダーはString文字列に2つまでしか入らない
    html_str3 += "<br>PID P:\r\n";
    html_str3 += ews.EWS_Canvas_Slider_T("GREEN", 250, 40, "#777777", "#00ff00"); //CanvasスライダーはString文字列に2つまでしか入らない

    html_str4 += "<br>PID I:\r\n";
    html_str4 += ews.EWS_Canvas_Slider_T("RED", 250, 40, "#777777", "#ff0000"); //CanvasスライダーはString文字列に2つまでしか入らない
    html_str4 += "<br>PID D:\r\n";
    html_str4 += ews.EWS_Canvas_Slider_T("_RGB", 250, 40, "#777777", "#ffff00");

    html_str7 += "<br><br>\r\n";
    html_str7 += ews.EWS_WebSocket_Reconnection_Button2("WS-Reconnect", "grey", 200, 40, "black" , 17);
    html_str7 += "<br><br>\r\n";
    html_str7 += ews.EWS_Close_Button2("WS CLOSE", "#bbb", 150, 40, "red", 17);
    html_str7 += ews.EWS_Window_ReLoad_Button2("ReLoad", "#bbb", 150, 40, "blue", 17);
    html_str7 += "</body></html>";

    //WebSocket ハンドシェイク関数
    ews.EWS_HandShake_main(3, HTM_head_file1, HTM_head_file2, HTML_body_file, dummy_file, LIP, html_str1, html_str2, html_str3, html_str4, html_str5, html_str6, html_str7);
  }
}