killercoda CKA:Troubleshooting - 2

killercoda CKA:Troubleshooting - 2

D瓜哥
1. Troubleshooting - Persistent Volume, Persistent Volume Claim - Issue Troubleshooting - Persistent Volume, Persistent Volume Claim - Issue my-pvc Persistent Volume Claim is stuck in a Pending state, fix this issue, make sure it is in Bound state # @author D瓜哥 · https://www.diguage.com $ kubectl get pvc my-pvc -o wide NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE my-pvc Pending standard <unset> 38s Filesystem $ kubectl get pv my-pv -o wide NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE VOLUMEMODE my-pv 100Mi RWO Retain Available standard <unset> 51s Filesystem $ kubectl get pvc my-pvc -o yaml | tee pv.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"my-pvc","namespace":"default"},"spec":{"accessModes":["ReadWriteMany"],"resources":{"requests":{"storage":"150Mi"}},"storageClassName":"standard"}} creationTimestamp: "2025-01-20T13:08:41Z" finalizers: - kubernetes.io/pvc-protection name: my-pvc namespace: default resourceVersion: "2002" uid: a4c6c044-4118-47a4-97b9-ceb69fac3bc2 spec: accessModes: - ReadWriteMany resources: requests: storage: 150Mi storageClassName: standard volumeMode: Filesystem status: phase: Pending $ kubectl get pv my-pv -o yaml apiVersion: v1 kind: PersistentVolume metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"name":"my-pv"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"100Mi"},"hostPath":{"path":"/mnt/data"},"persistentVolumeReclaimPolicy":"Retain","storageClassName":"standard"}} creationTimestamp: "2025-01-20T13:08:41Z" finalizers: - kubernetes.io/pv-protection name: my-pv resourceVersion: "2003" uid: 85a371c4-0931-4b57-87ea-fc1fceb674c1 spec: accessModes: - ReadWriteOnce capacity: storage: 100Mi hostPath: path: /mnt/data type: "" persistentVolumeReclaimPolicy: Retain storageClassName: standard volumeMode: Filesystem $ vim pv.yaml # 两个问题: # 1、 PVC 和 PV 的 accessModes 不一致,改为 ReadWriteOnce即可 # 2、 PVC 的存储是 150Mi,而 PV 只有 100Mi,也改为 100Mi 即可。 $ kubectl delete -f pv.yaml --force --grace-period 0 Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. persistentvolumeclaim "my-pvc" force deleted $ kubectl apply -f pv.yaml persistentvolumeclaim/my-pvc created $ kubectl get pvc my-pvc -o wide NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE my-pvc Bound my-pv 100Mi RWO standard <unset> 10s Filesystem
killercoda CKA:Troubleshooting - 1

killercoda CKA:Troubleshooting - 1

D瓜哥
1. Troubleshooting - Pod Issue Troubleshooting - Pod Issue hello-kubernetes pod not running, fix that issue # @author D瓜哥 · https://www.diguage.com $ kubectl get pod NAME READY STATUS RESTARTS AGE hello-kubernetes 0/1 RunContainerError 2 (6s ago) 29s $ kubectl describe pod hello-kubernetes Name: hello-kubernetes Namespace: default Priority: 0 Service Account: default Node: node01/172.30.2.2 Start Time: Mon, 20 Jan 2025 07:21:57 +0000 Labels: <none> Annotations: cni.projectcalico.org/containerID: 2e010161283b56bfd70d604c31ece3dc3189882f1e24c2ea57647dbaec3b2bdb cni.projectcalico.org/podIP: 192.168.1.4/32 cni.projectcalico.org/podIPs: 192.168.1.4/32 Status: Running IP: 192.168.1.4 IPs: IP: 192.168.1.4 Containers: echo-container: Container ID: containerd://4f01851fcb908cd7bd1031a1726b8b75873d69fb246a5eebdd5c3dc003be7c19 Image: redis Image ID: docker.io/library/redis@sha256:ca65ea36ae16e709b0f1c7534bc7e5b5ac2e5bb3c97236e4fec00e3625eb678d Port: <none> Host Port: <none> Command: shell -c while true; do echo 'Hello Kubernetes'; sleep 5; done State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: StartError Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "shell": executable file not found in $PATH: unknown Exit Code: 128 Started: Thu, 01 Jan 1970 00:00:00 +0000 Finished: Mon, 20 Jan 2025 07:22:20 +0000 Ready: False Restart Count: 2 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xk5qj (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: kube-api-access-xk5qj: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 41s default-scheduler Successfully assigned default/hello-kubernetes to node01 Normal Pulled 35s kubelet Successfully pulled image "redis" in 5.57s (5.57s including waiting). Image size: 45006722 bytes. Normal Pulled 33s kubelet Successfully pulled image "redis" in 422ms (422ms including waiting). Image size: 45006722 bytes. Normal Pulling 19s (x3 over 40s) kubelet Pulling image "redis" Normal Created 18s (x3 over 35s) kubelet Created container echo-container Warning Failed 18s (x3 over 34s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "shell": executable file not found in $PATH: unknown Normal Pulled 18s kubelet Successfully pulled image "redis" in 467ms (467ms including waiting). Image size: 45006722 bytes. Warning BackOff 6s (x4 over 32s) kubelet Back-off restarting failed container echo-container in pod hello-kubernetes_default(5a459cd4-866a-4e57-8d44-ae83156e1e0b) $ kubectl get pod hello-kubernetes -o yaml | tee pod.yaml apiVersion: v1 kind: Pod metadata: annotations: cni.projectcalico.org/containerID: 2e010161283b56bfd70d604c31ece3dc3189882f1e24c2ea57647dbaec3b2bdb cni.projectcalico.org/podIP: 192.168.1.4/32 cni.projectcalico.org/podIPs: 192.168.1.4/32 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"hello-kubernetes","namespace":"default"},"spec":{"containers":[{"command":["shell","-c","while true; do echo 'Hello Kubernetes'; sleep 5; done"],"image":"redis","name":"echo-container"}]}} creationTimestamp: "2025-01-20T07:21:57Z" name: hello-kubernetes namespace: default resourceVersion: "2157" uid: 5a459cd4-866a-4e57-8d44-ae83156e1e0b spec: containers: - command: - shell - -c - while true; do echo 'Hello Kubernetes'; sleep 5; done image: redis imagePullPolicy: Always name: echo-container resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-xk5qj readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: node01 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: kube-api-access-xk5qj projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace # 省略了 status 字段 $ vim pod.yaml # 根据提示,没有 shell,将 shell 修改为 sh 即可。 $ kubectl replace -f pod.yaml Error from server (Conflict): error when replacing "pod.yaml": Operation cannot be fulfilled on pods "hello-kubernetes": the object has been modified; please apply your changes to the latest version and try again # 不能替换,就直接删除,再重建 $ kubectl delete -f pod.yaml --force --grace-period 0 Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "hello-kubernetes" force deleted $ kubectl apply -f pod.yaml pod/hello-kubernetes created $ kubectl get pod NAME READY STATUS RESTARTS AGE hello-kubernetes 1/1 Running 0 5s
killercoda CKA:Workloads & Scheduling

killercoda CKA:Workloads & Scheduling

D瓜哥
1. Workloads & Scheduling - Pod Workloads & Scheduling - Pod Fresher deployed a pod named my-pod. However, while specifying the resource limits, they mistakenly given 100Mi storage limit instead of 50Mi node doesn’t have sufficient resources, So change it to 50Mi only. # @author D瓜哥 · https://www.diguage.com $ kubectl get pod my-pod -o yaml | tee pod.yaml apiVersion: v1 kind: Pod metadata: annotations: cni.projectcalico.org/containerID: 8414bfefda21fa6ca74ef8d499c92a22ae6cc0dbb6d0bc4d82eb0129a795d75d cni.projectcalico.org/podIP: 192.168.1.4/32 cni.projectcalico.org/podIPs: 192.168.1.4/32 kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"my-pod","namespace":"default"},"spec":{"containers":[{"image":"nginx:latest","name":"my-container","resources":{"limits":{"memory":"100Mi"},"requests":{"memory":"50Mi"}}}]}} creationTimestamp: "2025-01-14T07:53:50Z" name: my-pod namespace: default resourceVersion: "2026" uid: fcf1e97e-cec0-45b0-b82d-766ad0c51823 spec: containers: - image: nginx:latest imagePullPolicy: Always name: my-container resources: limits: memory: 100Mi requests: memory: 50Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-thchj readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: node01 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: kube-api-access-thchj projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace # 省略没用的 status 字段 $ vim pod.yaml # 将 limit 中,100Mi 改为 50Mi $ kubectl delete -f pod.yaml --force --grace-period 0 Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "my-pod" force deleted $ kubectl apply -f pod.yaml pod/my-pod created
killercoda CKA:Storage

killercoda CKA:Storage

D瓜哥
1. Storage - Persistent Volume Storage - Persistent Volume Create a PersistentVolume (PV) named black-pv-cka with the following specifications: Volume Type: hostPath Path: /opt/black-pv-cka Capacity: 50Mi # @author D瓜哥 · https://www.diguage.com $ vim pv.yaml # 编写 YAML 文件 $ cat pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: black-pv-cka spec: capacity: storage: 50Mi accessModes: - ReadWriteOnce persistentVolumeReclaimPolicy: Retain hostPath: path: /opt/black-pv-cka $ kubectl apply -f pv.yaml persistentvolume/black-pv-cka created 2. Storage - Persistent Volume Claim Storage - Persistent Volume Claim
killercoda CKA:Services & Networking

killercoda CKA:Services & Networking

D瓜哥
1. Services & Networking - Services Services & Networking - Services You have an existing Nginx pod named nginx-pod. Perform the following steps: Expose the nginx-pod internally within the cluster using a Service named nginx-service . Use port forwarding to service to access the Welcome content of nginx-pod using the curl command. # @author D瓜哥 · https://www.diguage.com $ kubectl get pod --show-labels NAME READY STATUS RESTARTS AGE LABELS nginx-pod 1/1 Running 0 8m48s app=nginx $ cat svc.yaml apiVersion: v1 kind: Service metadata: name: nginx-service spec: selector: app: nginx ports: - name: http protocol: TCP port: 80 targetPort: 80 $ kubectl apply -f svc.yaml service/nginx-service created $ kubectl port-forward service/nginx-service 8081:80 Forwarding from 127.0.0.1:8081 -> 80 Forwarding from [::1]:8081 -> 80 Handling connection for 8081 # 打开另外一个终端 $ curl localhost:8081 <!DOCTYPE html> <html> <head> <title>Welcome to nginx!</title> <style> html { color-scheme: light dark; } body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style> </head> <body> <h1>Welcome to nginx!</h1> <p>If you see this page, the nginx web server is successfully installed and working. Further configuration is required.</p> <p>For online documentation and support please refer to <a href="http://nginx.org/">nginx.org</a>.<br/> Commercial support is available at <a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p> </body> </html>
killercoda CKA:Architecture, Installation & Maintenance

killercoda CKA:Architecture, Installation & Maintenance

D瓜哥
1. Architecture, Installation & Maintenance - Create Pod Architecture, Installation & Maintenance - Create Pod Create a pod called sleep-pod using the nginx image and also sleep (using command ) for give any value for seconds. # @author D瓜哥 · https://www.diguage.com $ cat nginx.yaml apiVersion: v1 kind: Pod metadata: name: sleep-pod spec: containers: - name: nginx image: nginx command: - sleep - "3600" $ kubectl apply -f nginx.yaml pod/sleep-pod created $ kubectl get pod NAME READY STATUS RESTARTS AGE sleep-pod 1/1 Running 0 5s
玩转 Kubernetes(一):离线安装 Kubernetes

玩转 Kubernetes(一):离线安装 Kubernetes

D瓜哥
在 基于 Docker 搭建开发环境(三):链路追踪 等几篇文章中,D瓜哥分享了如何使用 Docker Compose 在本地搭建起来一套应用可观测性环境。感觉还不够好玩,毕竟正在在企业中,Kubernetes 已经是绝对的主流。要玩就玩最具挑战性的东西,玩最符合企业所需的技能和工具。所以,打算将上面那套简易玩具,按照企业级的要求,搬到 Kubernetes 上去。 如果想玩 Kubernetes,首先面临的一个问题就是 Kubernetes 集群的搭建。本来是一个非常简单的事情,但是由于众所周知的原因,变得非常具有挑战性。经过各种探索和多次试验,发现一种“离线”安装方式,感觉是一个不错的方式。 本方法是基于 Kubespray 的一种安装办法,Kubespray 是由 Kubernetes SIG 小组来负责维护的一整套安装方式。既可以支持在裸机环境上安装,也支持云上环境安装。而且,只需要简单几行可以复制粘贴的命令,即可完成安装工作。非常适合入门玩耍使用。 本安装方法所需的软件,D瓜哥都已经上传到 GitHub,如果需要下载,请移步: Kubespray-2.26.0 安装包大全。 搭建服务器集群 这里推荐使用 Vagrant 搭建集群。搭配 VirtualBox,只需要一个配置文件,就可以轻轻松松搭建一个 Linux 服务器集群。搭建集群的配置文件 Vagrantfile 如下: # -*- mode: ruby -*- # vi: set ft=ruby : # @author D瓜哥 · https://www.diguage.com/ # All Vagrant configuration is done below. The "2" in Vagrant.configure # configures the configuration version (we support older styles for # backwards compatibility). Please don't change it unless you know what # you're doing. Vagrant.configure("2") do |config| # The most common configuration options are documented and commented below. # For a complete reference, please see the online documentation at # https://docs.vagrantup.com. # 三节点集群 (1..3).each do |i| config.vm.define "node#{i}" do |node| # Every Vagrant development environment requires a box. You can search for # boxes at https://vagrantcloud.com/search. node.vm.box = "ubuntu2404" # 设置虚拟机的主机名 node.vm.hostname = "node#{i}" config.vm.boot_timeout = 600 # Disable automatic box update checking. If you disable this, then # boxes will only be checked for updates when the user runs # `vagrant box outdated`. This is not recommended. # config.vm.box_check_update = false # Create a forwarded port mapping which allows access to a specific port # within the machine from a port on the host machine. In the example below, # accessing "localhost:8080" will access port 80 on the guest machine. # NOTE: This will enable public access to the opened port # config.vm.network "forwarded_port", guest: 80, host: 8080 # Create a forwarded port mapping which allows access to a specific port # within the machine from a port on the host machine and only allow access # via 127.0.0.1 to disable public access # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1" # Create a private network, which allows host-only access to the machine # using a specific IP. # 设置虚拟机的IP node.vm.network "private_network", ip: "10.0.2.#{20+i}", auto_config: true # Create a public network, which generally matched to bridged network. # Bridged networks make the machine appear as another physical device on # your network. # config.vm.network "public_network" # Share an additional folder to the guest VM. The first argument is # the path on the host to the actual folder. The second argument is # the path on the guest to mount the folder. And the optional third # argument is a set of non-required options. # 设置主机与虚拟机的共享目录,根据需要开启 node.vm.synced_folder "/path/to/#{i}", "/data" # Disable the default share of the current code directory. Doing this # provides improved isolation between the vagrant box and your host # by making sure your Vagrantfile isn't accessible to the vagrant box. # If you use this you may want to enable additional shared subfolders as # shown above. # config.vm.synced_folder ".", "/vagrant", disabled: true # Provider-specific configuration so you can fine-tune various # backing providers for Vagrant. These expose provider-specific options. # Example for VirtualBox: node.vm.provider "virtualbox" do |vb| # 设置虚拟机的名称 # vb.name = "node#{i}" # if node.vm.hostname == "node1" # # Display the VirtualBox GUI when booting the machine # vb.gui = true # end # Customize the amount of memory on the VM: vb.memory = "6144" # 设置虚拟机的CPU个数 vb.cpus = 2 end # View the documentation for the provider you are using for more # information on available options. # Enable provisioning with a shell script. Additional provisioners such as # Ansible, Chef, Docker, Puppet and Salt are also available. Please see the # documentation for more information about their specific syntax and use. # config.vm.provision "shell", inline: <<-SHELL # sudo yum makecache --refresh # sudo yum install -y tcpdump # sudo yum install -y nc # sudo yum install -y net-tools # SHELL end end end
理解数据库分片

理解数据库分片

D瓜哥
最近在 DigitalOcean 社区看到一篇文章,讲解数据库分片架构的,感觉非常不错,图文并茂,翻译过来,分享给需要的朋友。 介绍 任何应用程序或网站,如果出现大幅增长,最终都需要进行扩展,以适应流量的增加。对于数据驱动型应用程序和网站来说,在进行扩展时必须确保数据的安全性和完整性。很难预测一个网站或应用程序会变得多受欢迎,或者它的受欢迎程度会维持多久,这就是为什么一些组织会选择一种允许他们动态扩展数据库的数据库架构。 在这篇概念性文章中,我们将讨论这样一种数据库架构:分片数据库。近年来,分片数据库受到了广泛关注,但很多人并不清楚什么是分片数据库,也不知道在哪些情况下分片数据库才有意义。我们将介绍什么是分片、分片的一些主要优点和缺点,以及几种常见的分片方法。 什么是分片? 分片是一种与水平分区相关的数据库架构模式,即把一个表的行分成多个不同的表,称为分区。每个分区都有相同的模式和列,但也有完全不同的行。同样,每个分区中的数据都是唯一的,与其他分区中的数据无关。 从水平分区与垂直分区的关系角度来思考水平分区可能会有所帮助。在垂直分区表中,整个列都被分离出来并放入新的、不同的表中。一个垂直分区中的数据独立于所有其他分区中的数据,每个分区都有不同的行和列。下图说明了如何对表格进行水平和垂直分区: 图 1. 水平分区与垂直分区 分片是指将数据分割成两个或多个较小的块,称为逻辑分片。然后,逻辑分片分布在不同的数据库节点上,称为物理分片,物理分片可容纳多个逻辑分片。尽管如此,所有分片中保存的数据共同代表了一个完整的逻辑数据集。 数据库分片是无共享架构的典范。这意味着分片是独立的,它们不共享任何相同的数据或计算资源。不过,在某些情况下,将某些表复制到每个分片中作为参考表是有意义的。例如,假设有一个应用程序的数据库依赖于重量测量的固定转换率。通过将包含必要转换率数据的表复制到每个分片中,有助于确保每个分片中都包含查询所需的所有数据。 通常,分片是在应用程序级实现的,这意味着应用程序包含定义向哪个分片传输读写的代码。不过,有些数据库管理系统内置了分片功能,允许你直接在数据库级实施分片。 鉴于以上对分片的概述,让我们来看看这种数据库架构的一些优点和缺点。 分片的优点 对数据库进行分片的主要吸引力在于,它有助于促进水平扩展,也称为向外扩展,横向扩展。水平扩展是指在现有堆栈中添加更多机器,以分散负载,允许更多流量和更快处理。这通常与垂直扩展(也称向上扩展)形成对比,后者涉及升级现有服务器的硬件,通常是增加更多内存或 CPU。 在一台机器上运行一个关系数据库,并根据需要通过升级其计算资源来扩大其规模相对简单。但归根结底,任何非分布式数据库在存储和计算能力方面都是有限的,因此可以自由横向扩展,会让你的设置更加灵活。 一些人选择分片数据库架构的另一个原因是为了加快查询响应速度。在未分片的数据库上提交查询时,数据库可能需要搜索查询表中的每一行,然后才能找到所需的结果集。对于使用大型单体数据库的应用程序来说,查询速度会慢得令人望而却步。不过,通过将一个表分片成多个表后,查询需要处理的行数就会减少,返回结果集的速度也会快得多。 分片还可以减轻中断造成的影响,从而提高应用程序的可靠性。如果您的应用程序或网站依赖的是未分片的数据库,中断有可能导致整个应用程序不可用。 而使用分片数据库时,故障可能只影响单个分片。尽管这可能会导致部分用户无法使用应用程序或网站的某些部分,但总体影响仍小于整个数据库崩溃的影响。 分片的缺点 虽然分片可以使数据库的扩展更容易并提高性能,但它也会带来一些限制。在此,我们将讨论其中的一些限制,以及为什么要避免使用分片。 人们在使用分片时遇到的第一个困难是正确实施分片数据库架构的复杂性。如果操作不当,分片过程很有可能导致数据丢失或表损坏。即使操作正确,分片也可能对团队的工作流程产生重大影响。用户必须跨多个分片位置管理数据,而不是从一个入口点访问和管理数据,这可能会对某些团队造成干扰。 用户在对数据库进行分片后有时会遇到一个问题,那就是分片最终会变得不平衡。举例来说,假设你的数据库有两个独立的分片,一个用于存储姓氏以字母 A 至 M 开头的客户,另一个用于存储姓氏以字母 N 至 Z 开头的客户。然而,你的应用程序为大量姓氏以字母 G 开头的人提供服务。 A-M 分区已成为所谓的数据库热点。在这种情况下,分片给数据库带来的任何好处都会被速度变慢和崩溃所抵消。数据库很可能需要修复和重新分片,以使数据分布更均匀。 另一个主要缺点是,一旦数据库被分片,就很难将其恢复到未分片的架构。数据库分片前的任何备份都不包括分片后写入的数据。 因此,要重建未分片的原始架构,就需要将新的分片数据与旧的备份合并,或者将分片后的数据库变回单一数据库,这两种方法都会耗费大量成本和时间。 最后一个需要考虑的缺点是,并非每个数据库引擎都支持分片。例如,PostgreSQL 不包括自动分片功能,但可以手动分片 PostgreSQL 数据库。 有一些 Postgres 变种确实包含自动分片功能,但它们往往落后于最新的 PostgreSQL 版本,而且缺乏某些其他功能。一些专门的数据库技术(如 MySQL Cluster 或某些数据库即服务产品(如 MongoDB Atlas))确实包含自动分片功能,但这些数据库管理系统的普通版本并不包含。因此,分片通常需要“自己开发”。这意味着通常很难找到分片文档或故障排除技巧。 当然,这些只是分片前需要考虑的一些一般性问题。根据其用例,对数据库进行分片可能会有更多潜在的缺点。 现在,我们已经介绍了分片的一些缺点和优点,下面将介绍几种不同的分片数据库架构。 分片架构 一旦决定对数据库进行分片,接下来需要考虑的就是如何分片。在运行查询或将输入数据分发到分片表或数据库时,将数据分发到正确的分片至关重要。否则,可能会导致数据丢失或查询缓慢。在本节中,我们将介绍几种常见的分片架构,每种架构都使用略有不同的流程在分片间分发数据。 基于键的分片 基于密钥的分片,也称为基于散列的分片,涉及使用从新写入的数据中提取的值,例如客户的 ID 编号、客户端应用程序的 IP 地址、邮政编码等并将其输入散列函数,以确定数据应进入哪个分片。散列函数是一种输入数据(如客户电子邮件)并输出离散值(即散列值)的函数。在分片的情况下,散列值是一个分片 ID,用于确定输入的数据将存储在哪个分片上。整个过程如下: 图 2. 基于键的分片 为确保条目以一致的方式放置于正确的分片,输入散列函数的值都应来自同一列。此列被称为分片键。简单来说,分片键与主键类似,都是用于为单个行建立唯一标识符的列。从广义上讲,分片键应该是静态的,也就是说,它不应该包含可能会随时间变化的值。否则,会增加更新操作的工作量,并可能降低性能。
基于 Docker 搭建开发环境(三):链路追踪

基于 Docker 搭建开发环境(三):链路追踪

D瓜哥
基于 Docker 搭建开发环境系列: 基于 Docker 搭建开发环境(一):数据库+监控 基于 Docker 搭建开发环境(二):EFK 日志套件 基于 Docker 搭建开发环境(三):链路追踪 在上一篇文章 基于 Docker 搭建开发环境(一):数据库+监控 和 基于 Docker 搭建开发环境(二):EFK 日志套件 两篇文章中,分别介绍了“数据库+监控”和“EFK 日志套件”。这篇文章给大家分享一下如何在本地搭建起一套简单的分布式链路追踪。 在 AI 的帮助下,如同砍瓜切菜一样,非常迅速地就完成了 基于 Docker 搭建开发环境(二):EFK 日志套件 的搭建。原以为搞这个也会分分钟的问题,结果应用的追踪数据一致无法正常发送到 Jaeger 中,各种改端口号都不行。后来,无意间看了 OpenTelemetry 的配置文档,增加了一个协议配置,全部流程竟然通了,非常神奇! 站在更高的视角去看,链路追踪其实是可观测性的一部分,包括上篇文章的日志,也是可观测性的一部分。日志、追踪、度量,三者是相辅相成的。 图 1. 可观测性 在 OpenTelemetry 出现之前,日志、追踪、度量是分离的,三者各各自为战。而 OpenTelemetry 的出现,则是试图将三者统一。目前 OpenTelemetry 是云原生架构中,最炙手可热的分布式链路追踪解决方案,它提供了一套相关标准,各个厂商可以在这套标准之上进行各种各样的组件开发,大家可以根据自己的需要,选择不同的组件,进行可插拔式的安装。 图 2. OpenTelemetry 的野心 在这篇文章中,链路追踪的解决方案选择的是 OpenTelemetry + OpenTelemetry Collector + Jaeger。 OpenTelemetry OpenTelemetry 并不需要在 Docker 中启动或者配置什么。在目前的架构中,Jaeger 是作为 OpenTelemetry 的一个实现来出现的。 OpenTelemetry 需要做的就是下载一个 Java Agent,执行 docker/config/opentelemetry/download-opentelemetry-agent.sh 脚本即可下载最新版的 Java Agent。在业务应用启动时,增加如下 JVM 参数:
基于 Docker 搭建开发环境(二):EFK 日志套件

基于 Docker 搭建开发环境(二):EFK 日志套件

D瓜哥
基于 Docker 搭建开发环境系列: 基于 Docker 搭建开发环境(一):数据库+监控 基于 Docker 搭建开发环境(二):EFK 日志套件 基于 Docker 搭建开发环境(三):链路追踪 在上一篇文章 基于 Docker 搭建开发环境(一):数据库+监控 中,介绍了一下如何使用 Docker 搭建起 MySQL + NACOS + Prometheus + Grafana 集成数据库、注册中心+配置管理、监控的开发环境。这篇文章来介绍一下如何在原来的基础上接入 Elasticsearch + Fluentd + Kibana 套件,并且将 NACOS 的日志接入到 Elasticsearch 里。 Elasticsearch 由于 Elasticsearch 8+ 的版本修改了安全策略,不允许 Kibana 使用超级管理员 elastic 连接 Elasticsearch,这里选用 7.x 版本做演示。 还有一点需要提醒,在设置 Elasticsearch 的超级管理员 elastic 的账户密码时,如果密码是全部的阿拉伯数字,那么需要用双引号或者单引号括起来。 在测试中,还遇到一个磁盘过载导致的只读问题。解决方式如下: curl -X GET "localhost:9200/_cat/allocation?v&pretty" 查看磁盘使用情况 解除只读状态 $ curl -X PUT "localhost:9200/test/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '