Longhorn 面试题

写入请求 → Engine → 并行写入所有 Replica（同步复制）
  → 每个 Replica 写入本地存储
  → 所有 Replica 返回确认 → Engine 返回写入成功

读取请求 → Engine → 从任意一个 Replica 读取（Round-Robin 负载均衡）
  → 读到的数据返回给上层

# 卷重建策略
replicaAutoBalance: "best-effort"   # 自动平衡 Replica 分布
replicaDiskSoftAntiAffinity: "ignore"  # 软反亲和性
replicaZoneSoftAntiAffinity: "ignore"  # 跨可用区

节点故障 → Longhorn Manager 检测到 Replica 不可达（~30秒超时）
  → 标记 Replica 为 Failed → 自动在其他健康节点新建 Replica
  → 新 Replica 从 Engine 处增量同步数据 → 恢复为 Healthy 状态

新 Replica 加入 → Engine 创建 Full Snapshot
  → 新 Replica 从 Engine 全量同步数据 → 同步完成后转为 Healthy
  → 后续写入按正常同步路径进行

节点宕机 → Kubernetes Node 状态 NotReady（约40秒）
  → Longhorn Manager 标记该节点上的所有 Replica 为 Unknown
  → 等待约 30 秒（设置 allowRecoveryWhileDetached: true）后标记 Failed
  → 创建新 Replica 在其他节点重建数据

节点恢复 → Instance Manager 重新启动
  → 检查本地 Replica 完整性 → 清理过期 Replica
  → Replica 状态从 Failed 转为 Healthy（数据在重建时已恢复）
  → Longhorn Manager 自动平衡删除多余的 Healthy Replica

Engine 进程崩溃 → Instance Manager 检测到进程退出
  → 自动启动新 Engine 进程（秒级恢复）
  → 新 Engine 加载卷配置和 Replica 映射
  → 恢复 I/O 请求处理
  → 客户端短暂 I/O 挂起后恢复（取决于客户端超时设置）

Backup (S3 bucket) ← 基于 Snapshot-3
  ↑
活跃数据 ─→ Snapshot-1 → Snapshot-2 → Snapshot-3 (本地)

1. PVC 创建 → CSI Provisioner 监听到请求
2. CSI Provisioner 调用 Longhorn Manager API 创建 Volume CR
3. Longhorn Manager 调度 Replica 到合适节点
4. Pod 调度 → CSI Attacher 调用 Longhorn 挂载 API
5. Engine 启动 → Replica 连接就绪 → 块设备出现在节点
6. kubelet 对块设备做格式化 → 挂载到 Pod 指定路径

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-fast
provisioner: driver.longhorn.io
allowVolumeExpansion: true
parameters:
  numberOfReplicas: "3"
  staleReplicaTimeout: "30"
  fromBackup: ""
  fsType: "ext4"
  dataLocality: "best-effort"
  diskSelector: "ssd,fast"
  nodeSelector: "storage-node"
  recurringJobs: '[{"name":"snap","task":"snapshot","cron":"*/5 * * * *"}]'

多个 Pod 挂载 PVC（accessModes: ReadWriteMany）
  → kubelet 调用 CSI 驱动 → CSI 检测到 RWX 请求
  → Longhorn Manager 创建 Share Manager Pod
  → Share Manager 挂载 Longhorn 块卷作为本地存储
  → Share Manager 启动 NFS Server 导出该卷
  → 各 Pod 通过 NFS Client 挂载到 Pod 内路径

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-data
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

# StorageClass 参数
parameters:
  staleReplicaTimeout: "30"   # 默认 30 分钟

# 手动清理 Stale Replica
kubectl -n longhorn-system get replicas | grep stale
kubectl -n longhorn-system delete replica <replica-name>

# 或通过 UI 清理
# Longhorn UI → Node → Replica → Remove

Stale Replica 超时 → Longhorn Manager 标记为 Failed
  → 检查可用节点和磁盘 → 在其他节点创建新 Replica
  → 新 Replica 从 Engine 同步数据 → 转为 Healthy
  → 原 Stale Replica 被清理删除

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-local
provisioner: driver.longhorn.io
parameters:
  dataLocality: "strict-local"
  numberOfReplicas: "3"  # 节点必须有 3 个磁盘

数据本地性 vs 读取延迟
Disabled:   Engine → 跨节点网络 → Replica（额外 0.1-0.5ms）
Best-effort: Engine → 本地 Replica（0.01-0.05ms）
Strict-local: Engine → 本地 Replica（0.01-0.05ms）

# StorageClass 必须允许扩容
allowVolumeExpansion: true

# 修改 PVC 大小
kubectl edit pvc my-data
# spec.resources.requests.storage: 10Gi → 20Gi

# 检查扩容状态
kubectl get volumes.longhorn.io my-data -o yaml
# status.conditions 显示 ExpandInProgress → ExpandCompleted

# Backup Target 配置
backupTarget:
  endpoint: nfs://192.168.1.100:/backups/longhorn
  credentialSecret: ""  # NFS 不需要密钥

# 创建 Secret
apiVersion: v1
kind: Secret
metadata:
  name: s3-secret
  namespace: longhorn-system
stringData:
  AWS_ACCESS_KEY_ID: "minioadmin"
  AWS_SECRET_ACCESS_KEY: "minioadmin"
  AWS_ENDPOINTS: "https://minio.example.com"
  AWS_REGION: "us-east-1"

# 通过 Longhorn CLI 检查
kubectl -n longhorn-system exec -it longhorn-manager-xxx -- longhorn-manager backup list
# 或检查卷的 Backup Status
kubectl get volumes.longhorn.io my-volume -o jsonpath='{.status.backupStatus}'

# 从 Backup 恢复为新的卷
kubectl create -f - <<EOF
apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: restored-volume
  namespace: longhorn-system
spec:
  fromBackup: "s3://bucket-name/backup-bundle/volume-name/?backup=backup-xxx"
  numberOfReplicas: 3
  size: "10737418240"  # 10Gi bytes
EOF

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: dr-volume
  namespace: longhorn-system
spec:
  fromBackup: "s3://bucket/backup-url"
  numberOfReplicas: 3
  standby: true  # 灾难恢复模式
  revisionCounterDisabled: true  # 灾备不维护修订计数器

主集群定期 Backup → 灾备集群的 DR Volume 自动检测新 Backup
  → 增量同步 Backup 数据 → DR Volume 保持 Standby 状态
  → 主集群故障 → 激活 DR Volume（standby: false → 读写模式）
  → 应用切换到灾备集群

# 通过 UI 或 API 将卷设为维护模式
kubectl -n longhorn-system edit volume my-volume
# spec.disableFrontend: true  # 启用维护模式

磁盘需要更换或升级 → 将关联的卷设为维护模式
  → 检查数据完整性 → 执行维护操作
  → 退出维护模式 → 恢复正常访问

# 查看 Orphaned Replica
kubectl -n longhorn-system get orphans.longhorn.io
# NAME                                       TYPE
# orphan-20240101-abcdef                     replica

# 全局配置
orphan:
  autoCleanup: true          # 自动清理开关
  cleanupIntervalSeconds: 3600  # 清理间隔（秒）

# 删除单个 orphan
kubectl -n longhorn-system delete orphans.longhorn.io orphan-20240101-abcdef

# 批量清理
kubectl -n longhorn-system delete orphans --all

Longhorn Manager 定期扫描各节点磁盘
  → 发现无关联卷的 Replica 数据 → 创建 Orphan CR
  → AutoCleanup 启用 → 验证 Orphan 无需保留 → 删除 Orphan 数据和 CR
  → 释放磁盘空间

# 全局设置
disk:
  storageReservedPercentage: 30  # 保留空间百分比
  storageOverProvisioningPercentage: 200  # 超分比例
  defaultDiskSoftAntiAffinity: false

# 通过 UI 触发磁盘驱逐
# Longhorn UI → Node → Disks → 选择磁盘 → Evacuate

# 或通过 API
kubectl -n longhorn-system get disks.longhorn.io
kubectl -n longhorn-system delete disks.longhorn.io <disk-name>

磁盘 Critical 或手动触发 Eviction
  → Longhorn Manager 标记该磁盘上的 Replica 为 Eviction
  → 检查其他可用节点和磁盘 → 创建新 Replica
  → 数据同步完成后自动清除原 Replica
  → 磁盘压力解除

# 启用指标
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: longhorn-alerts
  namespace: monitoring
spec:
  groups:
  - name: longhorn
    rules:
    - alert: VolumeDegraded
      expr: longhorn_volume_status{status="degraded"} > 0
      for: 5m
      annotations:
        summary: "卷 XQOPEN $labels.volume XQCLOSE 降级运行"
    - alert: DiskSpaceCritical
      expr: longhorn_disk_storage_available_bytes / longhorn_disk_storage_maximum_bytes < 0.1
      for: 5m
      annotations:
        summary: "磁盘 XQOPEN $labels.disk XQCLOSE 空间不足 10%"
    - alert: BackupFailure
      expr: time() - longhorn_backup_volume_last_sync > 3600
      for: 10m
      annotations:
        summary: "备份 XQOPEN $labels.volume XQCLOSE 超过 1 小时未同步"

1. 升级 Longhorn Manager（Deployment）
2. 升级 Instance Manager（DaemonSet，逐节点）
3. 升级 Engine Image（自动或手动触发）
4. 升级各卷的 Engine 版本

# 1. 更新 Helm 仓库
helm repo update

# 2. 升级 Longhorn
helm upgrade longhorn longhorn/longhorn \
  --namespace longhorn-system \
  --version 1.6.0

# 3. 等待所有 Pod 就绪
kubectl -n longhorn-system rollout status deployment/longhorn-manager
kubectl -n longhorn-system rollout status daemonset/longhorn-csi-plugin

# 4. 卷引擎升级（手动触发）
kubectl -n longhorn-system get engines.longhorn.io
# 检查 image 版本，不是最新则需要更新
kubectl -n longhorn-system edit engines.longhorn.io <engine-name>
# spec.image: longhornio/longhorn-engine:v1.6.0

# 安装 Velero CSI 插件
velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-csi:v0.7.0 \
  --features EnableCSI

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: longhorn-snapshot
driver: driver.longhorn.io
deletionPolicy: Delete

Velero Backup 请求
  → 创建 VolumeSnapshot（调用 CSI Snapshotter）
  → Longhorn 创建 VolumeSnapshot CR → 创建 Snapshot
  → Velero 备份 Pod 资源定义 + VolumeSnapshot 引用
  → 恢复时 VolumeSnapshot → Longhorn 从 Snapshot 创建新卷

# 使用 Longhorn Backup CRD
apiVersion: longhorn.io/v1beta2
kind: Backup
metadata:
  name: my-backup
  namespace: longhorn-system
spec:
  volumeName: "my-volume"
  snapshotName: "my-snapshot"
  labels:
    app: "my-app"
    backup-type: "weekly"

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: my-volume
  namespace: longhorn-system
spec:
  recurringJobs:
  - name: snapshot-5m
    task: "snapshot"
    cron: "*/5 * * * *"
    retain: 12
  - name: backup-daily
    task: "backup"
    cron: "0 2 * * *"
    retain: 30
  - name: backup-weekly
    task: "backup"
    cron: "0 3 * * 0"
    retain: 52

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-autobackup
provisioner: driver.longhorn.io
parameters:
  recurringJobs: '[{"name":"snap","task":"snapshot","cron":"*/5 * * * *","retain":12}]'

apiVersion: longhorn.io/v1beta2
kind: RecurringJob
metadata:
  name: global-snapshot
  namespace: longhorn-system
spec:
  task: "snapshot"
  cron: "0 */6 * * *"
  retain: 4
  groups:
  - default  # 应用到所有 default group 的卷

网络分区发生

Engine 侧：
  → Replica I/O 超时（30秒）
  → Engine 标记 Replica 为 Faulted（Non-responding）
  → 从 Replica 集合中移除 → 继续在剩余 Healthy Replica 上写入
  → 只有 1 个 Healthy Replica 时，Volume 降级为 Degraded

Replica 侧（被隔离的节点）：
  → Replica 检测不到 Engine 的心跳
  → Replica 继续等待并维护本地数据
  → 超时（默认 60 秒）后 Replica 标记自身为 Error

网络恢复后：
  → Engine 发现 Replica 重新可访问
  → 检查 Replica 的 Revision Counter → 判断数据状态
  → 如果 Replica 落后 → 触发增量重建
  → Replica 同步完成后重新加入集合

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-encrypted
provisioner: driver.longhorn.io
parameters:
  encrypted: "true"
  numberOfReplicas: "3"

# 加密 Secret 自动创建
apiVersion: v1
kind: Secret
metadata:
  name: longhorn-volume-<volume-uid>-passphrase
  namespace: longhorn-system
type: Opaque
stringData:
  CRYPTO_KEY_VALUE: "<auto-generated-key>"
  CRYPTO_KEY_CIPHER: "aes-xts-plain64"
  CRYPTO_KEY_HASH: "sha256"
  CRYPTO_KEY_SIZE: "256"
  CRYPTO_PBKDF: "argon2i"

Pod → 文件系统（ext4/xfs） → dm-crypt（加密设备映射）
  → Longhorn 块设备 → Replica（加密后存储）

# 节点磁盘配置（通过 UI 或 CRD）
apiVersion: longhorn.io/v1beta2
kind: Node
metadata:
  name: worker-1
  namespace: longhorn-system
spec:
  disks:
  - path: "/var/lib/longhorn"
    allowScheduling: true
    storageReserved: 10737418240  # 10GB 保留
    tags:
    - "ssd"
    - "fast"
  - path: "/mnt/extra-ssd"
    allowScheduling: true
    storageReserved: 5368709120   # 5GB 保留
    tags:
    - "ssd"
    - "slow"
  - path: "/mnt/hdd-storage"
    allowScheduling: false
    tags:
    - "hdd"
    - "cold"

# StorageClass 通过标签选择磁盘
parameters:
  diskSelector: "ssd,fast"

# 全局设置
storageOverProvisioningPercentage: 200  # 允许 2 倍超分
storageMinimalAvailablePercentage: 25   # 最少保留 25% 可用空间

# 添加新磁盘到节点
# Longhorn UI → Node → worker-1 → Edit Disks → Add Disk
# 或直接修改 Node CRD

# 移除磁盘（需先驱逐数据）
# Node → Disks → 选择磁盘 → Evacuate → 完成后移除

# Global Settings
spdk:
  enable: true

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: longhorn-performance
provisioner: driver.longhorn.io
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  # 数据冗余
  numberOfReplicas: "3"                    # 副本数（1-3）
  replicaAutoBalance: "best-effort"        # 自动平衡

  # 数据本地性
  dataLocality: "best-effort"              # 本地性策略

  # 磁盘选择
  diskSelector: "ssd,nvme"                 # 磁盘标签
  nodeSelector: "storage-node"             # 节点标签

  # 文件系统
  fsType: "ext4"                           # ext4 / xfs
  mkfsParams: "-I 256 -O ^has_journal"    # mkfs 参数

  # 备份
  fromBackup: ""                           # 从备份恢复

  # 快照和备份
  staleReplicaTimeout: "30"                # 过期副本超时（分钟）
  recurringJobs: '[{"name":"snap","task":"snapshot","cron":"*/5 * * * *","retain":12}]'

  # 加密
  encrypted: "false"                       # 是否加密

  # 迁移设置
  migrating: "false"

# 创建克隆卷
apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: cloned-volume
  namespace: longhorn-system
spec:
  fromVolume: "source-volume"  # 源卷
  numberOfReplicas: 3
  size: "10737418240"          # 与源卷大小一致

# or via Longhorn UI
# Volume → 选择源卷 → Clone → 输入新卷名 → 确认

# 导出卷为原始块设备文件
kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80
curl -X POST http://localhost:8080/v1/volumes/my-volume?action=snapshotCreate
curl -X POST http://localhost:8080/v1/volumes/my-volume?action=snapshotBackup \
  -H "Content-Type: application/json" \
  -d '{"name":"export-snap"}'

# UI 操作：Backup → 选择 Backup → Restore as new Volume
# 或 CLI（通过 Longhorn Manager API）
curl -X POST http://localhost:8080/v1/volumes \
  -H "Content-Type: application/json" \
  -d '{"name":"restored-volume","fromBackup":"s3://bucket/backup-url"}'

global:
  # 存储配置
  defaultSettings:
    defaultReplicaCount: "3"
    storageOverProvisioningPercentage: "200"
    storageMinimalAvailablePercentage: "10"
    replicaAutoBalance: "best-effort"
    allowVolumeCreationWithDegradedAvailability: "false"
    
    # 调度
    replicaNodeSoftAntiAffinity: "ignore"
    replicaDiskSoftAntiAffinity: "ignore"
    
    # 快照
    snapshotDataIntegrity: "fast-check"
    recurringJobs: '[{"name":"snapshot-hourly","task":"snapshot","cron":"0 * * * *","retain":24}]'
    
    # 性能
    engineReplicaTimeout: "30"
    concurrentAutomaticEngineUpgradePerNodeLimit: "1"

可用容量 = (Σ 节点存储容量) × (storageOverProvisioningPercentage / 100)
实际可用 = 可用容量 - (Σ 保留空间)

示例（3 节点，各 1TB SSD，超分 200%，保留 10%）：
  总容量 = 3TB
  超分后 = 6TB
  保留空间 = 600GB
  建议卷总量 = 5.4TB

kubectl describe pod <pod-name>
# Events 部分显示挂载失败原因

kubectl get pvc
kubectl get pv
kubectl get volumes.longhorn.io -n longhorn-system
# 确认卷状态是否为 Healthy

kubectl -n longhorn-system get pods | grep csi
# 确认 csi-provisioner / csi-attacher / csi-plugin 均为 Running
kubectl -n longhorn-system logs daemonset/longhorn-csi-plugin --tail=50

kubectl -n longhorn-system get instancemanagers
# 确认 Engine 和 Replica 的实例管理器正常
kubectl -n longhorn-system logs -l app=longhorn-instance-manager --tail=50

kubectl -n longhorn-system logs deployment/longhorn-manager --tail=100
# 搜索卷相关错误

# 紧急恢复：如果 Engine 卡死，可尝试强制重新挂载
kubectl -n longhorn-system annotate volume <volume> longhorn.io/force-detach=true
kubectl delete pod <pod-name>  # 触发重新挂载