NFS on Kubernetes 面试题库

graph LR
    Client["NFS Client<br/>Read Request → rsize<br/>Write Request → wsize<br/>Transport → nconnect<br/>Mount Options: rsize=1048576,wsize=1048576,<br/>nconnect=8,async,noatime,hard,intr"]

吞吐量分析（1 GbE 网络，NFSv4）：

  rsize=65536（64KB）:
  ──> 每 MB 需 16 次 RPC Read
  ──> 单线程读吞吐约 60 MB/s（受 RPC 延迟限制）

  rsize=1048576（1MB）:
  ──> 每 MB 仅需 1 次 RPC Read
  ──> 单线程读吞吐约 110 MB/s

# 挂载示例：最大化性能
mount -t nfs4 -o rsize=1048576,wsize=1048576,nconnect=8,async,noatime,hard  \
  192.168.1.100:/exports/data /mnt/nfs

graph TD
    PVC["PVC: my-pvc (10Gi)"]
    SC["StorageClass: nfs-client<br/>provisioner: k8s-sigs.io/nfs-subdir-external-provisioner<br/>archiveOnDelete: true<br/>pathPattern: ${PVC.namespace}-${PVC.name}"]
    PVC --> SC
    SC --> Provisioner
    subgraph Provisioner["NFS Subdir External Provisioner Pod"]
        Step1["1. Watch PVC 创建事件"]
        Step2["2. NFS Mount Server:/nfs-root"]
        Step3["3. mkdir ${namespace}-${pvcName}"]
        Step4["4. chmod 777"]
        Step5["5. 创建 PV 对象"]
    end
    Provisioner --> NFSServer["NFS Server<br/>/exports/<br/>pvc-ns1-pvc1/<br/>pvc-ns2-pvc2/"]

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
parameters:
  archiveOnDelete: "true"      # 删除 PVC 时归档子目录（非直接删除）
  pathPattern: "${.PVC.namespace}-${.PVC.name}"
  onDelete: "delete"           # delete | retain | archive
mountOptions:
  - nfsvers=4.1
  - rsize=1048576
  - wsize=1048576
  - nconnect=4
  - noatime
reclaimPolicy: Delete
volumeBindingMode: Immediate

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
spec:
  replicas: 1
  template:
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
            - name: NFS_SERVER
              value: 192.168.1.100
            - name: NFS_PATH
              value: /exports/k8s-pv
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.1.100
            path: /exports/k8s-pv

graph TD
    subgraph Controller["CSI Controller (Deployment)"]
        CtrlOps["- CreateVolume / DeleteVolume<br/>- ControllerPublishVolume"]
    end
    subgraph NodePlugin["CSI Node Plugin (DaemonSet)"]
        NodeOps["- NodeStageVolume<br/>- NodePublishVolume<br/>- 在目标节点上挂载 NFS"]
    end
    Controller --- NodePlugin

graph TD
    Client["NFS Client (v3/v4/v4.1)"]
    Client --> Ganesha
    subgraph Ganesha["NFS-Ganesha Server"]
        FSAL["FSAL (File System Abstraction Layer)"]
        FSAL_VFS["FSAL VFS"]
        FSAL_CEPH["FSAL CEPH"]
        FSAL_RGW["FSAL RGW"]
        FSAL --> FSAL_VFS
        FSAL --> FSAL_CEPH
        FSAL --> FSAL_RGW
    end
    FSAL_VFS --> LocalFS["本地文件系统"]
    FSAL_CEPH --> CephFS["CephFS"]
    FSAL_RGW --> CephRGW["Ceph RGW"]

apiVersion: v1
kind: Pod
metadata:
  name: nfs-ganesha
spec:
  containers:
    - name: ganesha
      image: nfs-ganesha:V5.6
      securityContext:
        privileged: true
      env:
        - name: GANESHA_CONFIGFILE
          value: /etc/ganesha/ganesha.conf
      volumeMounts:
        - name: ganesha-config
          mountPath: /etc/ganesha
        - name: data
          mountPath: /exports
      ports:
        - containerPort: 2049
          name: nfs
        - containerPort: 111
          name: portmapper
        - containerPort: 20048
          name: mountd
  volumes:
    - name: ganesha-config
      configMap:
        name: ganesha-config
    - name: data
      persistentVolumeClaim:
        claimName: ganesha-data
apiVersion: v1
kind: Service
metadata:
  name: nfs-ganesha
spec:
  ports:
    - name: nfs
      port: 2049
    - name: portmapper
      port: 111
    - name: mountd
      port: 20048
  selector:
    app: nfs-ganesha

NFSv4 Client                          NFSv4 Server
    │                                        │
    │  OPEN (with OPEN4_SHARE_ACCESS_WRITE)  │
    │──────────────────────────────────────>│
    │         stateid = 0x1234               │
    │<──────────────────────────────────────│
    │                                        │
    │  LOCK (stateid=0x1234, offset=0,       │
    │        length=0, type=WRITE_LT)        │
    │──────────────────────────────────────>│
    │         lock_stateid = 0x5678          │
    │<──────────────────────────────────────│
    │                                        │
    │  .... do work with lock held ....      │
    │                                        │
    │  LOCKU (lock_stateid=0x5678)           │
    │──────────────────────────────────────>│
    │                                        │
    │  CLOSE (stateid=0x1234)                │
    │──────────────────────────────────────>│

mountOptions:
  - nfsvers=4.1    # 使用 NFSv4，锁与 Client ID 绑定
  - hard           # 硬挂载，防止 Server 重启后文件句柄失效
  - intr           # 允许中断等待锁的操作

# 在节点上确保 NFS 客户端工具已安装
apt-get install -y nfs-common  # Debian/Ubuntu
yum install -y nfs-utils       # RHEL/CentOS

import fcntl

fd = open("/mnt/data/shared.log", "a")
fcntl.flock(fd, fcntl.LOCK_EX)  # 获取排他锁
# ... write ...
fcntl.flock(fd, fcntl.LOCK_UN)

Step 1: Kerberos KDC 配置
  创建 NFS Server 和 Client 的 Principal：
    addprinc -randkey nfs/nfs-server.example.com@EXAMPLE.COM
    addprinc -randkey nfs/nfs-client.example.com@EXAMPLE.COM
  keytab 导出：
    kadmin: ktadd -k /etc/krb5.keytab nfs/nfs-server.example.com

Step 2: NFS Server 配置 (/etc/exports)：
  /exports/data  gss/krb5p(rw,sync,no_subtree_check,no_root_squash)

Step 3: NFS Client 挂载：
  mount -t nfs4 -o sec=krb5p nfs-server.example.com:/exports/data /mnt/nfs

Step 4: K8s 场景配置 CSI Driver：
  apiVersion: v1
  kind: Secret
  metadata:
    name: nfs-krb5-secret
  type: kubernetes.io/nfs
  data:
    krb5.conf: <base64-encoded>
    krb5.keytab: <base64-encoded>

# /etc/exports 示例
/exports/data 10.0.0.0/8(rw,sync,no_root_squash,no_subtree_check)
/exports/secure 10.0.1.0/24(rw,sync,root_squash,sec=krb5p)

graph TD
    Client["NFS Clients"]
    Client --> Ganesha1["NFS-Ganesha-1<br/>FSAL_RGW"]
    Client --> Ganesha2["NFS-Ganesha-2<br/>FSAL_RGW"]
    Ganesha1 --> RGW["Ceph RGW (S3 API)<br/>(Active-Active)"]
    Ganesha2 --> RGW

graph TD
    subgraph Node1["Node-1 (Active)"]
        NFS1["NFS Server<br/>Virtual IP: 10.0.0.100<br/>DRBD Primary<br/>/exports/data (DRBD mount)"]
    end
    subgraph Node2["Node-2 (Passive)"]
        NFS2["DRBD Secondary<br/>Pacemaker 待命"]
    end
    Node1 -->|"DRBD Sync"| Node2

# 方案三：StatefulSet + 共享存储后端
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nfs-server
spec:
  serviceName: nfs-server
  replicas: 1  # 单副本 + 快速自愈
  template:
    spec:
      containers:
        - name: nfs-server
          image: gcr.io/google_containers/volume-nfs:0.8
          ports:
            - containerPort: 2049
            - containerPort: 111
            - containerPort: 20048
          securityContext:
            privileged: true
          volumeMounts:
            - name: data
              mountPath: /exports
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: nfs-data-pvc

需要 Shared Storage（RWX）？
├── 已有 NFS 基础设施？
│   └── 使用 NFS CSI Driver / Subdir Provisioner
├── 已有 Ceph 集群？
│   └── 使用 CephFS
├── 纯 K8s 环境，运维能力有限？
│   └── 使用 Longhorn（一键部署，K8s 原生）
├── 数据量巨大（PB 级），追求低成本？
│   └── 使用 JuiceFS（S3 后端成本低）
├── 数据库类工作负载（要求强一致性 + RWO）？
│   └── 使用 Longhorn 或 Ceph RBD
└── 传统企业环境，运维团队熟悉 NFS？
    └── 部署 NFS-Ganesha 高可用方案

apiVersion: v1
kind: Pod
metadata:
  name: nfs-server
spec:
  containers:
    - name: nfs-server
      image: k8s.gcr.io/volume-nfs:0.8
      securityContext:
        privileged: true            # NFS 需要内核模块 nfsd
        capabilities:
          add:
            - SYS_ADMIN            # mount 操作
            - SYS_MODULE           # 加载 nfsd 内核模块
            - SYS_RAWIO            # rpcbind 端口绑定(<1024)
      volumeMounts:
        - name: exports
          mountPath: /exports
        - name: modules
          mountPath: /lib/modules
          readOnly: true
  volumes:
    - name: exports
      persistentVolumeClaim:
        claimName: nfs-exports
    - name: modules
      hostPath:
        path: /lib/modules

# 场景：多个不同 Namespace 的 Pod 共享 NFS PVC
# 问题：不同 Pod 的 UID 可能不同，导致文件权限冲突
# 解决：使用 all_squash + 固定的 anonuid/anongid

# /etc/exports：
/exports/k8s  *(rw,sync,all_squash,anonuid=1000,anongid=1000)

# K8s Pod 中使用 fsGroup 统一组权限
apiVersion: v1
kind: Pod
spec:
  securityContext:
    fsGroup: 1000
    fsGroupChangePolicy: "OnRootMismatch"
  containers:
    - name: app
      volumeMounts:
        - name: nfs-vol
          mountPath: /data

# /etc/sysctl.d/99-nfs-performance.conf

# NFS Server 线程数（默认 8）
# 建议：CPU 核数 * 1–2，但不超过 128
# 查看当前值：cat /proc/fs/nfsd/threads
echo 64 > /proc/fs/nfsd/threads

# TCP 发送/接收缓冲区
net.core.rmem_max = 134217728      # 128 MB
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 65536 134217728

# 网络积压队列
net.core.netdev_max_backlog = 5000
net.core.somaxconn = 1024

# 针对大量小文件的优化
vm.dirty_ratio = 30                # 脏页最大比例（默认 20%）
vm.dirty_background_ratio = 5      # 后台刷新阈值

# NFS Server 写延迟优化
fs.nfs.nfs_congestion_kb = 262144 # 256 MB 拥塞控制阈值

# 针对 NFS 场景优化
mkfs.xfs -f -i maxpct=5 -m crc=1,reflink=1,rmapbt=1 \
  -d agcount=16,swidth=256k,sunit=64k \
  /dev/sdb1

# 挂载选项
mount -o noatime,nodiratime,logbufs=8,logbsize=256k,allocsize=256m /dev/sdb1 /exports

# /etc/ganesha/ganesha.conf
NFS_CORE_PARAM {
    # I/O 工作线程数
    Nb_Worker = 64;
    # NFS 协议块大小
    NFS_Protocols = 4.1;
    # 启用 pNFS
    pNFS {
        Enable = true;
        # MDS 和 DS 线程分离
        MDS_Threads = 16;
        DS_Threads = 8;
    }
}
EXPORT {
    Export_Id = 1;
    Path = "/exports/data";
    FSAL { Name = VFS; }
    # 访问缓存调优
    Attr_Expiration_Time = 60;      # 属性缓存 TTL (秒)
    Max_Offset_Set_Size = 1048576;  # READDIR 缓存条目数
}

Pod 启动 Hang / I/O 超时
│
├── 1. 检查是否为 NFS 相关
│   kubectl describe pod <pod-name>
│   查看 Events 是否包含 "mount.nfs" 相关错误
│
├── 2. 检查 NFS Server 可达性
│   kubectl exec -it <pod> -- sh -c "nc -zv <nfs-server> 2049"
│   或从节点执行：
│   showmount -e <nfs-server>
│   rpcinfo -p <nfs-server>
│
├── 3. 检查 NFS 挂载状态
│   在节点上：mount | grep nfs
│   查看是否为 "hard" 挂载且 Server 无响应 → Hang
│   查看是否为 "soft" 挂载且超时 → I/O Error
│
├── 4. 检查 DNS 解析
│   NFS Server 地址是否可解析（非 IP 场景）
│
└── 5. 检查挂载选项冲突
    mountOptions 中 nfsvers 与 Server 支持的版本是否匹配

#!/bin/bash
# 检测节点上异常的 NFS 挂载点
HUNG_MOUNTS=$(findmnt -t nfs4,nfs -o TARGET -n)

for mount_point in $HUNG_MOUNTS; do
  if ! timeout 5 ls "$mount_point" > /dev/null 2>&1; then
    echo "WARN: NFS mount $mount_point is hung"
    umount -f -l "$mount_point"  # lazy unmount
  fi
done

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-nfs-client
spec:
  podSelector:
    matchLabels:
      app: nfs-server
  ingress:
    - from:
        - namespaceSelector: {}
      ports:
        - port: 2049
          protocol: TCP

# /etc/sysconfig/nfs (RHEL/CentOS)
MOUNTD_PORT=20048
STATD_PORT=20046
LOCKD_TCPPORT=20047
LOCKD_UDPPORT=20047

# ganesha.conf
NFS_CORE_PARAM {
    NFS_Port = 2049;
    # NFSv3 辅助端口
    mountd_port = 20048;
    # 禁用 NFSv3（仅需 2049）
    Protocols = 4;
}

# NFS Server Prometheus Exporter 核心指标

# 1. RPC 操作统计（按 NFS 操作类型分组）
nfsd_rpc_operations_total{operation="READ"}       # READ RPC 调用总数
nfsd_rpc_operations_total{operation="WRITE"}      # WRITE RPC 调用总数
nfsd_rpc_operations_total{operation="COMMIT"}     # COMMIT 调用总数
nfsd_rpc_bytes_total{operation="READ"}            # 读取字节总数(Counter)
nfsd_rpc_bytes_total{operation="WRITE"}           # 写入字节总数(Counter)

# 2. 延迟与线程
nfsd_rpc_latency_seconds{operation="READ",quantile="0.99"}  # P99 读延迟
nfsd_thread_usage_ratio                              # nfsd 线程占用率
nfsd_pool_threads                                    # nfsd 线程数量

# 3. 缓存与 I/O
nfsd_read_ahead_kb                                   # 预读缓存大小
nfsd_write_pending_mb                                # 待刷写数据量

# 4. 错误统计
nfsd_reply_cache_hits_total                          # Reply Cache 命中
nfsd_reply_cache_misses_total                        # Reply Cache 未命中
nfsd_iostats_errors_total                            # I/O 错误总数

# nfsiostat 实时监控
nfsiostat 1 /mnt/nfs

# 关键输出指标：
#   ops/s：每秒操作数
#   rpc bklog：RPC 积压队列长度（>0 表示拥塞）
#   retrans/s：重传次数/秒（>0 表示网络丢包或 Server 过载）

groups:
  - name: nfs
    rules:
      - alert: NFSDThreadSaturation
        expr: nfsd_thread_usage_ratio > 0.9
        for: 5m
        annotations:
          summary: "NFS Server nfsd 线程使用率超过 90%"

      - alert: NFSWriteCommitRate
        expr: rate(nfsd_rpc_operations_total{operation="COMMIT"}[5m])
              / rate(nfsd_rpc_operations_total{operation="WRITE"}[5m]) > 0.3
        for: 10m
        annotations:
          summary: "NFS COMMIT/WRITE 比率过高，Client 可能未使用 async 挂载"

      - alert: NFSRPCBacklog
        expr: nfsd_rpc_backlog > 100
        for: 1m
        annotations:
          summary: "NFS Server RPC 积压队列 >100，可能存在性能瓶颈"

      - alert: NFSRetransmission
        expr: rate(nfsd_reply_cache_misses_total[1m]) > 10
        for: 3m
        annotations:
          summary: "NFS 重传率过高，网络或 Server 负载异常"

# 查询 Ganesha 导出状态
dbus-send --system --print-reply --dest=org.ganesha.nfsd \
  /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.ShowExports

# 查询客户端统计
dbus-send --system --print-reply --dest=org.ganesha.nfsd \
  /org/ganesha/nfsd/ClientMgr org.ganesha.nfsd.clientmgr.ShowClients

#!/bin/bash
# ZFS 自动快照脚本 (sanoid/syncoid 风格)
ZFS_POOL="tank/nfs"
SNAPSHOT_NAME="auto-$(date +%Y%m%d-%H%M)"

zfs snapshot ${ZFS_POOL}@${SNAPSHOT_NAME}

# 保留策略
# 保留最近 7 天的每日快照
zfs list -t snapshot -o name -s creation \
  | grep "${ZFS_POOL}@auto-" \
  | head -n -7 \
  | xargs -r -n1 zfs destroy

# 增量备份到远程 NFS 目标
rsync -avz --delete --link-dest=/backup/previous \
  /exports/data/ \
  backup-server:/backup/daily-$(date +%Y%m%d)/

# 使用 NFS CSI Driver 的 VolumeSnapshot（需要底层支持）
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: nfs-snapshot-20260126
spec:
  volumeSnapshotClassName: nfs-snapshot-class
  source:
    persistentVolumeClaimName: nfs-pvc

mount -t nfs4 -o \
  rsize=1048576,wsize=1048576,\
  nconnect=16,\
  async,noatime,hard,\
  timeo=600,retrans=5 \
  remote-nfs.example.com:/exports/data /mnt/nfs-remote

graph TD
    subgraph Client["NFSv4.1 pNFS Client"]
        LayoutDriver["Layout Driver<br/>(metadata)"]
        LayoutDev1["Layout Device 1"]
        LayoutDev2["Layout Device 2"]
    end
    Client --> MDS
    LayoutDev1 -->|"direct I/O"| DS1
    LayoutDev2 -->|"direct I/O"| DS2
    subgraph Server["Server Side"]
        MDS["MDS (Metadata Server)<br/>NFS-Ganesha (FSAL_VFS/CEPH)"]
        DS1["DS1 (Data Server)<br/>/data/chunk/0-1023"]
        DS2["DS2 (Data Server)<br/>/data/chunk/1024..."]
    end
    MDS --> DS1
    MDS --> DS2

# /etc/ganesha/ganesha.conf
NFS_CORE_PARAM {
    NFS_Protocols = 4.1;            # 必须启用 4.1
    pNFS {
        Enable = true;
        MDS_Threads = 16;
        DS_Threads = 8;
    }
}

# MDS 导出配置
EXPORT {
    Export_Id = 1000;
    Path = "/exports/data";
    Pseudo = "/data";
    FSAL { Name = VFS; }

    # pNFS 配置
    PNFS_DS {
        DS1 { Hostname = "ds1.example.com"; Port = 2049; }
        DS2 { Hostname = "ds2.example.com"; Port = 2049; }
        DS3 { Hostname = "ds3.example.com"; Port = 2049; }
    }
    # 条带大小
    PNFS_Stripesize = 4194304;      # 4 MB
}

# 每个 DS 节点配置
EXPORT {
    Export_Id = 1001;
    Path = "/data/stripe-0";
    PNFS_ROLE = DS;                 # 标记为 Data Server
    FSAL { Name = VFS; }
}

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: pnfs-sc
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-ganesha-mds.default.svc.cluster.local
  share: /data
  mountPermissions: "0777"
mountOptions:
  - nfsvers=4.1
  - pnfs           # 启用 pNFS 客户端

# /etc/exports
# 场景 1：单一应用独占 NFS PV
/exports/app1  10.0.0.0/8(rw,sync,no_root_squash,no_subtree_check)

# 场景 2：多租户共享 NFS PV（推荐 all_squash）
/exports/shared  10.0.0.0/8(rw,sync,all_squash,anonuid=1000,anongid=1000,no_subtree_check)

# 场景 3：CSI Node Plugin 需要 root 权限执行 mount
/exports/k8s-csi  10.0.0.0/8(rw,sync,no_root_squash,no_subtree_check)

apiVersion: v1
kind: Pod
spec:
  securityContext:
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1000          # 卷挂载后 chgrp 1000
  containers:
    - name: app
      volumeMounts:
        - name: nfs-vol
          mountPath: /data

graph TD
    A["Pod 用户<br/>UID=1001, GID=2000"] --> B["NFS Client 发送 RPC 认证<br/>AUTH_SYS: UID=1001, GID=2000"]
    B --> C["NFS Server 端 all_squash<br/>映射 UID=1000, GID=1000"]
    C --> D["文件系统层操作实际以<br/>UID=1000, GID=1000 执行"]
    D --> E["Pod 端看到文件属主为<br/>1000:1000"]

# 挂载时调整缓存行为
mount -t nfs4 -o \
  acregmin=3,acregmax=60,     # 文件属性缓存：3s–60s
  acdirmin=30,acdirmax=120,   # 目录属性缓存：30s–120s
  actimeo=10                   # 统一设置所有属性缓存 TTL
  server:/exports /mnt/nfs

# 强一致性要求场景（多 Client 并发读写同一文件）
mount -t nfs4 -o noac,sync server:/exports /mnt/nfs
# noac：禁用属性缓存，每次操作前先 GetAttr
# 代价：元数据操作吞吐下降 50–80%

graph TD
    A["Server 重启前<br/>Client 持有 ClientID=0xAAA<br/>StateID=0xBBB"] --> B["Server 重启<br/>Grace Period 默认 90s"]
    B --> C["Client 发起 RPC<br/>携带旧 ClientID=0xAAA"]
    C -->|"Grace Period 内"| D["返回 NFS4ERR_GRACE<br/>告知 Client 等待"]
    C -->|"超出 Grace Period"| E["返回 NFS4ERR_EXPIRED<br/>ClientID 无法恢复"]
    E --> F["Client 自动处理"]
    F --> F1["销毁所有旧 StateID"]
    F --> F2["重新执行 OPEN<br/>获取新 StateID"]
    F --> F3["重新执行 LOCK<br/>获取新 Lock StateID"]
    F --> F4["应用层捕获 EIO / ENOLCK<br/>并重新获取锁"]

# NFS Server 内核参数
echo 120 > /proc/fs/nfsd/nfsv4leasetime      # 租约时间（秒），默认 90
echo 120 > /proc/fs/nfsd/nfsv4gracetime      # 宽限期（秒），默认 90

mountOptions:
  - hard              # Server 不可达时持续重试（而非直接返回错误）
  - intr              # 允许在硬挂载等待期间被信号中断
  - retrans=5         # 每次重传次数（仅 soft 挂载有效）
  - timeo=600         # 超时时间（0.1s 单位）

// Go 应用示例：NFS 操作的自动重试
func nfsWriteWithRetry(f *os.File, data []byte) error {
    for i := 0; i < 3; i++ {
        _, err := f.Write(data)
        if err == nil {
            return nil
        }
        if errors.Is(err, syscall.EIO) || errors.Is(err, syscall.ESTALE) {
            // NFS 错误，重新打开文件
            f.Close()
            f, err = os.OpenFile(f.Name(), os.O_WRONLY|os.O_APPEND, 0644)
            if err != nil {
                return err
            }
            continue
        }
        return err
    }
    return fmt.Errorf("exhausted retries")
}

graph TD
    Client["NFS Client (v4)"]
    Client --> Ganesha
    subgraph Ganesha["NFS-Ganesha Server"]
        FSAL["FSAL_RGW<br/>File → S3 Object<br/>Dir → S3 Prefix<br/>Lock → RGW Lock"]
    end
    Ganesha --> Librados["librados (C 库)"]
    Librados --> RGW
    subgraph RGW["Ceph RGW (radosgw)"]
        Bucket["Bucket: nfs-export<br/>obj1 | obj2"]
    end
    RGW --> OSD["Ceph OSD Cluster"]

# /etc/ganesha/ganesha.conf
NFS_CORE_PARAM {
    NFS_Protocols = 4;
}

EXPORT {
    Export_Id = 1;
    Path = "/";
    Pseudo = "/nfs-rgw";
    Access_Type = RW;
    Protocols = 4;
    Transports = TCP;
    FSAL {
        Name = RGW;
        User_Id = "nfs-ganesha";
        Secret_Access_Key = "your-ceph-secret-key";
        # Ceph RGW 配置
        Ceph_Conf = "/etc/ceph/ceph.conf";
        # RGW Bucket
        Bucket = "nfs-export";
        # Ceph 集群名称
        Name = "client.nfs-ganesha";
        # 对象前缀（所有 NFS 文件将以该前缀存储）
        Object_Prefix = "nfs/";
    }
}

nfsd_threads = max(
    8,                         # 最小值
    CPU_CORES * 2,             # 有 I/O 等待时
    Peak_Concurrent_RPC / 200  # 按 RPC 并发
)

# Client 端统计
nfsstat -c

# 关注指标：
#   retrans         - RPC 重传次数（累积）
#   authrefrsh      - 认证刷新次数
#   calls           - RPC 调用总数

# 计算重传率：retrans / calls * 100
# 重传率 > 1%：需要排查
# 重传率 > 5%：严重性能问题

# 查看 NFS TCP 连接信息
ss -tipn | grep ':2049'

# 关注：
#   retrans:<n>      - TCP 重传次数（非零需关注）
#   send <bytes>     - 发送缓冲区占比（100% 表示拥塞窗口满）
#   rto:<ms>         - 重传超时（波动说明网络不稳定）

# 抓包分析
tcpdump -i eth0 -w nfs.pcap port 2049
# 后续在 Wireshark 中过滤 nfs && nfs.status != 0 查看错误

# 挂载时指定 timeo 和 retrans
mount -t nfs4 -o timeo=600,retrans=5 server:/exports /mnt/nfs

# timeo: RPC 超时（单位 0.1s）→ 600=60s
# retrans: 超时前重试次数
# 总等待 = timeo * 0.1 * (2^retrans)（指数退避）

# soft vs hard 对重传行为的影响：
#  hard: 无限重试直至 Server 恢复 → 无 I/O 错误但可能 Hang
#  soft: retrans 次后返回 I/O 错误 → 不 Hang 但有数据丢失风险

# 限制从 NFS Server 到特定 Client 的出方向带宽
tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1: classid 1:1 htb rate 1gbit
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 500mbit ceil 1gbit
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 \
  match ip dst 10.0.1.100/32 flowid 1:10

# 在 NFS Server 容器内使用 cgroup v2 限制 I/O
# 需挂载 cgroup2 文件系统
mkdir -p /sys/fs/cgroup/nfs
echo "+io" > /sys/fs/cgroup/cgroup.subtree_control
echo "8:0 wbps=104857600" > /sys/fs/cgroup/nfs/io.max  # 限制写 100 MB/s
echo "8:0 rbps=209715200" > /sys/fs/cgroup/nfs/io.max  # 限制读 200 MB/s

# ganesha.conf (Ganesha v4+)
EXPORT {
    Export_Id = 1;
    Path = "/exports/data";

    # 每个 Client 的最大并发操作数
    Max_Client_Ops = 64;

    # Client 连接节流
    Client {
        # 限制特定 Client
        Clients = 10.0.1.0/24;
        Max_IOPS = 1000;
        Max_Throughput_MB = 100;
    }
}

# 通过 Pod Resource Limits 间接限制 NFS I/O
apiVersion: v1
kind: LimitRange
metadata:
  name: nfs-client-limits
spec:
  limits:
    - type: Container
      max:
        cpu: "2"
        memory: 4Gi
      default:
        cpu: "500m"
        memory: 1Gi

# 启用 NFS Server RPC 日志
echo 1 > /proc/sys/sunrpc/nfsd_debug
echo 1 > /proc/sys/sunrpc/rpc_debug

# 查看日志
journalctl -u nfs-server -f
dmesg | grep nfsd

# 开启审计守护进程
auditctl -a exit,always -F arch=b64 -S open -F dir=/exports

# /etc/ganesha/ganesha.conf
LOG {
    # 系统日志
    Default_Log_Level = EVENT;
    # 审计日志（独立文件）
    Facility {
        Name = FILE;
        Destination = "/var/log/ganesha/audit.log";
        # 日志级别
        Enable = INFO, WARN, EVENT, CRIT;
    }
}

# 每个导出配置审计
EXPORT {
    Export_Id = 1;
    Path = "/exports/data";

    # 记录成功的读/写操作
    Log_Level = FULL_DEBUG;
}

{
  "timestamp": "2026-01-26T10:30:45.123Z",
  "event": "NFS_READ",
  "client_ip": "10.0.1.50",
  "user": "UID:1000",
  "export_id": 1,
  "path": "/exports/data/project/file.txt",
  "handle": "0xabcd1234",
  "offset": 0,
  "length": 4096,
  "latency_us": 230,
  "result": "OK"
}

apiVersion: v1
kind: Pod
metadata:
  name: nfs-ganesha
spec:
  containers:
    - name: nfs-ganesha
      image: nfs-ganesha:v5.6
      volumeMounts:
        - name: audit-log
          mountPath: /var/log/ganesha
    - name: filebeat-sidecar
      image: docker.elastic.co/beats/filebeat:8.11.0
      volumeMounts:
        - name: audit-log
          mountPath: /var/log/ganesha
          readOnly: true
        - name: filebeat-config
          mountPath: /usr/share/filebeat/filebeat.yml
          subPath: filebeat.yml
  volumes:
    - name: audit-log
      emptyDir: {}
    - name: filebeat-config
      configMap:
        name: filebeat-config

# 症状：initContainer 执行 chown -R 1000:1000 /data 返回 EPERM
# 原因：NFS Server 开启了 root_squash

# 方案 A：Server 端预先设置权限
# 在 NFS Server 上：
chown -R 1000:1000 /exports/k8s/pvc-ns-xxx/

# 方案 B：使用 all_squash 统一权限
# NFS Server /etc/exports：
/exports/k8s  *(rw,sync,all_squash,anonuid=1000,anongid=1000)

# K8s 尝试在卷挂载后设置 fsGroup
# NFS Server root_squash 开启时，kubelet（以 root 运行）的 chgrp 被拒绝

# 方案：使用 fsGroupChangePolicy
securityContext:
  fsGroup: 1000
  fsGroupChangePolicy: "OnRootMismatch"  # 仅在根目录组不匹配时尝试

# 共享场景推荐 all_squash + 固定 anonuid/anongid
/exports/k8s-shared 10.0.0.0/8(rw,sync,all_squash,anonuid=1000,anongid=1000)

# 所有 Pod 以同一身份访问，避免 permission denied
# 缺点：所有 Pod 共享相同的文件权限

parameters:
  archiveOnDelete: "true"   # 删除时归档（重命名为 archived-<pvc-name>-<timestamp>）
  onDelete: "delete"        # delete | retain | archive

graph TD
    A["Provisioning"] --> B["Available"]
    B --> C["Bound"]
    C --> D["Released"]
    D -->|"Retain"| F["Failed / Retained"]
    D -->|"Delete"| G["Deleted"]

# 1. 找到 Released 状态的 PV
kubectl get pv | grep Released

# 2. 清理 NFS Server 上的数据
ssh nfs-server "rm -rf /exports/k8s/<pvc-path>"

# 3. 删除 PV 对象
kubectl delete pv <pv-name>

# 4. 或者将 PV 标记为 Available 以重新绑定
kubectl patch pv <pv-name> -p '{"spec":{"claimRef":null}}'

graph TD
    Memory["NFS Server 内存分布"]
    Memory --> A["nfsd 内核线程栈<br/>(nfsd_threads × 8KB) → 通常 < 10 MB"]
    Memory --> B["sunrpc 缓冲区 → 10–100 MB"]
    Memory --> C["Reply Cache<br/>(重复请求去重) → 50–200 MB"]
    Memory --> D["Page Cache<br/>(数据缓存) → 主要内存消耗"]
    Memory --> E["Inode / Dentry Cache<br/>→ 百万文件级别可达数 GB"]

# 监控 Page Cache 回收情况
sar -B 1

# pgscank/s（后台回收）和 pgscand/s（直接回收）
# pgscand/s > 0 时表示内存压力严重

# 缓解方案
sysctl -w vm.dirty_ratio=10     # 降低脏页上限
sysctl -w vm.dirty_background_ratio=3

# Client 端使用 Direct I/O 绕过 Page Cache（特殊场景）
mount -t nfs4 -o rsize=1048576,wsize=1048576,noac server:/exports /mnt/nfs

# 或者在应用层使用 O_DIRECT 打开文件
# （注意：NFS 的 O_DIRECT 支持在内核 5.8+ 才稳定）

# 查看当前 Inode/Dentry Cache
slabtop -o | grep -E 'dentry|inode'

# 手动回收（有性能影响，仅紧急情况）
echo 2 > /proc/sys/vm/drop_caches  # 回收 dentries 和 inodes

NFS Server 目录结构：
/exports/
├── data/          (本地 ext4 — 导出为 /exports/data)
│   ├── shared/    (NFS 挂载：另一个 NFS Server:/shared → /exports/data/shared)
│   │   └── file_c
│   ├── file_a
│   └── file_b

K8s Pod 挂载 server:/exports/data → /mnt

Pod 内访问 /mnt/shared/file_c：
  ├── NFS v3：如果 server /etc/exports 未配置 crossmnt，无法访问嵌套挂载点
  ├── NFS v4：Server 端使用 fsid=0 且 NFSv4 Pseudofilesystem 可访问
  └── K8s CSI：CSI 仅挂载顶层导出，嵌套挂载点可能显示为空目录

# NFS v3 需要显式启用 crossmnt
/exports/data  *(rw,sync,crossmnt,no_subtree_check)

# NFS v4 使用 nohide 确保嵌套挂载点可见
/exports/data  *(rw,sync,nohide,no_subtree_check)

# 不要在 NFS 导出中嵌套其他 NFS 导出
# 每个独立后端使用独立的 StorageClass 和 PVC

# 推荐架构：
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-nfs
spec:
  storageClassName: nfs-data
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 100Gi
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shared-nfs
spec:
  storageClassName: nfs-shared
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 50Gi

# 推荐挂载选项组合
mountOptions:
  - nfsvers=4.1
  - rsize=1048576     # 1 MB
  - wsize=1048576     # 1 MB
  - nconnect=4        # 4 TCP 连接（内核 5.3+）
  - async             # 异步写（除数据库 WAL 外）
  - noatime           # 不更新 atime
  - hard              # 硬挂载，避免静默数据丢失
  - intr              # 允许中断（配合 hard）
  - timeo=600         # 60s 超时

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-production
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs-server.production.svc.cluster.local
  share: /exports/k8s
  mountPermissions: "0777"
mountOptions:
  - nfsvers=4.1
  - rsize=1048576
  - wsize=1048576
  - nconnect=4
  - async
  - noatime
  - hard
  - intr
  - timeo=600
reclaimPolicy: Retain          # 生产环境用 Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false    # NFS 不支持真正 resize