在 Istio 上安装

在 Istio 服务网格上安装 Loki 时，您必须完成一些额外步骤。如果没有这些步骤，ingester、querier 等可能会启动，但您会看到以下日志

loki level=debug ts=2021-11-24T11:33:37.352544925Z caller=broadcast.go:48 msg="Invalidating forwarded broadcast" key=collectors/distributor version=123 oldVersion=122 content=[loki-distributor-59c4896444-t9t6g[] oldContent=[loki-distributor-59c4896444-t9t6g[]

这意味着 Pod 未能加入环 (ring)。

如果您尝试将 loki 添加到 Grafana 数据源，您会看到类似（empty ring）的日志

loki level=warn ts=2021-11-24T08:02:42.08262122Z caller=logging.go:72 traceID=3fc821042d8ada1a orgID=fake msg="GET /loki/api/v1/labels?end=1637740962079859431&start=1637740361925000000 (500) 97.4µs Response: \"empty ring\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3fc821042d8ada1a:1feed8872deea75c:1180f95a8235bb6c:0; "

在 Loki 运行的命名空间上启用 istio-injection 后，您还需要修改 Loki 服务的配置。由于 Istio 不允许 Pod 使用 IP 地址解析另一个 Pod，您还必须修改 memberlist 服务。

所需更改

Query frontend 服务

对 Loki Query Frontend 服务的配置文件进行以下修改。

将 grpc 端口的名称更改为 grpclb。这由依赖 SRV 记录的 grpc 负载均衡策略使用。否则，querier 将无法访问 query-frontend。请参见 https://github.com/grafana/loki/blob/0116aa61c86fa983ddcbbd5e30a2141d2e89081a/production/ksonnet/loki/common.libsonnet#L19 和 https://grpc.github.io/grpc/core/md_doc_load-balancing.html
将 grpclb 的 appProtocol 设置为 tcp
将 publishNotReadyAddresses 设置为 true

apiVersion: v1
kind: Service
metadata:
  labels:
    app: loki-query-frontend
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-query-frontend
spec:
  ports:
  - appProtocol: http
    name: http
    port: 3100
    protocol: TCP
    targetPort: http
  - appProtocol: tcp
    name: grpclb
    port: 9095
    protocol: TCP
    targetPort: grpc
  publishNotReadyAddresses: true
  selector:
    app: loki-query-frontend
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-query-frontend
  type: ClusterIP

Querier 服务

对 Loki Querier 服务的配置文件进行以下修改。

将 grpc 服务的 appProtocol 设置为 tcp

apiVersion: v1
kind: Service
metadata:
  labels:
    app: loki-querier
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-querier
  name: loki-querier
  namespace: observability
spec:
  ports:
  - appProtocol: http
    name: http
    port: 3100
    protocol: TCP
    targetPort: http
  - appProtocol: tcp
    name: grpc
    port: 9095
    protocol: TCP
    targetPort: grpc
  selector:
    app: loki-querier
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-querier
  type: ClusterIP

Ingester 服务和 Ingester Headless 服务

对 Loki Query Ingester 和 Ingester Headless 服务的配置文件进行以下修改。

将 grpc 端口的 appProtocol 设置为 tcp

apiVersion: v1
kind: Service
metadata:
  labels:
    app: loki-ingester-(headless)
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-ingester
  name: loki-ingester-headless
spec:  
  clusterIP: None (if headless)
  ports:
  - name: http
    port: 3100
    protocol: TCP
    targetPort: http
  - appProtocol: tcp
    name: grpc
    port: 9095
    protocol: TCP
    targetPort: grpc
  selector:
    app: loki-ingester
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-ingester
  type: ClusterIP

Distributor 服务

对 Loki Distributor 服务的配置文件进行以下修改。

将 grpc 端口的 appProtocol 设置为 tcp

apiVersion: v1
kind: Service
metadata:
  labels:
    app: loki-distributor
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-distributor
spec:
  ports:
  - name: http
    port: 3100
    protocol: TCP
    targetPort: http
  - name: grpc
    port: 9095
    protocol: TCP
    targetPort: grpc
    appProtocol: tcp
  selector:
    app: loki-distributor
    app.kubernetes.io/instance: observability
    app.kubernetes.io/name: loki-distributor
  sessionAffinity: None
  type: ClusterIP

Memberlist 服务

对 Memberlist 服务的配置文件进行以下修改。

将 http 端口的 appProtocol 设置为 tcp

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: observability
  name: loki-memberlist
  namespace: observability
spec:
  clusterIP: None
  ports:
    - name: http
      port: 7946
      protocol: TCP
      targetPort: 7946
      appProtocol: tcp
  selector:
    app.kubernetes.io/instance: observability
    app.kubernetes.io/part-of: memberlist