3、服务发现及打标

- targets:
  - localhost:9090
  labels:
    app: prometheus
    job: prometheus
- targets:
  - node1.cce.com:9100
  - node2.cce.com:9100
  labels:
    app: node-expoter
    job: node

global:
  scrape_interval: 15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - target:
rule_files:
scrape_configs:
  - job_name: 'prometheus'
    file_sd_configs:
    - files:
      - targets/prometheus*.yaml # 指定要加载的文件列表，支持通配符
      refresh_interval: 2m # 每个2分钟重新加载一次文件中定义的Target，默认为5m
  - job_name: 'nodes'
    file_sd_configs:
    - files:
      - targets/nodes*.yaml
      refresh_interval: 2m

# 示例：_prometheus._tcp.magedu.com
# dns_sd_config # 配置段
# A list of DNS domain names to be queried.
names:
  [ - <string> ]

# The type of DNS query to perform. One of SRV, A, or AAAA.
[ type: <string> | default = 'SRV' ]

# The port number used if the query type is not SRV.
[ port: <int>]

# The time after which the provided names are refreshed.
[ refresh_interval: <duration> | default = 30s ]

__meta.kubernetes_node_name  # 节点名称
__meta.kubernetes_node_label_<labelname>  # 节点的资源的标签
__meta.kubernetes_node_labelpresent_<labelname> # 节点资源标签是否存在
__meta.kubernetes_node_annotation_<annotation> # 节点的注解
__meta.kubernetes_node_annotationpresen_<annotation> # 节点资源注解是否存在
__meta.kubernetes_node_address_<address_type> # 节点地址是什么类型的地址，是NodeExternalIP还是NodeInternalIP等；

__meta_kubernetes_namespace: The namespace of the pod object.
__meta_kubernetes_pod_name: The name of the pod object.
__meta_kubernetes_pod_ip: The pod IP of the pod object.
__meta_kubernetes_pod_label_<labelname>: Each label from the pod object.
__meta_kubernetes_pod_labelpresent_<labelname>: truefor each label from the pod object.
__meta_kubernetes_pod_annotation_<annotationname>: Each annotation from the pod object.
__meta_kubernetes_pod_annotationpresent_<annotationname>: true for each annotation from the pod object.
__meta_kubernetes_pod_container_init: true if the container is an InitContainer
__meta_kubernetes_pod_container_name: Name of the container the target address points to.
__meta_kubernetes_pod_container_port_name: Name of the container port.
__meta_kubernetes_pod_container_port_number: Number of the container port.
__meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
__meta_kubernetes_pod_ready: Set to true or false for the pod's ready state.
__meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
__meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
__meta_kubernetes_pod_host_ip: The current host IP of the pod object.
__meta_kubernetes_pod_uid: The UID of the pod object.
__meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
__meta_kubernetes_pod_controller_name: Name of the pod controller.

__meta_kubernetes_namespace: The namespace of the service object.
__meta_kubernetes_service_annotation_<annotationname>: Each annotation from the service object.
__meta_kubernetes_service_annotationpresent_<annotationname>: "true" for each annotation of the service object.
__meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
__meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
__meta_kubernetes_service_label_<labelname>: Each label from the service object.
__meta_kubernetes_service_labelpresent_<labelname>: true for each label of the service object.
__meta_kubernetes_service_name: The name of the service object.
__meta_kubernetes_service_port_name: Name of the service port for the target.
__meta_kubernetes_service_port_protocol: Protocol of the service port for the target.
__meta_kubernetes_service_type: The type of the service.

__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
# 对于通过附加在Endpoint资源上端口发现的各target，还有如下meta标签
    __meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
    __meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.    
    __meta_kubernetes_endpoint_ready: Set to true or false for the endpoint's ready state.
    __meta_kubernetes_endpoint_port_name: Name of the endpoint port.
    __meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
    __meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.    
    __meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
# 如果发现的Endpoint资源属于某个Service，则相关Service的元标签也会添加到该Endpoint上；
# 如果发现的Endpoint资源的后段端点是Pod，则相关Pod的元标签也会添加到该Endpoint上；

__meta_kubernetes_namespace: The namespace of the ingress object.
__meta_kubernetes_ingress_name: The name of the ingress object.
__meta_kubernetes_ingress_label_<labelname>: Each label from the ingress object.
__meta_kubernetes_ingress_labelpresent_<labelname>: true for each label from the ingress object.
__meta_kubernetes_ingress_annotation_<annotationname>: Each annotation from the ingress object.
__meta_kubernetes_ingress_annotationpresent_<annotationname>: true for each annotation from the ingress object.
__meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLS config is set. Defaults to http.
__meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.

# The information to access the Kubernetes API.

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

# The Kubernetes role of entities that should be discovered.
# One of endpoints, service, pod, node, or ingress.
role: <string>

# Optional authentication information used to authenticate to the API server.
# Note that `basic_auth` and `authorization` options are
# mutually exclusive.
# password and password_file are mutually exclusive.

# Optional HTTP basic authentication information.
basic_auth:
  [ username: <string> ]
  [ password: <secret> ]
  [ password_file: <string> ]

# Optional the `Authorization` header configuration.
authorization:
  # Sets the authentication type.
  [ type: <string> | default: Bearer ]
  # Sets the credentials. It is mutually exclusive with
  # `credentials_file`.
  [ credentials: <secret> ]
  # Sets the credentials with the credentials read from the configured file.
  # It is mutually exclusive with `credentials`.
  [ credentials_file: <filename> ]

# Optional OAuth 2.0 configuration.
# Cannot be used at the same time as basic_auth or authorization.
oauth2:
  [ <oauth2> ]

# Optional proxy URL.
[ proxy_url: <string> ]

# Configure whether HTTP requests follow HTTP 3xx redirects.
[ follow_redirects: <bool> | default = true ]

# TLS configuration.
tls_config:
  [ <tls_config> ]

# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
  names:
    [ - <string> ]

# The source labels select values from existing labels. Their content is concatenated
# using the configured separator and matched against the configured regular expression
# for the replace, keep, and drop actions.
[ source_labels: '[' <labelname> [, ...] ']' ] # 指定源标签，多个源标签

# Separator placed between concatenated source label values.
[ separator: <string> | default = ; ]  # 

# Label to which the resulting value is written in a replace action.
# It is mandatory for replace actions. Regex capture groups are available.
[ target_label: <labelname> ]

# Regular expression against which the extracted value is matched.
[ regex: <regex> | default = (.*) ]

# Modulus to take of the hash of the source label values.
[ modulus: <int> ]

# Replacement value against which a regex replace is performed if the
# regular expression matches. Regex capture groups are available.
[ replacement: <string> | default = $1 ]

# Action to perform based on regex matching.
[ action: <relabel_action> | default = replace ] # 此配置对标签做何种操作，默认为将source_targets的值替换target_label

<relabel_config>字段用于定义重新标记，其可用取之如下
替换标签值
    replace：首先将source_labels中指定的各标签值进行串联，而后将regex字段中的正则表达式对源标签值进行匹配判定，若匹配，则将target_label字段中指定的标签值替换为replacement字段中的值；
        replacement可按需引用保存在regex中的某个“分组模式”匹配到的值，默认保存整个regex匹配的内容；
        进行值替换时，replacement字段中指定标签的值也支持以分组格式进行引用；
    hashmod：将target_label的值设置为一个hash值，该hash值由modules字段指定的hash模对source_labels上各标签的串联值进行hash计算生成的；
删除指标(该处的每个指标名称对应一个Target)
    keep：regex不能匹配到target上的source_labels上各标签的串联值时，则删除该target；
    drop：regex能匹配到target上的source_labels上各标签的串联值时，则删除该target；
创建或删除标签
    labelmap：将regex对所有的标签名进行匹配判定，而后将匹配到的标签的值赋给replacement字段指定的标签吗之上，通常用于取出匹配的标签名的一部分生成新的标签；
    labeldrop：将regex对所有的标签名进行匹配判定，能够匹配到的标签将从该Target的标签集中删除；
    labelkeep：将regex对所有的标签名进行匹配判定，不能够匹配到的标签将从该Target的标签集中删除；
# 需要注意的是，要确保labeldrop或labelkeep操作后，余下的标签集依然能唯一标示该指标；

  - job_name: 'nodes'
    file_sd_configs:
    - files:
      - targets/node*.yaml
      refresh_interval: 30s
    relabel_configs:
    - source_labels:
      - __scheme__
      - __address__
      - __metrics_path__
      regex: "(http|https)(.*)"
      separator: ""
      target_label: "endpoint"
      replacement: "${1}://${2}"
      action: replace

  - job_name: 'nodes'
    file_sd_configs:
    - files:
      - targets/node*.yaml
      refresh_interval: 30s
    relabel_configs:
    - regex: "^(job|app).*"
      replacement: "${1}_name"
      action: labelmap
    - regex: "^(job|app)$"
      action: labeldrop

  - job_name: 'nodes'
    file_sd_configs:
    - files:
      - targets/node*.yaml
      refresh_interval: 30s
    relabel_configs:
    - source_labels:
      - __address__
      regex: "^node[2-9].*"
      action: drop

# 使用正则表达式来匹配需要删除的Metrics名
    metric_relabel_configs:
    - source_labels:
      - __name__
      regex: '(go_gc_duration_seconds|go_info)'  # 匹配要删除的标签名称，如果有多个正则，用分号隔开`;`
      action: drop
# 将所有Metric的instance标签值复制给cce标签
    metric_relabel_configs:
    - source_labels:
      - instance
      regex: '(.*)'
      replacement: $1
      target_label: cce
      action: replace
# 删除标签
    metric_relabel_configs:
    - source_labels:
      regex: 'instance'
      action: labeldrop