{"id":2054,"date":"2020-06-22T16:04:36","date_gmt":"2020-06-22T13:04:36","guid":{"rendered":"https:\/\/artem.services\/?p=2011"},"modified":"2022-04-09T15:36:16","modified_gmt":"2022-04-09T12:36:16","slug":"2054","status":"publish","type":"post","link":"https:\/\/artem.services\/?p=2054&lang=en","title":{"rendered":"AWS &#8212; EKS Fargate &#8212; Fluentd CloudWatch"},"content":{"rendered":"<p><img loading=\"lazy\" class=\"aligncenter size-full wp-image-214\" src=\"https:\/\/artem.services\/wp-content\/uploads\/2018\/11\/AWS-Logo.png\" alt=\"\" width=\"975\" height=\"450\" srcset=\"https:\/\/artem.services\/wp-content\/uploads\/2018\/11\/AWS-Logo.png 975w, https:\/\/artem.services\/wp-content\/uploads\/2018\/11\/AWS-Logo-300x138.png 300w, https:\/\/artem.services\/wp-content\/uploads\/2018\/11\/AWS-Logo-768x354.png 768w, https:\/\/artem.services\/wp-content\/uploads\/2018\/11\/AWS-Logo-954x440.png 954w\" sizes=\"(max-width: 975px) 100vw, 975px\" \/><\/p>\n<p>At the time of writing, <strong>EKS Fargate<\/strong> does not support a driver log for recording to <strong>CloudWatch<\/strong>. The only option is to use <strong>Sidecar<\/strong><\/p>\n<p>Let&#8217;s create a <strong>ConfigMap<\/strong>, in which we indicate the name of the <strong>EKS<\/strong> cluster, <strong>region<\/strong> and <strong>namespace<\/strong>:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\nkubectl create configmap cluster-info \\\r\n--from-literal=cluster.name=YOUR_EKS_CLUSTER_NAME \\\r\n--from-literal=logs.region=YOUR_EKS_CLUSTER_REGION -n KUBERNETES_NAMESPACE\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>Next, let&#8217;s create a service account and a <strong>ConfigMap<\/strong> with a configuration file for <strong>Fluentd<\/strong>. To do this, copy the text below and save it as &quot;<strong>fluentd.yaml<\/strong>&quot;<\/p>\n<pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\r\napiVersion: v1\r\nkind: ServiceAccount\r\nmetadata:\r\n  name: fluentd\r\n  namespace: {{NAMESPACE}}\r\n---\r\napiVersion: rbac.authorization.k8s.io\/v1\r\nkind: ClusterRole\r\nmetadata:\r\n  name: fluentd-role\r\nrules:\r\n  - apiGroups: [&quot;&quot;]\r\n    resources:\r\n      - namespaces\r\n      - pods\r\n      - pods\/logs\r\n    verbs: [&quot;get&quot;, &quot;list&quot;, &quot;watch&quot;]\r\n---\r\napiVersion: rbac.authorization.k8s.io\/v1\r\nkind: ClusterRoleBinding\r\nmetadata:\r\n  name: fluentd-role-binding\r\nroleRef:\r\n  apiGroup: rbac.authorization.k8s.io\r\n  kind: ClusterRole\r\n  name: fluentd-role\r\nsubjects:\r\n  - kind: ServiceAccount\r\n    name: fluentd\r\n    namespace: {{NAMESPACE}}\r\n---\r\napiVersion: v1\r\nkind: ConfigMap\r\nmetadata:\r\n  name: fluentd-config\r\n  namespace: {{NAMESPACE}}\r\n  labels:\r\n    k8s-app: fluentd-cloudwatch\r\ndata:\r\n  fluent.conf: |\r\n    @include containers.conf\r\n \r\n    &lt;match fluent.**&gt;\r\n      @type null\r\n    &lt;\/match&gt;\r\n  containers.conf: |\r\n    &lt;source&gt;\r\n      @type tail\r\n      @id in_tail_container_logs\r\n      @label @containers\r\n      path \/var\/log\/application.log\r\n      pos_file \/var\/log\/fluentd-containers.log.pos\r\n      tag *\r\n      read_from_head true\r\n      &lt;parse&gt;\r\n        @type none\r\n        time_format %Y-%m-%dT%H:%M:%S.%NZ\r\n      &lt;\/parse&gt;\r\n    &lt;\/source&gt;\r\n \r\n    &lt;label @containers&gt;\r\n      &lt;filter **&gt;\r\n        @type kubernetes_metadata\r\n        @id filter_kube_metadata\r\n      &lt;\/filter&gt;\r\n \r\n      &lt;filter **&gt;\r\n        @type record_transformer\r\n        @id filter_containers_stream_transformer\r\n        &lt;record&gt;\r\n          stream_name &quot;#{ENV.fetch(&#039;HOSTNAME&#039;)}&quot;\r\n        &lt;\/record&gt;\r\n      &lt;\/filter&gt;\r\n \r\n      &lt;filter **&gt;\r\n        @type concat\r\n        key log\r\n        multiline_start_regexp \/^\\S\/\r\n        separator &quot;&quot;\r\n        flush_interval 5\r\n        timeout_label @NORMAL\r\n      &lt;\/filter&gt;\r\n \r\n      &lt;match **&gt;\r\n        @type relabel\r\n        @label @NORMAL\r\n      &lt;\/match&gt;\r\n    &lt;\/label&gt;\r\n \r\n    &lt;label @NORMAL&gt;\r\n      &lt;match **&gt;\r\n        @type cloudwatch_logs\r\n        @id out_cloudwatch_logs_containers\r\n        region &quot;#{ENV.fetch(&#039;REGION&#039;)}&quot;\r\n        log_group_name &quot;\/aws\/containerinsights\/#{ENV.fetch(&#039;CLUSTER_NAME&#039;)}\/application&quot;\r\n        log_stream_name_key stream_name\r\n        remove_log_stream_name_key true\r\n        auto_create_stream true\r\n        &lt;buffer&gt;\r\n          flush_interval 5\r\n          chunk_limit_size 2m\r\n          queued_chunks_limit_size 32\r\n          retry_forever true\r\n        &lt;\/buffer&gt;\r\n      &lt;\/match&gt;\r\n    &lt;\/label&gt;\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>And apply it:<\/p>\n<pre class=\"brush: bash; title: ; notranslate\" title=\"\">\r\ncurl fluentd.yaml | sed &quot;s\/{{NAMESPACE}}\/default\/&quot; | kubectl apply -f -\r\n<\/pre>\n<p>&nbsp;<\/p>\n<blockquote><p>Where &quot;<strong>default<\/strong>&quot; is the name of the required namespace<\/p><\/blockquote>\n<p>&nbsp;<\/p>\n<p>An example of a sidecar deployment:<\/p>\n<h3>deployment.yaml<\/h3>\n<pre class=\"brush: yaml; title: ; notranslate\" title=\"\">\r\napiVersion: apps\/v1\r\nkind: Deployment\r\nmetadata:\r\n  labels:\r\n    app: testapp\r\n  name: testapp\r\nspec:\r\n  replicas: 1\r\n  selector:\r\n    matchLabels:\r\n      app: testapp\r\n  strategy: {}\r\n  template:\r\n    metadata:\r\n      labels:\r\n        app: testapp\r\n    spec:\r\n      serviceAccountName: fluentd\r\n      terminationGracePeriodSeconds: 30\r\n      initContainers:\r\n        - name: copy-fluentd-config\r\n          image: busybox\r\n          command: [&#039;sh&#039;, &#039;-c&#039;, &#039;cp \/config-volume\/..data\/* \/fluentd\/etc&#039;]\r\n          volumeMounts:\r\n            - name: config-volume\r\n              mountPath: \/config-volume\r\n            - name: fluentdconf\r\n              mountPath: \/fluentd\/etc\r\n      containers:\r\n      - image: alpine:3.10\r\n        name: alpine\r\n        command: [&quot;\/bin\/sh&quot;]\r\n        args: [&quot;-c&quot;, &quot;while true; do echo hello 2&gt;&amp;1 | tee -a \/var\/log\/application.log; sleep 10;done&quot;]\r\n        volumeMounts:\r\n        - name: fluentdconf\r\n          mountPath: \/fluentd\/etc\r\n        - name: varlog\r\n          mountPath: \/var\/log\r\n      - image: fluent\/fluentd-kubernetes-daemonset:v1.7.3-debian-cloudwatch-1.0\r\n        name: fluentd-cloudwatch\r\n        env:\r\n          - name: REGION\r\n            valueFrom:\r\n              configMapKeyRef:\r\n                name: cluster-info\r\n                key: logs.region\r\n          - name: CLUSTER_NAME\r\n            valueFrom:\r\n              configMapKeyRef:\r\n                name: cluster-info\r\n                key: cluster.name\r\n          - name: AWS_ACCESS_KEY_ID\r\n            value: &quot;XXXXXXXXXXXXXXX&quot;\r\n          - name: &quot;AWS_SECRET_ACCESS_KEY&quot;\r\n            value: &quot;YYYYYYYYYYYYYYY&quot;\r\n        resources:\r\n          limits:\r\n            memory: 400Mi\r\n          requests:\r\n            cpu: 100m\r\n            memory: 200Mi\r\n        volumeMounts:\r\n          - name: config-volume\r\n            mountPath: \/config-volume\r\n          - name: fluentdconf\r\n            mountPath: \/fluentd\/etc\r\n          - name: varlog\r\n            mountPath: \/var\/log\r\n      volumes:\r\n        - name: config-volume\r\n          configMap:\r\n            name: fluentd-config\r\n        - name: fluentdconf\r\n          emptyDir: {}\r\n        - name: varlog\r\n          emptyDir: {}\r\n<\/pre>\n<p>&nbsp;<\/p>\n<blockquote><p>In this deployment, the variables are set to &quot;<strong>AWS_ACCESS_KEY_ID<\/strong>&quot; and &quot;<strong>AWS_SECRET_ACCESS_KEY<\/strong>&quot;, since at the moment there are endpoints for <strong>IAM<\/strong> roles only for services: <strong>EC2<\/strong>, <strong>ECS Fargate<\/strong> and <strong>Lambda<\/strong>. For avoid this you can use <strong>OpenID Connect<\/strong> provider for <strong>EKS<\/strong>.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>At the time of writing, EKS Fargate does not support a driver log for recording to CloudWatch. The only option is to use Sidecar Let&#8217;s create a ConfigMap, in which we indicate the name of the EKS cluster, region and namespace: &nbsp; Next, let&#8217;s create a service account and a ConfigMap with a configuration file &hellip; <a href=\"https:\/\/artem.services\/?p=2054&#038;lang=en\" class=\"more-link\">\u041f\u0440\u043e\u0434\u043e\u043b\u0436\u0438\u0442\u044c \u0447\u0438\u0442\u0430\u0442\u044c<span class=\"screen-reader-text\"> &quot;AWS &#8212; EKS Fargate &#8212; Fluentd CloudWatch&quot;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[599],"tags":[543,887,1563,1685,1687,549,551,1689],"_links":{"self":[{"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/posts\/2054"}],"collection":[{"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/artem.services\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2054"}],"version-history":[{"count":4,"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/posts\/2054\/revisions"}],"predecessor-version":[{"id":2225,"href":"https:\/\/artem.services\/index.php?rest_route=\/wp\/v2\/posts\/2054\/revisions\/2225"}],"wp:attachment":[{"href":"https:\/\/artem.services\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2054"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/artem.services\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2054"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/artem.services\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2054"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}