CX Cloud Docs
Primary version
Primary version
  • Welcome To CX Cloud
  • Introduction
    • Why CX Cloud?
    • Architecture Overview
    • How To Participate?
    • Change Log
  • Getting Started
    • Requirements
      • Prepare Local Environment
      • Install CX Cloud CLI
    • Setting up a CX Cloud Project
      • Provision Infrastructure
      • Generating Core Services
      • Generating a Frontend Demo
      • Configuring Routing Manifest
      • Configuring CI/CD Pipeline
      • Configuring Data Engine
  • Guides
    • Infrastructure Operations
      • Getting Access To An Already Running Infrastructure
      • Deleting Deployments
      • Managing Secrets
    • Application Development
      • Git Repository Conventions
      • Process Engine
Powered by GitBook
On this page
  • Architecture
  • Fluentd
  • Kinesis Firehose, S3 and Elasticsearch
  1. Getting Started
  2. Setting up a CX Cloud Project

Configuring Data Engine

PreviousConfiguring CI/CD PipelineNextInfrastructure Operations

Last updated 5 years ago

The CX Cloud Kubernetes cluster provisioned with kops doesn't come out of the box with centralized logging for the Pods or application. Hence, the data engine 1.0 mostly focus on logging but with a data lake possibility.

Architecture

The architecture can be seen in the picture below. This version only support at the moment AWS since we use AWS services like Kinesis Firehose.

  • Fluentd stream the stdout logs lines to Kinesis Firehose from all Kubernetes pods.

  • Kinesis Firehose load the streaming data into Amazon S3 and Amazon Elasticsearch service.

  • Amazon S3 store compressed logs that can be used for backups or for further analysis.

  • Amazon Elasticsearch service store the logs that can be easily searched with Kibana, which is a part of the managed service from AWS.

Fluentd

Fluentd should be deployed to the cluster as a deamon set in order to read all pods.

To get started:

  • Install the repository:

helm repo add cxcloud-fluentd https://raw.githubusercontent.com/cxcloud/helm-fluentd-kinesis-firehose/master
  • Update repositories:

helm repo update
  • Install the chart with version 0.1.0 and the release name my-fluentd-release into the namespace kube-system:

helm install cxcloud-fluentd/helm-fluentd-kinesis-firehose \
  --version 0.1.0 \
  --name my-release \
  --namespace kube-system

Kinesis Firehose, S3 and Elasticsearch

The Fluentd daemonset requires that an AWS account has already been provisioned with a Kinesis Firehose stream and with its data stores (eg. Amazon S3 bucket, Amazon Elasticsearch Service, etc).

The following example show how the module can be used in Terraform.

module "kinesis-firehose-elasticsearch" {
  source                       = "github.com/cxcloud/terraform-kinesis-firehose-elasticsearch?ref=v1.1.0"
  region                       = "eu-west-1"
  es_name                      = "cxcloud"
  es_ver                       = 6.5
  es_instance_type             = "t2.small.elasticsearch"
  es_instance_count            = 1
  es_dedicated_master_enabled  = false
  es_ebs_size                  = 35
  es_snapshot_start_hour       = 23
  es_name_tag                  = "CX Cloud"
  es_whitelisted_ips           = ["34.240.135.89/32"]
  stream_name                  = "cxcloud"
  s3_bucket                    = "sandbox-logging"
  s3_buffer_size               = 10
  s3_buffer_interval           = 60
  s3_compression_format        = "GZIP"
  es_index_name                = "cxcloud"
  es_type_name                 = "logs"
  es_buffering_size            = 10
  es_buffering_interval        = 60
  s3_backup_mode               = "AllDocuments"
  whitelisted_aws_account_arns = ["arn:aws:iam::012345678901:root"]
}

There is a CX Cloud provided for installing Fluentd to the Kubernetes cluster.

The helm chart is more in detail documented on the GitHub repository, .

Available is a CX Cloud provided Terraform module, for helping with the installation of Kinesis Firehose, Amazon S3 bucket and Amazon Elasticsearch Service.

The Terraform module is more in detail documented on the GitHub repository, .

helm chart
helm-fluentd-kinesis-firehose
terraform-kinesis-firehose-elasticsearch
terraform-kinesis-firehose-elasticsearch
Data Engine, centralized logging