# Welcome To CX Cloud

Welcome to CX Cloud. This site will help you through the journey of setting up and operating a CX Cloud project.

## What is CX Cloud?

CX Cloud is an idea for creating scalable microservices based architecture on AWS Cloud. This project is created and maintained by [Tieto CEM](https://www.tieto.com/cem). We use this idea, accelerators, architecture and 3rd party tools (e.g. commercetools, Algolia, Contentful) for creating high quality online services for our customers.

CX Cloud is not a ready-made product, but more of a collection of accelerators and best practices to help project teams get up and running more quickly. In our approach, we emphasize the optimal balance between centralized governance and team level autonomy. Targeting to enable coprorate-wide usage of cloud native technologies for optimal flexibility and solution specific needs, in addition to long-term sustainability and optimal TCO.

For additional information, see CX Cloud [implementation scenarios](/v3.0-docs/introduction/how_to_use) and [architecture](/v3.0-docs/introduction/architecture-overview). Check also our [Changelog](/v3.0-docs/introduction/changelog) for quarterly summaries of the most important changes and improvements.

And although our CX Cloud CLI is built for AWS, there is no deep dependency on AWS in CX Cloud in general. You are most welcomed to run CX Cloud with MS Azure or Google Cloud as well.

## Getting Started

For getting started, check out the [Getting Started guide](/v3.0-docs/getting-started-1/getting-started). After you set up the basics, you can move forward with [Setting up your own CX Cloud project](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project) and start exploring the possibilities.

## Online demo

Front-end: <https://demo.cxcloud.com/>\
Content API: <https://demo.cxcloud.com/api/content/v1/api-docs/> ·\
Search API: <https://demo.cxcloud.com/api/search/v1/api-docs/> ·\
Commerce API: <https://demo.cxcloud.com/api/commerce/v1/api-docs/>

## Source code

CX Cloud components and demo infra and services are available at [GitHub](https://github.com/cxcloud).

## License

CX Cloud and its components are released under the MIT license.


# Why CX Cloud?

CX Cloud is a battle-proven solution accelerator for cloud native architecture. It is based on API-first approach, microservices and cloud infrastructure providing fast time-to-market, agility in development and lack of waste via automation.

Example solutions include web shops, customer self-service solutions, content rich web sites and API-layer for mobile apps and other clients. Solution implementation can also be part of "decomposing the monolith" program where your customer targets to create new online solutions with modern omni-channel architecture while at the same time reducing investments towards legacy platforms.

We have seen many cloud native initiatives fail to reach the high expectations due to siloed approach and lack of governance in the areas of end-to-end feasibility, common services and long term architecture. For us, CX Cloud is a tool to promote also these long-term aspects in addition to enabling fast innovtion and benefitting from cloud native approach.

It is good to remember that even if CX Cloud provides guidance and accelerators to create an end-to-end online solution including infra, services and front-end, it is not a ready-made solution. It is more of a framework and its granularity of accelerators enable you to hand pick only the ones you need. For example, you can just use the infra and CI/CD part to get your team up and running in one day and then continue with customizing the rest of the stack.

In high level, there are three main scenarios for usage.

* A green field project to create a new solution or replace existing solution
* Use CX Cloud for a "decomposing the monolith" program
* Add CX Cloud capablities to an existing microservices stack

## 1. "Greenfield"

In this scenario, the solution scope has little dependencies on the existing IT stack. You have quite big freedom in technology choices regarding both services, front-ends, infrastructure, and CI/CD configurations. As the target is quite a "stand-alone" solution stack, no separate API Gateway is necessarily needed in first phase as CX Cloud runs Kubernetes with an internal ingress loadbalancer.

For a greenfield project, you can make a clean start for your project using [Kubernetes Infra Template](https://github.com/cxcloud/demo-cxcloud-k8s) to install the infra with Kubernetes and [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli) and generate services needed for your project in hand.

## 2. "Decomposing the monolith"

Quite similar to the greenfield. But there is a need for separate API Gateway to enable multi-vendor approach (different vendors can expose API's via one gateway). This gateway can also be utilized to expose API's of current legacy platforms for new client/front-end solutions. CX Cloud enables generating and running AWS API Gateway outside the Kubernetes cluster but you can also configure the tool of choice (e.g. WSO2, Azure API Management) separately.

In this case, you can make a clean start for your project using [Kubernetes Infra Template](https://github.com/cxcloud/demo-cxcloud-k8s) to install the infra with Kubernetes and [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli) and generate services needed for your project.

## 3. Add capabilities for your existing microservice stack

In this scenario, there is already a microservices architecture in place, typically including centralized API Gateway, version control, and DevOps pipeline. Your task might be for example to add additional capabilities (e.g. commerce, content, search, process-engine) to the existing stack. And in this area, CX Cloud services might come in handy.

In this case, you can generate required services or service template using [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli). More information about generating core services can be found here: [Generating Core Services](https://docs.cxcloud.com/setting-up-a-cxcloud-project/generating-core-services).


# Architecture Overview

CXCloud projects are essentially services and functions that are deployed to a Kubernetes cluster. There's nothing magical or different about this architecture and the aim of CXCloud is just to make this process as simple and easy to understand as possible.

![](/files/-LLy1tIm2NxoTjaQHYfy)


# How To Participate?

CX Cloud is developed further also via customer projects. As an example, in one case a simple IAM (identity and access management) solution was needed, so the project team developed a service for AWS Cognito. This was then taken as a core module to CX Cloud and today via CLI anyone can generate a Cognito powered Auth-service to their CX Cloud project.

## Participate in CX Cloud development

## Develop CX Cloud via customer project


# Change Log

## Changelog

We constantly develop Tieto CX Cloud based on customer projects, internal ideas, and partner contributions. Naturally, you can follow progress via Github but here are quarterly summaries of the most important changes and improvements.

### Planned for H2/2019

* Add GraphQL to basic set-up.
* Create more templates to enable custom microservices generation with various technologies (e.g. Node, Java, Groovy) using CX Cloud CLI.
* Add Azure support for infra provisioning.

## Changes in 2019

### H1/2019

* CI/CD pipeline between Github and AWS.
* Data Engine 1.0 for log management.
* New core service for Reach marketplace.

## Changes in 2018

### Q4/2018

Created AWS Cognito core service and enabled creation of Auth service with CLI. Improved CX Cloud documentation. Implemented front-end demo based on React.

### Q3/2018

Created CX Cloud CLI to enable more easy project set-up with infra, service and front-end generation. Switched from AWS ECS to Kubernetes.

### Q2/2018

We split core module to more granular modules (e.g. commerce, content, search). Also enabled generation of services and service specific API's with selected level of granularity based on available core modules.

### Q1/2018

Initial release.


# Requirements

Before you set up a CX Cloud project, you need to prepare your environment and configure the tools needed for the process.

Please start by [preparing your environment](/v3.0-docs/getting-started-1/getting-started/prepare-your-environment) and then move forward installing [CX Cloud CLI](/v3.0-docs/getting-started-1/getting-started/install-cxcloud-cli).&#x20;


# Prepare Local Environment

## Install Git client

Currently, our CLI only supports GitHub as a versioning system. Once you have installed Git client of your preference, make sure to configure global `username` and `email` properties with values from your GitHub account like this:

```bash
git config --global user.name "username"
git config --global user.email "example@email.com"
```

## Install and Configure the AWS CLI tools

You naturally need an AWS account. For quick testing, You can create your own [AWS Free Tier account](https://aws.amazon.com/free/). Then install the AWS Cli tools using [this guide](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) and configure a profile:

```bash
$ aws configure --profile myproject
```

Check [this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) for more information on configuring AWS CLI.

Export your AWS profile to an environment variable. This variable is used by CX Cloud tools to interact with your AWS account:

```bash
$ export AWS_PROFILE=myproject
```

CX Cloud CLI supports preset AWS API keys as environment variables. This is useful for example when running [AWS Vault](https://github.com/99designs/aws-vault) for storing the AWS secrets in your operating system's secure keystore. The `AWS_PROFILE` doesn't need to be exported in case AWS Vault is used.

## Install Kubernetes and Kops

Before you start, you need to have Kubernetes and it's tools installed on your computer. Follow [this guide](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to Install those. On Mac, you can install Kubernetes tools using `homebrew`:

```bash
$ brew install kubernetes-cli
```

CXCloud CLI uses Kops to create and configure your Kubernetes cluster. You can install it on your Mac using `homebrew`:

```bash
$ brew update && brew install kops
```

Or on your Linux machine:

```bash
curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops
```

Check [Kops' documentation](https://github.com/kubernetes/kops#installing) for more information.

## Install Helm

[Install the Helm client](https://docs.helm.sh/using_helm/#installing-helm) on your machine. On Mac this can be done using `homebrew`:

```bash
$ brew install kubernetes-helm
```

## Install Terraform

[Install Terraform ](https://www.terraform.io/intro/getting-started/install.html) on your machine. On Mac this can be done using `homebrew`:

```bash
$ brew install terraform
```

## Install Node.js and NPM

You need to install Node.js and NPM on your computer. It is recommended to install the latest Node.js LTS version. On macOS you can do this using `homebrew`:

```
$ brew install node
```

For other systems, you can check [Node.js official documentation](https://nodejs.org/en/download/package-manager/).

Once you have installed Node, you can confirm that NPM is working using `npm --version`

## Install Docker

To be able to compose a deployable image you have to install Docker. Head over to [this page](https://docs.docker.com/install/#supported-platforms) and get the Docker for your system. You'll need to create a Docker account as well.

## Install Ansible

To be able to access the private services in Kubernetes an VPN solution is needed. Ansible is used to install OpenVPN for the CX Cloud Demo.

[Install Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on your machine. On Mac this can be done using `homebrew`:

```
$ brew install ansible
```


# Install CX Cloud CLI

## Installing CX Cloud CLI Tools

{% hint style="info" %}
Before proceeding, make sure you have prepared your local environment as instructed in [this guide](/v3.0-docs/getting-started-1/getting-started/prepare-your-environment).
{% endhint %}

Install CX Cloud CLI by running the following command:

```bash
$ npm install -g cxcloud
```

If you need to update CX Cloud CLI, run this command\`:

```
$ npm i -g cxcloud@latest
```

After installation is complete, you can call CX Cloud directly from command line:

```bash
$ cxcloud --version
```

Or if you don't want to install CX Cloud globally, you can call it directly using `npx`:

```
$ npx cxcloud --version
```


# Setting up a CX Cloud Project

Great, you have now [prepared your environment and configured the tools needed](/v3.0-docs/getting-started-1/getting-started). So the next phase is to set up your own CX Cloud project following these steps:

1. [Provision Infrastructure](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/provision-infrastructure)
2. [Generating core services](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/generating-core-services)
3. [Generating a frontend](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/generating-a-frontend)
4. [Routing manifest](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/generating-a-frontend)
5. [Configuring CI/CD pipeline](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd)
6. [Configuring Data Engine 1.0 (logging)](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/data-engine)


# Provision Infrastructure

Cx Cloud modules and deployments is developed for Kubernetes, hence, CX Cloud can run on any existing Kubernetes cluster.

The CX Cloud demo setup will install the infra with Kubernetes, running on AWS. When the setup is up and running it can be used for the [CX Cloud Demo app](https://github.com/cxcloud/demo-cxcloud-monorepo-angular) or any other CX Cloud related project.

## Getting stated with CX Cloud Demo

{% hint style="warning" %}
Windows is not supported. You can only provision the infrastructure using a Unix based OS. Eg. macOS or Linux.
{% endhint %}

Make sure you have [installed all the requirements](/v3.0-docs/getting-started-1/getting-started/prepare-your-environment) on your computer.

### Installation steps

There are few steps that has to be done in order to install the infrastructure. Follow the instructions from the following steps:

1. [Provision the infra](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/terraform/README.md).
2. [Install OpenVPN](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/openvpn-ansible/README.md) in order to access services in the private subnets on AWS.
3. [Install Kubernetes](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/kubernetes/README.md).

{% hint style="info" %}
If you have enabled Multi-Factor Authentication on your AWS account or you are assuming another role which does, you might be asked to enter your MFA token (using your Authenticator app) in this step.
{% endhint %}

{% hint style="info" %}
AWS credentials has to be [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) in order to provision the infra for CX Cloud demo. It is possible to provision with preset API keys as environment variables. This is useful for example when running [AWS Vault](https://github.com/99designs/aws-vault) for storing the secrets in your operating system's secure keystore.
{% endhint %}

### Configuring a domain for your online service

After you have generated the Kubernetes infrastructure, you would want to assign a domain to your future online service (or services)

The assumed starting state is that you have access to manage "example.com" DNS. CX Cloud is to be configured to utilize this domain to explore its possibilities. As there might already be a site running in [www.example.com](http://www.example.com), the target state with CX Cloud would look like this:

* `newsite.example.com` (customer front-end)
* `newsite.example.com/api/service-commerce/v1/` (commerce service API)
* `newsite.example.com/api/service-content/v1/` (content service API)
* `newsite.example.com/api/service-search/v1/` (search service API)
* `newsite.example.com/api/service-auth/v1/` (auth service API)
* `*.dev.newsite.example.com` (Pull Request environments)

We have used this assumption in our documentation and configuration examples. Most of these configurations are done with a routing manifest later and at this stage we only need to point the domain to Kubernetes load balancer like this:

* First figure out your Load Balancer(s) URL. The URL(s) are displayed after generating the infrastructure. But you can also find them using:

```
kubectl get ingress --all-namespaces
```

* Then you have to point your own domain name to this load balancer URL. To do this:
  * If you manage your domain using AWS' route53 service, update the settings in [terraform.tfvars](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/setting-up-a-cxcloud-project/example_vars/terraform.tfvars) and re-run the `terraform apply` command in order to update the domain name aliases. Follow the instructions from [Install Kubernetes](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/kubernetes/README.md).
  * If you manage your domain using a provider that's not AWS, navigate to your domain's management panel and create an `CNAME` record for `newsite.example.com` pointing to the load balancer URL.
    * [GoDaddy guide](https://fi.godaddy.com/help/add-a-cname-record-19236)
    * [Namecheap guide](https://www.namecheap.com/support/knowledgebase/article.aspx/9646/2237/how-can-i-set-up-a-cname-record-for-my-domain)
    * [Name.com guide](https://www.name.com/support/articles/115004895548-Adding-a-CNAME-Record)


# Generating Core Services

CX Cloud provides selected ready-made (Node.js) core services (e.g. commerce, content, search, authentication) which can be generated and added to the project using CX Cloud CLI. These can, of course, be customized later on.&#x20;

## Requirements

It is required to use a monorepo structure in order to deploy all microservices with CI/CD pipeline. More information about monorepo development strategy can be found [here](/v3.0-docs/guides/application-development/create-a-git-repository).

## Service generation

You can start by creating a new folder for a new service in monorepo `packages` folder, for example:

```
my-monorepo
├── packages
|   ├── package-commerce
|   |   ├── .cxcloud.yaml
|   |   └── ...
└── ...
```

Then navigate inside the newly created folder and start service generation by running the following commands:

```bash
$ cd package-commerce
$ cxcloud generate service
```

You will go through a number of questions and get to choose which service to create.

To enable ready-made functionality for services, CX Cloud provides a set of core modules that can be used when generating services. To enable the desired level of granularity, one can use multiple core modules for one service or create multiple services based on the same core module. It should be noted that if generating service with multiple core modules, some services might conflict with each other. In this case, you will get a warning. Example of such conflict is Commerce and Auth services. Commerce includes own authentication service, making Auth in this case redundant.

Currently CLI supports the following core modules:

* Commerce (commercetools)
* Content (Contentful)
* Auth (AWS Cognito)
* Search (Algolia)

## Configuration

After you have generated the service, it's time to provide it with configurations.

Create a `.env` file in the root directory of the monorepo and add environment-specific variables on new lines in the form `NAME=VALUE`. For example:

```
CONTENTFUL_API_KEY=12345
CTP_CLIENT_ID=qwerty-567
AUTH_URL=https://auth.example.com
...
```

{% hint style="warning" %}
**Do Not** commit sensitive information such as API keys to GitHub or other versioning systems. Make sure the `.env` file is included in `.gitignore` of the root folder.
{% endhint %}

In order to provide a consistent configuration interface to the services, which can be extended and overridden during further development, the configurations are stored in configuration files within the service. For these purposes, we are using `node-config`. The generated with CLI services are already prepackaged with `node-config` and basic configuration files. Among other settings, the configuration files include references to the environment variables which are loaded from `.env` file into `process.env`. Documentation for node-config can be found [here](https://www.npmjs.com/package/config).

The configurations below are used in CX Cloud Demo and can be copied and modified in default configuration file `config/default.js` of generated service.

Example configuration for Commerce:

```javascript
"store": {
    "defaultCurrency": "EUR",
    "supportedCurrencies": ["EUR", "USD", "GBP"]
},
"commerceTools": {
    "authHost": `${process.env.CTP_AUTH_URL}`,
    "apiHost": `${process.env.CTP_API_URL}`,
    "projectKey": process.env.CTP_PROJECT_KEY,
    "admin": {
        "clientId": process.env.CTP_CLIENT_ID,
        "clientSecret": process.env.CTP_CLIENT_SECRET
    },
    "user": {
        "clientId": process.env.CTP_USER_CLIENT_ID,
        "clientSecret": process.env.CTP_USER_CLIENT_SECRET
    }
}
```

Example configuration for Content:

```javascript
"contentful": {
    "sdkConfig": {
        "space": process.env.CONTENTFUL_SPACE_ID,
        "accessToken": process.env.CONTENTFUL_DELIVERY_API_TOKEN
    }
}
```

Example configuration for Auth:

```javascript
"cognito": {
    "userPoolId": process.env.USER_POOL_ID,
    "clientId": process.env.CLIENT_ID,
    "encryptionKey": process.env.ENCRYPTION_ID
}
```

Example configuration for Search:

```javascript
"algolia": {
    "applicationId": process.env.ALGOLIA_APPLICATION_ID,
    "apiKey": process.env.ALGOLIA_API_KEY
}
```

{% hint style="info" %}
CXCloud does not provide any testable configurations for 3rd party solutions. You must get test account or purchase licenses for selected services and set them up yourself.
{% endhint %}

## Local test

After generation is finished and configurations are set, the generated service will be working as a single API. To test locally, run the following set of commands at the root of your repository with the service:

```bash
$ npm install
$ npm run build
$ npm run start
```

It will install all required npm packages, build your Swagger documentation and start the local server for you to be able to test your configurations. Swagger provides an interface for testing right in the browser.

In case of `commerce` service and default settings, you should be able to see Swagger API documentation at `http://localhost:4003/api/commerce/v1/api-docs`

## Deployment

CI/CD pipeline handles the deployment of microservices within the monorepo. Therefore, CI/CD pipeline should be set up and configured prior deployment of the services. More information about CI/CD pipeline configuration and monorepo settings for deployment can be found [here](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd).

{% hint style="info" %}
When applications is deployed, all sensitive information such as keys, secrets or tokens are stored as Kubernetes secret object in the intended namespace. You can find an example of using Kubernetes' secrets to configure your project [here](/v3.0-docs/guides/managing-infrastructure/storing-secrets-in-kubernetes#example-storing-and-using-secrets-in-nodejs-and-node-config-module).
{% endhint %}

## Routing

After deployment, you can make your service available to the world using a [Routing Manifest](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/routing-manifest).

## Commit

As a final step, add git remote origin corresponding to your monorepo and commit changes to your Github repository by running the following commands:

```bash
$ git remote add origin https://github.com/user/my-monorepo.git
$ git add .
$ git commit -m "initial commit"
$ git push
```


# Generating a Frontend Demo

Cool, You now have infra and core services up and running! For front-end, CX Cloud CLI allows you to generate a front-end demo to test the concept with end-to-end functionality. Currently, we provide Angular 7 demo. React is coming soon.

## Front-end generation

You start by creating a new package repository for the service in git monorepo, named for example `package-frontend`. Then navigate to this folder and start front-end demo generation by running the following commands:

```bash
$ cd package-frontend
$ cxcloud generate demo
```

You will go through a set of simple questions and receive an end-to-end working demo.

## Configuration

By default, the generated demo uses our live test API including end points with pre-filled sets of data for Commerce, Content and Search services.

In order to enable the frontend demo to work with the APIs you have generated and deployed, configuration files need to be modified. Configuration files can be found in this project directory `/src/environments/`.

To connect with new APIs, change the value of `apiUrl` for each service (notice that environments files should be modified depending on the used environment). If frontend and APIs running on the same domain, it is enough to use only relative paths to the APIs:

```javascript
export const environment = {
  production: false,
  siteName: "CXCloud DEV",
  commerce: {
    apiUrl: "/api/service-commerce/v1",
    indexName: "dev_COMMERCE"
  },
  content: {
    apiUrl: "/api/service-content/v1",
    indexName: "dev_CONTENT"
  },
  auth: {
    apiUrl: "/api/service-auth/v1"
  },
  search: {
    apiUrl: "/api/service-search/v1"
  }
};
```

## Local test

In case of Angular demo, to explore it, from inside of the project folder you should run the following command:

```bash
$ npm run dev
```

By default, your demo will be available at `http://localhost:4200/` but check the console logs.

## Deployment

CI/CD pipeline handles the deployment of microservices within the monorepo. Therefore, CI/CD pipeline should be set up and configured prior deployment of the services. More information about CI/CD pipeline configuration and monorepo settings for deployment can be found [here](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd).

## Routing

After deployment, you can make your service available to the world using a [Routing Manifest](/v3.0-docs/getting-started-1/setting-up-a-cxcloud-project/routing-manifest).

## Commit

As a final step, add git remote origin corresponding to your monorepo and commit changes to your Github repository by running the following commands:

```bash
$ git remote add origin https://github.com/user/my-monorepo.git
$ git add .
$ git commit -m "initial commit"
$ git push
```


# Configuring Routing Manifest

Routing manifest is needed to make multiple services available in the same domain.

For example, let's assume you have created a front-end and 3 microservices and you want them all to be available on the same domain:

* `newsite.example.com/` should load `frontend` service
* `newsite.example.com/api/service-commerce/` should load `package-commerce` service
* `newsite.example.com/api/service-content` should load `package-content` service
* `newsite.example.com/api/service-search` should load `package-search` service

You have to create a routing manifest to achieve this. (In addition to pointing the domain in question to your Kubernetes cluster during [infra generation](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/setting-up-a-cxcloud-project/generating-infrastructure.md#configuring-a-domain-for-your-online-service))

## Create manifest

Add `.cxcloud.yaml` to the root folder of your monorepo.

{% code title="my-monorepo/.cxcloud.yaml" %}

```yaml
namespace: $GIT_BRANCH
routing:
  domain: $GIT_BRANCH.dev.newsite.example.com
  ingressClass: $INGRESS_CLASS
  lbCert: $LB_CERT
  scheme: $SCHEME
  ssl: false
  rules:
    - path: /api/service-commerce
      serviceName: package-commerce
      servicePort: 4003
    - path: /api/service-content
      serviceName: package-content
      servicePort: 4003
    - path: /api/service-search
      serviceName: package-search
      servicePort: 4003
    - path: /
      serviceName: package-frontend
      servicePort: 80
```

{% endcode %}

Please note that `path` should match the API prefix you defined when generating the service with CLI. By default, during service generation, CX Cloud CLI suggests service name as the API prefix.

After committing changes to Git repository, CI/CD pipeline will deploy changes automatically.

## Test

To test for example commerce service, `newsite.example.com/api/service-commerce/` should return {"health":"OK"} and `newsite.example.com/api/service-commerce/v1/api-docs` should present to you Swagger documentation.


# Configuring CI/CD Pipeline

When using CX Cloud CLI, it is fairly easy for one to both create and then deploy CX Cloud infra, selected services and demo frontends to AWS. This is typically enough for testing and exploration needs. We also have [tools and guidelines](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/best-practices/how-to-run-infra.md) for DevOps engineer to manually manage your environment.

But when preparing for customer project development phase, creating a CI/CD pipeline between your Github (or the Git host of your choice eg. Bitbucket) and various environments in AWS (or selected cloud provider eg. Azure) is the next thing to do. As this area can be very dependent on customer preferences and existing set up, you probably will customize your pipeline. Regardless, we have created this basic set up.

## Development workflow

The CX Cloud CI/CD pipeline has been designed with the following development workflow in mind.

![Development workflow](/files/-LlbRTyPxB4rifmjBpe0)

1. A new feature branch is created, the pipeline will checkout the code and run unit tests and perform code quality analysis. This step will be repeated every time a new commit is pushed to the branch..
2. When the feature is ready for review and testing a pull request (PR) is created. The pipeline will automatically run tests and code quality analysis. The pipeline will fail in case the tests fails or if the quality analysis doesn't pass the specified quality level. The pipeline will continue with building the micro services and deploy them into a Kubernetes namespace specific for the PR. The pipeline will send notifications to communication channels and publish the web url as a comment to the PR. Now the PR is ready for code review and quality assurance. New commits pushed to the PR will only analyze and deploy the modified projects.
3. When the pull request has passed all steps, it's possible to merge to master. When new code is pushed to master the pipeline will run almost the same steps as for pull requests except this time the pipeline will deploy to the staging environment (or Kubernetes namespace staging).
4. The process has been designed so that master is alway deployable to production. To deploy to production a release tag has to be made. The pipeline will automatically recognize all git tags and any of the tags can be with one click in Jenkins build and deployed to production.

### Requirements for the pipeline

* Kubernetes cluster
* Jenkins with preinstalled plugins
* Monorepo architecture of the micro services

### Configure the services for the pipeline

The Pipeline requires that the development strategy uses a monorepo in order to deploy all services. The directory structure for the monorepo should look like:

```
.
├── packages
|   ├── service 1
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   ├── service 2
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   └── service n
|       ├── .cxcloud.yaml
|       └── ...
├── .cxcloud.yaml
├── Jenkinsfile
└── ...
```

#### Repository root

In the root of the repository should be two files, `Jenkinsfile` and `.cxcloud.yaml`. The pipeline code is in the Jenkinsfile and the `.cxcloud.yaml` file contains the Kubernetes routing for the microservices.

Example of the `.cxcloud.yaml` file in the root of the repository:

```bash
namespace: $GIT_BRANCH
routing:
  domain: $GIT_BRANCH.dev.cxcloud.com
  ingressClass: $INGRESS_CLASS
  lbCert: $LB_CERT
  scheme: $SCHEME
  ssl: false
  rules:
    - path: /graphql
      serviceName: package-graphql-proxy
      servicePort: 80
    - path: /images
      serviceName: package-image-proxy
      servicePort: 80
    - path: /
      serviceName: package-frontend
      servicePort: 80
```

#### Repository services

Every microservice should preferable be in the packages folder but there is no naming convention for the services. Every microservice has to have a `.cxcloud.yaml` in the folder containing the Kuberntes deployments.

Kubernetes secrets should be stored in the namespace they are intended for. However, secrets intended for the PR environments / namespaces should be stored in the applications namespace. The pipeline will copy them from the applications namespace into the specific PR namespace.

Example of the content in the `.cxcloud.yaml` file for a microservice:

```bash
namespace: $GIT_BRANCH
deployment:
  name: $APP_NAME
  image:
    name: $APP_NAME
    repository: $ECR_REPOSITORY
    version: $APP_VERSION
  containerPort: 3000
  port: 80
  replicas: $MIN_REPLICAS
  cpuRequest: $CPU_REQUEST
  nodeSelector:
    kops.k8s.io/instancegroup: $INSTANCE_GROUP
  autoscaling:
    minReplicas: $MIN_REPLICAS
    maxReplicas: $MAX_REPLICAS
    targetAverageUtilization: 75
  env:
    - name: NODE_ENV
      value: production
    - name: CONTENTFUL_MANAGEMENT_API_TOKEN
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_MANAGEMENT_API_TOKEN
    - name: CONTENTFUL_ENVIRONMENT_ID
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_ENVIRONMENT_ID
    - name: CONTENTFUL_SPACE_ID
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_SPACE_ID
```

## Jenkins

The pipeline has been developed for Jenkins. A customized Jenkins image can be build from our [Jenkins repository](https://github.com/cxcloud/jenkins).

The docker image is tailored for Kubernetes and all required plugins will be preinstalled in the Docker image. The Docker pod itself will not run the CI/CD pipeline but a dependency worker will be launched when needed and perform all the steps for the pipeline. The worker will be terminated after the jobs is finished. Several workers can run at the same time as long as the underlying infra can handle the load.

The Jenkins configuration and installation is in detail explained on the [Jenkins repository](https://github.com/cxcloud/jenkins).

## CI/CD Pipeline

A working example of the CI/CD pipeline is in the CX Cloud demo application, [cxcloud-monorepo-angular](https://github.com/cxcloud/demo-cxcloud-monorepo-angular).

### Jenkinsfile structure

The [Jenkinsfile](https://github.com/cxcloud/demo-cxcloud-monorepo-angular/blob/master/Jenkinsfile) structure for the CX Cloud demo explained here is a guideline and should be modified depending on technologies and deployment strategies used. The important part is that the Jenkinsfile should be configured as a multibranch pipeline in Jenkins and that there are different actions specified e.g. for normal branches, pull requests and base branch.

#### Variables

The pipeline starts with defining the variables for all environments, DEV/TEST (pull requests), staging (base branch, master) and production (git tag).

The first stage in the pipeline, `populate variables` will set the values for many of the variables that has to be calculated while the pipeline is running. E.g. some values for pull requests can only be populated in case the job execution is an actual pull request.

#### Code Quality

The following stage, `run tests` will run the tests. The pipeline will stop in case any of the tests fails. When we work in a branch all tests will run for all projects. However, when a pull requests is created, only tests for modified projects will be executed. When deploying a release tag, none of the tests will run, since all tests had already to pass previously in the base branch.

Stage `SonarQube analysis` will run sonar-scanner and analyse the projects on SonarQube. The pipeline will not check at this stage if the code quality for the projects pass or fail.

Stage `SonarQube quality gate` ensure that all the projects has passed the SonarQube analysis. The pipeline will fail and send notifications in case any of the projects doesn't pass the "quality gate".

#### Create Kubernetes namespace

The pipeline will deploy all of the services into a new dev/test environment or to the staging or production environment. Stage `Create namespace` will create a new kubernetes namespace for the deployment in case it doesn't already exist.

#### Copy namespace secrets

The CX Cloud demo uses the namespace `applications` for storing secrets for the test/dev environments. Stage `Copy namespace secrets to DEV/TEST environment` will copy the secrets from the application namespace into the test/dev environments.

Secrets will only be copied for pull requests. Secrets for other environments (staging and production) has to be created independently of the pipeline.

#### Deploy projects

Stage `Deploy projects` will use the cxcloud command line tool, [cxcloud-cli](https://github.com/cxcloud/cxcloud-cli) to deploy the projects. The command will create docker images, upload them to a private docker registry and deploy the new services into the Kubernetes cluster.

The pipeline will take care of only deploying modified projects unless it's to a new namespace. Also deployments to staging and production will deploy all of the services.

#### Clean up test environments

It is vital to delete old environment since every pull request will create a new dev/test environment. Stage `Cleanup development environments` will delete all dev/test environment for pull requests that has been closed. This stage will only be executed in case any new code will be pushed to the base branch, master.

#### Post actions

The post actions will always run in the end of job. The pipeline for the CX Cloud demo will always notify Flowdock if the execution succeeded or failed. The notification will contain details about the url to the running environment, GitHub url, Jenkins execution url and git commit messages.

The post section can easily be modified to send notifications to other destinations like Slack.

In case the job fails, the post action will delete the namespace if it was created during the same execution.

### Example

Below is a Jenkins screenshot of a deployment to staging for the [CX Cloud demo application](https://github.com/cxcloud/demo-cxcloud-monorepo-angular).

![Deployment to Staging](/files/-LlbRTyZ_fb2_lDj12dE)


# Configuring Data Engine

The CX Cloud Kubernetes cluster provisioned with kops doesn't come out of the box with centralized logging for the Pods or application. Hence, the data engine 1.0 mostly focus on logging but with a data lake possibility.

## Architecture

The architecture can be seen in the picture below. This version only support at the moment AWS since we use AWS services like Kinesis Firehose.

![Data Engine, centralized logging](/files/-LlbTJcRV2rlEd4AGTrw)

* Fluentd stream the stdout logs lines to Kinesis Firehose from all Kubernetes pods.
* Kinesis Firehose load the streaming data into Amazon S3 and Amazon Elasticsearch service.
* Amazon S3 store compressed logs that can be used for backups or for further analysis.
* Amazon Elasticsearch service store the logs that can be easily searched with Kibana, which is a part of the managed service from AWS.

### Fluentd

Fluentd should be deployed to the cluster as a deamon set in order to read all pods.

There is a CX Cloud provided [helm chart](https://github.com/cxcloud/helm-fluentd-kinesis-firehose) for installing Fluentd to the Kubernetes cluster.

To get started:

* Install the repository:

```bash
helm repo add cxcloud-fluentd https://raw.githubusercontent.com/cxcloud/helm-fluentd-kinesis-firehose/master
```

* Update repositories:

```bash
helm repo update
```

* Install the chart with version 0.1.0 and the release name `my-fluentd-release` into the namespace `kube-system`:

```bash
helm install cxcloud-fluentd/helm-fluentd-kinesis-firehose \
  --version 0.1.0 \
  --name my-release \
  --namespace kube-system
```

The helm chart is more in detail documented on the GitHub repository, [helm-fluentd-kinesis-firehose](https://github.com/cxcloud/helm-fluentd-kinesis-firehose).

### Kinesis Firehose, S3 and Elasticsearch

The Fluentd daemonset requires that an AWS account has already been provisioned with a Kinesis Firehose stream and with its data stores (eg. Amazon S3 bucket, Amazon Elasticsearch Service, etc).

Available is a CX Cloud provided Terraform module, [terraform-kinesis-firehose-elasticsearch](https://github.com/cxcloud/terraform-kinesis-firehose-elasticsearch) for helping with the installation of Kinesis Firehose, Amazon S3 bucket and Amazon Elasticsearch Service.

The following example show how the module can be used in Terraform.

```
module "kinesis-firehose-elasticsearch" {
  source                       = "github.com/cxcloud/terraform-kinesis-firehose-elasticsearch?ref=v1.1.0"
  region                       = "eu-west-1"
  es_name                      = "cxcloud"
  es_ver                       = 6.5
  es_instance_type             = "t2.small.elasticsearch"
  es_instance_count            = 1
  es_dedicated_master_enabled  = false
  es_ebs_size                  = 35
  es_snapshot_start_hour       = 23
  es_name_tag                  = "CX Cloud"
  es_whitelisted_ips           = ["34.240.135.89/32"]
  stream_name                  = "cxcloud"
  s3_bucket                    = "sandbox-logging"
  s3_buffer_size               = 10
  s3_buffer_interval           = 60
  s3_compression_format        = "GZIP"
  es_index_name                = "cxcloud"
  es_type_name                 = "logs"
  es_buffering_size            = 10
  es_buffering_interval        = 60
  s3_backup_mode               = "AllDocuments"
  whitelisted_aws_account_arns = ["arn:aws:iam::012345678901:root"]
}
```

The Terraform module is more in detail documented on the GitHub repository, [terraform-kinesis-firehose-elasticsearch](https://github.com/cxcloud/terraform-kinesis-firehose-elasticsearch).


# Infrastructure Operations


# Getting Access To An Already Running Infrastructure

In most real world cases, you haven't probably generated and set up the Kubernetes infrastructure yourself. But you might need to access it to deploy services or change settings.

To do that, you should have access to the AWS account that the infrastructure is running in, and export the `AWS_PROFILE` variable like before:

```bash
$ export AWS_PROFILE=my-cxcloud-dev
```

Now run the following script to download the configuration from the running cluster to your system:

```bash
$ kops export kubecfg somecluster.k8s.local --state=s3://somecluster.k8s.local
```

This command will download the configuration to your system and sets up a Kubernetes `context` for easier use. Now you can run any `kubectl` or `cxcloud` command to interact with your cluster.

## Kubernetes Contexts

In case you have access to more than one Kubernetes cluster, you can use `kubectl config` tool and switch between different contexts.

To see a list of your available contexts run:

```bash
$ kubectl config get-contexts
```

This command will print a list of configured contexts on your system and the currently in use context.

To switch between contexts, run:

```bash
$ kubectl config use-context mycluster.k8s.local
```


# Deleting Deployments

After you have deployed a service using `cxcloud deploy` you can run the following command to delete the deployment and all of it's configurations:

```bash
$ cxcloud deploy --purge
```

This will undo anything that is done by running `cxcloud deploy` except it's namespace. If you wish to destroy the namespace as well, run:

```bash
$ cxcloud deploy --purge-all
```

{% hint style="warning" %}
If multiple services are using the same namespace that you have defined in your manifest, this command will purge all other services as well because it will destroy the namespace.
{% endhint %}


# Managing Secrets

Kubernetes has the capability to store secrets and make them available to your services. You can store different secrets and use them through your CX Cloud deployment environment variables. To store a secret in `applications` namespace (it's the default namespace for CX Cloud apps) run the following command:

```bash
$ kubectl create secret generic prod-db-secret \
    --from-literal=username=produser \
    --from-literal=password=Y4nys7f11 \
    --namespace=applications
```

{% hint style="info" %}
You have to specify which namespace you want your secret to be created in, because only deployments in that namespace will be able to access the secrets.
{% endhint %}

Now when you create a deployment, you can reference your secret:

{% code title=".cxcloud.yaml" %}

```yaml
deployment:
  name: my-backend
  image:
    name: my-backend-image
    repository: YOUR_AWS_ECR_REPOSITORY_URL_HERE
    version: 1.1.1
  containerPort: 8080
  replicas: 2
  env:
    - name: NODE_ENV
      value: production
    - name: PROD_DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: prod-db-secret
          key: username
    - name: PROD_DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: prod-db-secret
          key: password
```

{% endcode %}

When the service is deployed, your secrets will become available under the environment variables you have specified. For example in NodeJS, you can access them like so:

```javascript
console.log(process.env.PROD_DB_USERNAME); // produser
console.log(process.env.PROD_DB_PASSWORD); // Y4nys7f11
```

For more information about Kubernetes secrets, [visit this article](https://kubernetes.io/docs/concepts/configuration/secret/).

## Example: Storing and using secrets in NodeJS and \`node-config\` module

One of the core services that you can generate using the CX Cloud CLI is Commerce service. It is communicating with **commercetools** platform and thus requires some configurations (like API key, etc). We can take advantage of Kubernetes secrets explained above to store these information without having to publish them to GitHub.

First, generate a service using the CLI and choose Commercetools, [as explained in its section](broken://pages/-Lqbg5B8OJmtzggV5BVe).

Then, according to `node-config`'s [documentation](https://github.com/lorenwest/node-config/wiki/Environment-Variables#custom-environment-variables), create a file named `custom-environment-variables.json` in the `config` folder of the generated service with the following content:

```javascript
{
  "commerceTools": {
    "projectKey": "COMMERCETOOLS_PROJECT_KEY",
    "admin": {
      "clientId": "COMMERCETOOLS_ADMIN_CLIENT_ID",
      "clientSecret": "COMMERCETOOLS_ADMIN_CLIENT_SECRET"
    },
    "user": {
      "clientId": "COMMERCETOOLS_USER_CLIENT_ID",
      "clientSecret": "COMMERCETOOLS_USER_CLIENT_SECRET"
    }
  }
}
```

This file will tell `node-config` to look for those environment variables and map them to certain keys. For example the configuration key `commerceTools.admin.clientId` will map to `COMMRCETOOLS_ADMIN_CLIENT_ID` and so on.

Now we have to store those data in Kubernetes and make them available to our service using the specified environment variables. To do that, first let's create a secret in `applications` namespace:

```bash
$ kubectl create secret generic prod-commercetools \
    --from-literal=projectKey=xxxxxxx \
    --from-literal=adminClientId=xxxxxxx \
    --from-literal=adminClientSecret=xxxxxxx \
    --from-literal=userClientId=xxxxxxx \
    --from-literal=userClientSecret=xxxxxxx \
    --namespace=applications
```

Replace the `xxxxxxx` above with your actual data. A secret will be created.

Now modify your `.cxcloud.yaml` file and add the proper environment variables referencing the secret you just created (new values are added from line 12):

{% code title=".cxcloud.yaml" %}

```yaml
deployment:
  name: $APP_NAME
  image:
    name: $APP_NAME
    repository: xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/newsite.example.com
    version: $APP_VERSION
  containerPort: 4003
  replicas: 2
  env:
    - name: NODE_ENV
      value: production
    - name: COMMERCETOOLS_PROJECT_KEY
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: projectKey
    - name: COMMERCETOOLS_ADMIN_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: adminClientId
    - name: COMMERCETOOLS_ADMIN_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: adminClientSecret
    - name: COMMERCETOOLS_USER_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: userClientId
    - name: COMMERCETOOLS_USER_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: userClientSecret
```

{% endcode %}

When done, increase the version in your `package.json` file and run:

```bash
$ cxcloud deploy
```


# Application Development


# Git Repository Conventions

The development strategy required to use a "monorepo" in order to deploy all services using CI/CD pipeline. The repository contains multiple packages (microservices) that can but do not have to be related.

## Project structure

The file structure for the monorepo should look like:

```
my-monorepo
├── packages
|   ├── package 1
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   ├── package 2
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   └── package n
|       ├── .cxcloud.yaml
|       └── ...
├── .cxcloud.yaml
└── ...
```

Every microservice folder should include `.cxcloud.yaml` file with the configuration for Kubernetes.

There is no naming convention for the packages, but it is recommended to add a prefix to the microservice such as `"package-"` or `"service-"`.

In order to simplify and optimize the workflow around development with the multi-package repository, the managing tool such as [Lerna](https://lerna.js.org/) can be in use.


# Process Engine

A simple helper for using multiple AWS SQS queues at the same time. This tool provides an easy way to map incoming events to actions.

## Installation and Usage

```bash
npm install @cxcloud/process-engine
```

```typescript
import {
  createQueueProcessor,
  createQueuePool
} from '@cxcloud/process-engine-core';

const pool = createQueuePool([
  createQueueProcessor(
    {
      name: 'my-sqs-queue',
      concurrency: 2
    },
    [
      {
        conditions: [
          {
            path: 'myEvent.name',
            value: 'someValue'
          },
          {
            path: 'customer.type',
            value: 'gold'
          }
        ],
        action: (message, sendMessage) => {
          console.log('Received Message:', message.data);
          message.deleteMessage().then(() => {
            // Next should be called after each message is processed
            message.next();
          });
        }
      }
    ],
    // CatchAll function (for events that don't match any of the processors)
    message => {
      console.error('No processor found');
      message.next();
    }
  ),
  createQueueProcessor(/* ... */),
  createQueueProcessor(/* ... */)
]);

pool.start();
```

## Documentation

### createQueuePool(processors: QueueProcessor\[]) ⇒ QueuePool

This function creates a queue pool that can be started at the same time and queried to access each item.

The resulting `QueuePool` instance has the following methods:

* `start()` — Start all the queue processor instances
* `findByName(name: String) ⇒ QueueProcessor` — Find a queue processor instance

  by it's name

### createQueueProcessor(options, actionMap, fallbackFn) ⇒ QueueProcessor

* `options` (Object)
  * **name** (String) — ***Required***: name of the remote queue to be watched
  * **region** (String) — the region to send/read service requests. Default is

    `process.env.AWS_REGION`
  * **accessKeyId** (String) — your AWS access key ID. Default is

    `process.env.AWS_ACCESS_KEY`
  * **secretAccessKey** (String) — your AWS secret access key. Default is

    `process.env.AWS_SECRET_KEY`
  * **visibilityTimeout** (Integer) — duration (in seconds) that the received

    messages are hidden from subsequent retrieve requests after being retrieved

    by a ReceiveMessage request.
  * **waitTimeSeconds** (Integer) — duration (in seconds) for which the call

    will wait for a message to arrive in the queue before returning. If a

    message is available, the call will return sooner than WaitTimeSeconds.

    Default is 20
  * **maxNumberOfMessages** (Integer) — maximum number of messages to return.

    Amazon SQS never returns more messages than this value but may return fewer.

    Default is 1
  * **concurrency** (Integer) — number of concurrency fetcher to start. Default

    is 1
  * **debug** (Boolean) — enable debug mode. Default is false
* `actionMap` (Array of `ActionMap`). Each `ActionMap` (Object):
  * **conditions** (Array of Objects) — An array of conditions to meet. All of

    the conditions must be met for the function to be triggered.

    * **path** (String) — Object path (of the received event body)
    * **value** (String) — Value of the object path
  * **action** (Function) — The action function to be called when the conditions are met for a received event. Params:
    * **event** (Object) — The received evemt
      * type (String): default is "Message"
      * data (Unknown): JSON.parsed message.Body or a string (if could not be

        parsed)
      * message (Object): reference to the received message
      * name (String): name of the remote queue
      * url (String): url of the connected queue
      * **deleteMessage() ⇒ Promise** (Function):

        Helper to deleteMessage (or `SQS.deleteMessage()`) when the job is completed.
      * **changeMessageVisibility(timeout) ⇒ Promise** (Function):

        Helper to changeMessageVisibility (or `SQS.changeMessageVisibility()`) when the job is completed.
      * **delay(timeout) ⇒ Promise** (Function):

        Helper to changeMessageVisibility (or `SQS.changeMessageVisibility()`) without completing the job.
      * **sendMessage(params = {}) ⇒ Promise** (Function): send a new message in the queue
      * **next()** (Function): call this method when you've completed your jobs

        in the event callback.
    * **sendMessage(params = {}) ⇒ Promise** (Function) — A shortcut to send a message to the same queue processor that the event came from
* `fallbackFn` — A fallback action in case a message doesn't match any conditions. Signature is same as the `action` mentioned above.


# Welcome To CX Cloud

Welcome to CX Cloud. This site will help you through the journey of setting up and operating a CX Cloud project.

## What is CX Cloud?

CX Cloud is an idea for creating scalable microservices based architecture on AWS. This project is created and maintained by [TietoEvry CX](https://www.tietoevry.com/en/services/digital-experience/customer-experience/) practice. We use this idea, accelerators, architecture and 3rd party tools (e.g. commercetools, Algolia, Contentful) for creating high quality digital services for our customers.

CX Cloud is not a ready-made product, but more of a collection of accelerators (e.g. CI/CD, Data Engine) and best practices to help autonomous project teams to get up and running more quickly. In addition accelerators, its about sharing experiences between teams and also open sourcing (as CX Cloud Utilities) such components that are worth sharing. Join the discussions in our [Slack channel](https://join.slack.com/t/cxcloud/shared_invite/zt-bvn29pqe-tMzv5kDnlj7M68S4pmkRCg).

For additional information, see CX Cloud [implementation scenarios](/introduction/how_to_use) and [architecture](/introduction/architecture-overview). Check also our [Changelog](/introduction/changelog) for quarterly summaries of the most important changes and improvements.

And although our CX Cloud CLI is built for AWS, there is no deep dependency on AWS in CX Cloud in general. You are most welcomed to run CX Cloud with MS Azure or Google Cloud as well.

## Why CX Cloud?

In our approach, we emphasize to find the optimal balance between centralized governance and team level autonomy. In general, we are targeting to enable our customers to use cloud native technologies for optimal flexibility and meeting solution specific needs, but also take long-term sustainability, target architecture and optimal TCO into account when developing digital solutions in scale.

In cloud native space, technology evolution is quite dynamic. Also, solution use cases are versatile and ambitioius. From development perspective, this requires having motivated and autonomous teams and using fit-for-purpose tech stacks. However, to reduce the negative impacts of siloed projects and to embrace reusability and maintainability, some common ground is needed.

For us, CX Cloud is a way to facilitate sharing code, best practices and experiences between teams to reach this common ground. Accelerators are provided as open source to make sure various teams can benefit without cost impacts and also to avoid any unintended vendor lock-in mechanisms when doing so.

## Getting Started

For getting started, check out the [Getting Started guide](/getting-started-1/getting-started). After you set up the basics, you can move forward with [Setting up your own CX Cloud project](/getting-started-1/setting-up-a-cxcloud-project) and start exploring the possibilities.

## Online demo

Front-end: <https://demo.cxcloud.com/>\
Content API: <https://demo.cxcloud.com/api/content/v1/api-docs/> ·\
Search API: <https://demo.cxcloud.com/api/search/v1/api-docs/> ·\
Commerce API: <https://demo.cxcloud.com/api/commerce/v1/api-docs/>

## Source code

CX Cloud components and demo infra and services are available at [GitHub](https://github.com/cxcloud).

## License

CX Cloud and its components are released under the MIT license.


# Why CX Cloud?

CX Cloud is a battle-proven solution accelerator for cloud native architecture. It is based on API-first approach, microservices and cloud infrastructure providing fast time-to-market, agility in development and lack of waste via automation.

Example solutions include web shops, customer self-service solutions, content rich web sites and API-layer for mobile apps and other clients. Solution implementation can also be part of "decomposing the monolith" program where your customer targets to create new online solutions with modern omni-channel architecture while at the same time reducing investments towards legacy platforms.

We have seen many cloud native initiatives fail to reach the high expectations due to siloed approach and lack of governance in the areas of end-to-end feasibility, common services and long term architecture. For us, CX Cloud is a tool to promote also these long-term aspects in addition to enabling fast innovation and benefitting from cloud native approach.

It is good to remember that even if CX Cloud provides guidance and accelerators to create an end-to-end online solution including infra, services and front-end, it is not a ready-made solution. It is more of a framework and its granularity of accelerators enable you to hand pick only the ones you need. For example, you can just use the infra and CI/CD part to get your team up and running in one day and then continue with customizing the rest of the stack.

In high level, there are three main scenarios for usage.

* A green field project to create a new solution or replace existing solution
* Use CX Cloud for a "decomposing the monolith" program
* Add CX Cloud capablities to an existing microservices stack

## 1. "Greenfield"

In this scenario, the solution scope has little dependencies on the existing IT stack. You have quite big freedom in technology choices regarding both services, front-ends, infrastructure, and CI/CD configurations. As the target is quite a "stand-alone" solution stack, no separate API Gateway is necessarily needed in first phase as CX Cloud runs Kubernetes with an internal ingress loadbalancer.

For a greenfield project, you can make a clean start for your project using [Kubernetes Infra Template](https://github.com/cxcloud/demo-cxcloud-k8s) to install the infra with Kubernetes and [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli) and generate services needed for your project in hand.

## 2. "Decomposing the monolith"

Quite similar to the greenfield. But there is a need for separate API Gateway to enable multi-vendor approach (different vendors can expose API's via one gateway). This gateway can also be utilized to expose API's of current legacy platforms for new client/front-end solutions. CX Cloud enables generating and running AWS API Gateway outside the Kubernetes cluster but you can also configure the tool of choice (e.g. WSO2, Azure API Management) separately.

In this case, you can make a clean start for your project using [Kubernetes Infra Template](https://github.com/cxcloud/demo-cxcloud-k8s) to install the infra with Kubernetes and [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli) and generate services needed for your project.

## 3. Add capabilities for your existing microservice stack

In this scenario, there is already a microservices architecture in place, typically including centralized API Gateway, version control, and DevOps pipeline. Your task might be for example to add additional capabilities (e.g. commerce, content, search, process-engine) to the existing stack. And in this area, CX Cloud services might come in handy.

In this case, you can generate required services or service template using [CX Cloud CLI](https://github.com/cxcloud/cxcloud-cli). More information about generating core services can be found here: [Generating Core Services](https://docs.cxcloud.com/setting-up-a-cxcloud-project/generating-core-services).


# Architecture Overview

CXCloud projects are essentially services and functions that are deployed to a Kubernetes cluster. There's nothing magical or different about this architecture and the aim of CXCloud is just to make this process as simple and easy to understand as possible.

![](/files/-LLy1tIm2NxoTjaQHYfy)


# How To Participate?

CX Cloud is developed further also via customer projects. As an example, in one case a simple IAM (identity and access management) solution was needed, so the project team developed a service for AWS Cognito. This was then taken as a core module to CX Cloud and today via CLI anyone can generate a Cognito powered Auth-service to their CX Cloud project.

## Participate in CX Cloud development

## Develop CX Cloud via customer project


# Change Log

## Changelog

We constantly develop Tieto CX Cloud based on customer projects, internal ideas, and partner contributions. Naturally, you can follow progress via Github but here are quarterly summaries of the most important changes and improvements.

### Planned for H2/2020

* Add GraphQL to basic set-up.
* Create more templates to enable custom microservices generation with various technologies (e.g. Node, Java, Groovy) using CX Cloud CLI.
* Add Azure support for infra provisioning.
* Detail CX Cloud Utilities concept

## Changes in 2019

### H1/2019

* CI/CD pipeline between Github and AWS.
* Data Engine 1.0 for log management.
* New core service for Reach marketplace.

## Changes in 2018

### Q4/2018

Created AWS Cognito core service and enabled creation of Auth service with CLI. Improved CX Cloud documentation. Implemented front-end demo based on React.

### Q3/2018

Created CX Cloud CLI to enable more easy project set-up with infra, service and front-end generation. Switched from AWS ECS to Kubernetes.

### Q2/2018

We split core module to more granular modules (e.g. commerce, content, search). Also enabled generation of services and service specific API's with selected level of granularity based on available core modules.

### Q1/2018

Initial release.


# Requirements

Before you set up a CX Cloud project, you need to prepare your environment and configure the tools needed for the process.

Please start by [preparing your environment](/getting-started-1/getting-started/prepare-your-environment) and then move forward installing [CX Cloud CLI](/getting-started-1/getting-started/install-cxcloud-cli).


# Prepare Local Environment

## Install Git client

Currently, our CLI only supports GitHub as a versioning system. Once you have installed Git client of your preference, make sure to configure global `username` and `email` properties with values from your GitHub account like this:

```bash
git config --global user.name "username"
git config --global user.email "example@email.com"
```

## Install and Configure the AWS CLI tools

You naturally need an AWS account. For quick testing, You can create your own [AWS Free Tier account](https://aws.amazon.com/free/). Then install the AWS Cli tools using [this guide](https://docs.aws.amazon.com/cli/latest/userguide/installing.html) and configure a profile:

```bash
$ aws configure --profile myproject
```

Check [this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html) for more information on configuring AWS CLI.

Export your AWS profile to an environment variable. This variable is used by CX Cloud tools to interact with your AWS account:

```bash
$ export AWS_PROFILE=myproject
```

CX Cloud CLI supports preset AWS API keys as environment variables. This is useful for example when running [AWS Vault](https://github.com/99designs/aws-vault) for storing the AWS secrets in your operating system's secure keystore. The `AWS_PROFILE` doesn't need to be exported in case AWS Vault is used.

## Install Kubernetes and Kops

Before you start, you need to have Kubernetes and it's tools installed on your computer. Follow [this guide](https://kubernetes.io/docs/tasks/tools/install-kubectl/) to Install those. On Mac, you can install Kubernetes tools using `homebrew`:

```bash
$ brew install kubernetes-cli
```

CXCloud CLI uses Kops to create and configure your Kubernetes cluster. You can install it on your Mac using `homebrew`:

```bash
$ brew update && brew install kops
```

Or on your Linux machine:

```bash
curl -LO https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x kops-linux-amd64
sudo mv kops-linux-amd64 /usr/local/bin/kops
```

Check [Kops' documentation](https://github.com/kubernetes/kops#installing) for more information.

## Install Helm

[Install the Helm client](https://docs.helm.sh/using_helm/#installing-helm) on your machine. On Mac this can be done using `homebrew`:

```bash
$ brew install kubernetes-helm
```

## Install Terraform

[Install Terraform ](https://www.terraform.io/intro/getting-started/install.html) on your machine. On Mac this can be done using `homebrew`:

```bash
$ brew install terraform
```

## Install Node.js and NPM

You need to install Node.js and NPM on your computer. It is recommended to install the latest Node.js LTS version. On macOS you can do this using `homebrew`:

```
$ brew install node
```

For other systems, you can check [Node.js official documentation](https://nodejs.org/en/download/package-manager/).

Once you have installed Node, you can confirm that NPM is working using `npm --version`

## Install Docker

To be able to compose a deployable image you have to install Docker. Head over to [this page](https://docs.docker.com/install/#supported-platforms) and get the Docker for your system. You'll need to create a Docker account as well.

## Install Ansible

To be able to access the private services in Kubernetes an VPN solution is needed. Ansible is used to install OpenVPN for the CX Cloud Demo.

[Install Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) on your machine. On Mac this can be done using `homebrew`:

```
$ brew install ansible
```


# Install CX Cloud CLI

## Installing CX Cloud CLI Tools

{% hint style="info" %}
Before proceeding, make sure you have prepared your local environment as instructed in [this guide](/getting-started-1/getting-started/prepare-your-environment).
{% endhint %}

Install CX Cloud CLI by running the following command:

```bash
$ npm install -g cxcloud
```

If you need to update CX Cloud CLI, run this command\`:

```
$ npm i -g cxcloud@latest
```

After installation is complete, you can call CX Cloud directly from command line:

```bash
$ cxcloud --version
```

Or if you don't want to install CX Cloud globally, you can call it directly using `npx`:

```
$ npx cxcloud --version
```


# Setting up a CX Cloud Project

Great, you have now [prepared your environment and configured the tools needed](/getting-started-1/getting-started). So the next phase is to set up your own CX Cloud project following these steps:

1. [Provision Infrastructure](/getting-started-1/setting-up-a-cxcloud-project/provision-infrastructure)
2. [Generating core services](/getting-started-1/setting-up-a-cxcloud-project/generating-core-services)
3. [Generating a frontend](/getting-started-1/setting-up-a-cxcloud-project/generating-a-frontend)
4. [Routing manifest](/getting-started-1/setting-up-a-cxcloud-project/generating-a-frontend)
5. [Configuring CI/CD pipeline](/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd)
6. [Configuring Data Engine 1.0 (logging)](/getting-started-1/setting-up-a-cxcloud-project/data-engine)


# Provision Infrastructure

Cx Cloud modules and deployments is developed for Kubernetes, hence, CX Cloud can run on any existing Kubernetes cluster.

The CX Cloud demo setup will install the infra with Kubernetes, running on AWS. When the setup is up and running it can be used for the [CX Cloud Demo app](https://github.com/cxcloud/demo-cxcloud-monorepo-angular) or any other CX Cloud related project.

## Getting stated with CX Cloud Demo

{% hint style="warning" %}
Windows is not supported. You can only provision the infrastructure using a Unix based OS. Eg. macOS or Linux.
{% endhint %}

Make sure you have [installed all the requirements](/getting-started-1/getting-started/prepare-your-environment) on your computer.

### Installation steps

There are few steps that has to be done in order to install the infrastructure. Follow the instructions from the following steps:

1. [Provision the infra](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/terraform/README.md).
2. [Install OpenVPN](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/openvpn-ansible/README.md) in order to access services in the private subnets on AWS.
3. [Install Kubernetes](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/kubernetes/README.md).

{% hint style="info" %}
If you have enabled Multi-Factor Authentication on your AWS account or you are assuming another role which does, you might be asked to enter your MFA token (using your Authenticator app) in this step.
{% endhint %}

{% hint style="info" %}
AWS credentials has to be [configured](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) in order to provision the infra for CX Cloud demo. It is possible to provision with preset API keys as environment variables. This is useful for example when running [AWS Vault](https://github.com/99designs/aws-vault) for storing the secrets in your operating system's secure keystore.
{% endhint %}

### Configuring a domain for your online service

After you have generated the Kubernetes infrastructure, you would want to assign a domain to your future online service (or services)

The assumed starting state is that you have access to manage "example.com" DNS. CX Cloud is to be configured to utilize this domain to explore its possibilities. As there might already be a site running in [www.example.com](http://www.example.com), the target state with CX Cloud would look like this:

* `newsite.example.com` (customer front-end)
* `newsite.example.com/api/service-commerce/v1/` (commerce service API)
* `newsite.example.com/api/service-content/v1/` (content service API)
* `newsite.example.com/api/service-search/v1/` (search service API)
* `newsite.example.com/api/service-auth/v1/` (auth service API)
* `*.dev.newsite.example.com` (Pull Request environments)

We have used this assumption in our documentation and configuration examples. Most of these configurations are done with a routing manifest later and at this stage we only need to point the domain to Kubernetes load balancer like this:

* First figure out your Load Balancer(s) URL. The URL(s) are displayed after generating the infrastructure. But you can also find them using:

```
kubectl get ingress --all-namespaces
```

* Then you have to point your own domain name to this load balancer URL. To do this:
  * If you manage your domain using AWS' route53 service, update the settings in [terraform.tfvars](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/setting-up-a-cxcloud-project/example_vars/terraform.tfvars) and re-run the `terraform apply` command in order to update the domain name aliases. Follow the instructions from [Install Kubernetes](https://github.com/cxcloud/demo-cxcloud-k8s/blob/master/kubernetes/README.md).
  * If you manage your domain using a provider that's not AWS, navigate to your domain's management panel and create an `CNAME` record for `newsite.example.com` pointing to the load balancer URL.
    * [GoDaddy guide](https://fi.godaddy.com/help/add-a-cname-record-19236)
    * [Namecheap guide](https://www.namecheap.com/support/knowledgebase/article.aspx/9646/2237/how-can-i-set-up-a-cname-record-for-my-domain)
    * [Name.com guide](https://www.name.com/support/articles/115004895548-Adding-a-CNAME-Record)


# Generating Core Services

CX Cloud provides selected ready-made (Node.js) core services (e.g. commerce, content, search, authentication) which can be generated and added to the project using CX Cloud CLI. These can, of course, be customized later on.

## Requirements

It is required to use a monorepo structure in order to deploy all microservices with CI/CD pipeline. More information about monorepo development strategy can be found [here](/guides/application-development/create-a-git-repository).

## Service generation

You can start by creating a new folder for a new service in monorepo `packages` folder, for example:

```
my-monorepo
├── packages
|   ├── package-commerce
|   |   ├── .cxcloud.yaml
|   |   └── ...
└── ...
```

Then navigate inside the newly created folder and start service generation by running the following commands:

```bash
$ cd package-commerce
$ cxcloud generate service
```

You will go through a number of questions and get to choose which service to create.

To enable ready-made functionality for services, CX Cloud provides a set of core modules that can be used when generating services. To enable the desired level of granularity, one can use multiple core modules for one service or create multiple services based on the same core module. It should be noted that if generating service with multiple core modules, some services might conflict with each other. In this case, you will get a warning. Example of such conflict is Commerce and Auth services. Commerce includes own authentication service, making Auth in this case redundant.

Currently CLI supports the following core modules:

* Commerce (commercetools)
* Content (Contentful)
* Auth (AWS Cognito)
* Search (Algolia)

## Configuration

After you have generated the service, it's time to provide it with configurations.

Create a `.env` file in the root directory of the monorepo and add environment-specific variables on new lines in the form `NAME=VALUE`. For example:

```
CONTENTFUL_API_KEY=12345
CTP_CLIENT_ID=qwerty-567
AUTH_URL=https://auth.example.com
...
```

{% hint style="warning" %}
**Do Not** commit sensitive information such as API keys to GitHub or other versioning systems. Make sure the `.env` file is included in `.gitignore` of the root folder.
{% endhint %}

In order to provide a consistent configuration interface to the services, which can be extended and overridden during further development, the configurations are stored in configuration files within the service. For these purposes, we are using `node-config`. The generated with CLI services are already prepackaged with `node-config` and basic configuration files. Among other settings, the configuration files include references to the environment variables which are loaded from `.env` file into `process.env`. Documentation for node-config can be found [here](https://www.npmjs.com/package/config).

The configurations below are used in CX Cloud Demo and can be copied and modified in default configuration file `config/default.js` of generated service.

Example configuration for Commerce:

```javascript
"store": {
    "defaultCurrency": "EUR",
    "supportedCurrencies": ["EUR", "USD", "GBP"]
},
"commerceTools": {
    "authHost": `${process.env.CTP_AUTH_URL}`,
    "apiHost": `${process.env.CTP_API_URL}`,
    "projectKey": process.env.CTP_PROJECT_KEY,
    "admin": {
        "clientId": process.env.CTP_CLIENT_ID,
        "clientSecret": process.env.CTP_CLIENT_SECRET
    },
    "user": {
        "clientId": process.env.CTP_USER_CLIENT_ID,
        "clientSecret": process.env.CTP_USER_CLIENT_SECRET
    }
}
```

Example configuration for Content:

```javascript
"contentful": {
    "sdkConfig": {
        "space": process.env.CONTENTFUL_SPACE_ID,
        "accessToken": process.env.CONTENTFUL_DELIVERY_API_TOKEN
    }
}
```

Example configuration for Auth:

```javascript
"cognito": {
    "userPoolId": process.env.USER_POOL_ID,
    "clientId": process.env.CLIENT_ID,
    "encryptionKey": process.env.ENCRYPTION_ID
}
```

Example configuration for Search:

```javascript
"algolia": {
    "applicationId": process.env.ALGOLIA_APPLICATION_ID,
    "apiKey": process.env.ALGOLIA_API_KEY
}
```

{% hint style="info" %}
CXCloud does not provide any testable configurations for 3rd party solutions. You must get test account or purchase licenses for selected services and set them up yourself.
{% endhint %}

## Local test

After generation is finished and configurations are set, the generated service will be working as a single API. To test locally, run the following set of commands at the root of your repository with the service:

```bash
$ npm install
$ npm run build
$ npm run start
```

It will install all required npm packages, build your Swagger documentation and start the local server for you to be able to test your configurations. Swagger provides an interface for testing right in the browser.

In case of `commerce` service and default settings, you should be able to see Swagger API documentation at `http://localhost:4003/api/commerce/v1/api-docs`

## Deployment

CI/CD pipeline handles the deployment of microservices within the monorepo. Therefore, CI/CD pipeline should be set up and configured prior deployment of the services. More information about CI/CD pipeline configuration and monorepo settings for deployment can be found [here](/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd).

{% hint style="info" %}
When applications is deployed, all sensitive information such as keys, secrets or tokens are stored as Kubernetes secret object in the intended namespace. You can find an example of using Kubernetes' secrets to configure your project [here](/guides/managing-infrastructure/storing-secrets-in-kubernetes#example-storing-and-using-secrets-in-nodejs-and-node-config-module).
{% endhint %}

## Routing

After deployment, you can make your service available to the world using a [Routing Manifest](/getting-started-1/setting-up-a-cxcloud-project/routing-manifest).

## Commit

As a final step, add git remote origin corresponding to your monorepo and commit changes to your Github repository by running the following commands:

```bash
$ git remote add origin https://github.com/user/my-monorepo.git
$ git add .
$ git commit -m "initial commit"
$ git push
```


# Generating a Frontend Demo

Cool, You now have infra and core services up and running! For front-end, CX Cloud CLI allows you to generate a front-end demo to test the concept with end-to-end functionality. Currently, we provide Angular 7 demo. React is coming soon.

## Front-end generation

You start by creating a new package repository for the service in git monorepo, named for example `package-frontend`. Then navigate to this folder and start front-end demo generation by running the following commands:

```bash
$ cd package-frontend
$ cxcloud generate demo
```

You will go through a set of simple questions and receive an end-to-end working demo.

## Configuration

By default, the generated demo uses our live test API including end points with pre-filled sets of data for Commerce, Content and Search services.

In order to enable the frontend demo to work with the APIs you have generated and deployed, configuration files need to be modified. Configuration files can be found in this project directory `/src/environments/`.

To connect with new APIs, change the value of `apiUrl` for each service (notice that environments files should be modified depending on the used environment). If frontend and APIs running on the same domain, it is enough to use only relative paths to the APIs:

```javascript
export const environment = {
  production: false,
  siteName: "CXCloud DEV",
  commerce: {
    apiUrl: "/api/service-commerce/v1",
    indexName: "dev_COMMERCE"
  },
  content: {
    apiUrl: "/api/service-content/v1",
    indexName: "dev_CONTENT"
  },
  auth: {
    apiUrl: "/api/service-auth/v1"
  },
  search: {
    apiUrl: "/api/service-search/v1"
  }
};
```

## Local test

In case of Angular demo, to explore it, from inside of the project folder you should run the following command:

```bash
$ npm run dev
```

By default, your demo will be available at `http://localhost:4200/` but check the console logs.

## Deployment

CI/CD pipeline handles the deployment of microservices within the monorepo. Therefore, CI/CD pipeline should be set up and configured prior deployment of the services. More information about CI/CD pipeline configuration and monorepo settings for deployment can be found [here](/getting-started-1/setting-up-a-cxcloud-project/configuring-cicd).

## Routing

After deployment, you can make your service available to the world using a [Routing Manifest](/getting-started-1/setting-up-a-cxcloud-project/routing-manifest).

## Commit

As a final step, add git remote origin corresponding to your monorepo and commit changes to your Github repository by running the following commands:

```bash
$ git remote add origin https://github.com/user/my-monorepo.git
$ git add .
$ git commit -m "initial commit"
$ git push
```


# Configuring Routing Manifest

Routing manifest is needed to make multiple services available in the same domain.

For example, let's assume you have created a front-end and 3 microservices and you want them all to be available on the same domain:

* `newsite.example.com/` should load `frontend` service
* `newsite.example.com/api/service-commerce/` should load `package-commerce` service
* `newsite.example.com/api/service-content` should load `package-content` service
* `newsite.example.com/api/service-search` should load `package-search` service

You have to create a routing manifest to achieve this. (In addition to pointing the domain in question to your Kubernetes cluster during [infra generation](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/setting-up-a-cxcloud-project/generating-infrastructure.md#configuring-a-domain-for-your-online-service))

## Create manifest

Add `.cxcloud.yaml` to the root folder of your monorepo.

{% code title="my-monorepo/.cxcloud.yaml" %}

```yaml
namespace: $GIT_BRANCH
routing:
  domain: $GIT_BRANCH.dev.newsite.example.com
  ingressClass: $INGRESS_CLASS
  lbCert: $LB_CERT
  scheme: $SCHEME
  ssl: false
  rules:
    - path: /api/service-commerce
      serviceName: package-commerce
      servicePort: 4003
    - path: /api/service-content
      serviceName: package-content
      servicePort: 4003
    - path: /api/service-search
      serviceName: package-search
      servicePort: 4003
    - path: /
      serviceName: package-frontend
      servicePort: 80
```

{% endcode %}

Please note that `path` should match the API prefix you defined when generating the service with CLI. By default, during service generation, CX Cloud CLI suggests service name as the API prefix.

After committing changes to Git repository, CI/CD pipeline will deploy changes automatically.

## Test

To test for example commerce service, `newsite.example.com/api/service-commerce/` should return {"health":"OK"} and `newsite.example.com/api/service-commerce/v1/api-docs` should present to you Swagger documentation.


# Configuring CI/CD Pipeline

When using CX Cloud CLI, it is fairly easy for one to both create and then deploy CX Cloud infra, selected services and demo frontends to AWS. This is typically enough for testing and exploration needs. We also have [tools and guidelines](https://github.com/cxcloud/cxcloud-documentation/tree/7585ecd6d3f3a8c408f0919987af56c53decff01/best-practices/how-to-run-infra.md) for DevOps engineer to manually manage your environment.

But when preparing for customer project development phase, creating a CI/CD pipeline between your Github (or the Git host of your choice eg. Bitbucket) and various environments in AWS (or selected cloud provider eg. Azure) is the next thing to do. As this area can be very dependent on customer preferences and existing set up, you probably will customize your pipeline. Regardless, we have created this basic set up.

## Development workflow

The CX Cloud CI/CD pipeline has been designed with the following development workflow in mind.

![Development workflow](/files/-LlbRTyPxB4rifmjBpe0)

1. A new feature branch is created, the pipeline will checkout the code and run unit tests and perform code quality analysis. This step will be repeated every time a new commit is pushed to the branch..
2. When the feature is ready for review and testing a pull request (PR) is created. The pipeline will automatically run tests and code quality analysis. The pipeline will fail in case the tests fails or if the quality analysis doesn't pass the specified quality level. The pipeline will continue with building the micro services and deploy them into a Kubernetes namespace specific for the PR. The pipeline will send notifications to communication channels and publish the web url as a comment to the PR. Now the PR is ready for code review and quality assurance. New commits pushed to the PR will only analyze and deploy the modified projects.
3. When the pull request has passed all steps, it's possible to merge to master. When new code is pushed to master the pipeline will run almost the same steps as for pull requests except this time the pipeline will deploy to the staging environment (or Kubernetes namespace staging).
4. The process has been designed so that master is alway deployable to production. To deploy to production a release tag has to be made. The pipeline will automatically recognize all git tags and any of the tags can be with one click in Jenkins build and deployed to production.

### Requirements for the pipeline

* Kubernetes cluster
* Jenkins with preinstalled plugins
* Monorepo architecture of the micro services

### Configure the services for the pipeline

The Pipeline requires that the development strategy uses a monorepo in order to deploy all services. The directory structure for the monorepo should look like:

```
.
├── packages
|   ├── service 1
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   ├── service 2
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   └── service n
|       ├── .cxcloud.yaml
|       └── ...
├── .cxcloud.yaml
├── Jenkinsfile
└── ...
```

#### Repository root

In the root of the repository should be two files, `Jenkinsfile` and `.cxcloud.yaml`. The pipeline code is in the Jenkinsfile and the `.cxcloud.yaml` file contains the Kubernetes routing for the microservices.

Example of the `.cxcloud.yaml` file in the root of the repository:

```bash
namespace: $GIT_BRANCH
routing:
  domain: $GIT_BRANCH.dev.cxcloud.com
  ingressClass: $INGRESS_CLASS
  lbCert: $LB_CERT
  scheme: $SCHEME
  ssl: false
  rules:
    - path: /graphql
      serviceName: package-graphql-proxy
      servicePort: 80
    - path: /images
      serviceName: package-image-proxy
      servicePort: 80
    - path: /
      serviceName: package-frontend
      servicePort: 80
```

#### Repository services

Every microservice should preferable be in the packages folder but there is no naming convention for the services. Every microservice has to have a `.cxcloud.yaml` in the folder containing the Kuberntes deployments.

Kubernetes secrets should be stored in the namespace they are intended for. However, secrets intended for the PR environments / namespaces should be stored in the applications namespace. The pipeline will copy them from the applications namespace into the specific PR namespace.

Example of the content in the `.cxcloud.yaml` file for a microservice:

```bash
namespace: $GIT_BRANCH
deployment:
  name: $APP_NAME
  image:
    name: $APP_NAME
    repository: $ECR_REPOSITORY
    version: $APP_VERSION
  containerPort: 3000
  port: 80
  replicas: $MIN_REPLICAS
  cpuRequest: $CPU_REQUEST
  nodeSelector:
    kops.k8s.io/instancegroup: $INSTANCE_GROUP
  autoscaling:
    minReplicas: $MIN_REPLICAS
    maxReplicas: $MAX_REPLICAS
    targetAverageUtilization: 75
  env:
    - name: NODE_ENV
      value: production
    - name: CONTENTFUL_MANAGEMENT_API_TOKEN
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_MANAGEMENT_API_TOKEN
    - name: CONTENTFUL_ENVIRONMENT_ID
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_ENVIRONMENT_ID
    - name: CONTENTFUL_SPACE_ID
      valueFrom:
        secretKeyRef:
          name: cxcloud-secret
          key: CONTENTFUL_SPACE_ID
```

## Jenkins

The pipeline has been developed for Jenkins. A customized Jenkins image can be build from our [Jenkins repository](https://github.com/cxcloud/jenkins).

The docker image is tailored for Kubernetes and all required plugins will be preinstalled in the Docker image. The Docker pod itself will not run the CI/CD pipeline but a dependency worker will be launched when needed and perform all the steps for the pipeline. The worker will be terminated after the jobs is finished. Several workers can run at the same time as long as the underlying infra can handle the load.

The Jenkins configuration and installation is in detail explained on the [Jenkins repository](https://github.com/cxcloud/jenkins).

## CI/CD Pipeline

A working example of the CI/CD pipeline is in the CX Cloud demo application, [cxcloud-monorepo-angular](https://github.com/cxcloud/demo-cxcloud-monorepo-angular).

### Jenkinsfile structure

The [Jenkinsfile](https://github.com/cxcloud/demo-cxcloud-monorepo-angular/blob/master/Jenkinsfile) structure for the CX Cloud demo explained here is a guideline and should be modified depending on technologies and deployment strategies used. The important part is that the Jenkinsfile should be configured as a multibranch pipeline in Jenkins and that there are different actions specified e.g. for normal branches, pull requests and base branch.

#### Variables

The pipeline starts with defining the variables for all environments, DEV/TEST (pull requests), staging (base branch, master) and production (git tag).

The first stage in the pipeline, `populate variables` will set the values for many of the variables that has to be calculated while the pipeline is running. E.g. some values for pull requests can only be populated in case the job execution is an actual pull request.

#### Code Quality

The following stage, `run tests` will run the tests. The pipeline will stop in case any of the tests fails. When we work in a branch all tests will run for all projects. However, when a pull requests is created, only tests for modified projects will be executed. When deploying a release tag, none of the tests will run, since all tests had already to pass previously in the base branch.

Stage `SonarQube analysis` will run sonar-scanner and analyse the projects on SonarQube. The pipeline will not check at this stage if the code quality for the projects pass or fail.

Stage `SonarQube quality gate` ensure that all the projects has passed the SonarQube analysis. The pipeline will fail and send notifications in case any of the projects doesn't pass the "quality gate".

#### Create Kubernetes namespace

The pipeline will deploy all of the services into a new dev/test environment or to the staging or production environment. Stage `Create namespace` will create a new kubernetes namespace for the deployment in case it doesn't already exist.

#### Copy namespace secrets

The CX Cloud demo uses the namespace `applications` for storing secrets for the test/dev environments. Stage `Copy namespace secrets to DEV/TEST environment` will copy the secrets from the application namespace into the test/dev environments.

Secrets will only be copied for pull requests. Secrets for other environments (staging and production) has to be created independently of the pipeline.

#### Deploy projects

Stage `Deploy projects` will use the cxcloud command line tool, [cxcloud-cli](https://github.com/cxcloud/cxcloud-cli) to deploy the projects. The command will create docker images, upload them to a private docker registry and deploy the new services into the Kubernetes cluster.

The pipeline will take care of only deploying modified projects unless it's to a new namespace. Also deployments to staging and production will deploy all of the services.

#### Clean up test environments

It is vital to delete old environment since every pull request will create a new dev/test environment. Stage `Cleanup development environments` will delete all dev/test environment for pull requests that has been closed. This stage will only be executed in case any new code will be pushed to the base branch, master.

#### Post actions

The post actions will always run in the end of job. The pipeline for the CX Cloud demo will always notify Flowdock if the execution succeeded or failed. The notification will contain details about the url to the running environment, GitHub url, Jenkins execution url and git commit messages.

The post section can easily be modified to send notifications to other destinations like Slack.

In case the job fails, the post action will delete the namespace if it was created during the same execution.

### Example

Below is a Jenkins screenshot of a deployment to staging for the [CX Cloud demo application](https://github.com/cxcloud/demo-cxcloud-monorepo-angular).

![Deployment to Staging](/files/-LlbRTyZ_fb2_lDj12dE)


# Configuring Data Engine

The CX Cloud Kubernetes cluster provisioned with kops doesn't come out of the box with centralized logging for the Pods or application. Hence, the data engine 1.0 mostly focus on logging but with a data lake possibility.

## Architecture

The architecture can be seen in the picture below. This version only support at the moment AWS since we use AWS services like Kinesis Firehose.

![Data Engine, centralized logging](/files/-LlbTJcRV2rlEd4AGTrw)

* Fluentd stream the stdout logs lines to Kinesis Firehose from all Kubernetes pods.
* Kinesis Firehose load the streaming data into Amazon S3 and Amazon Elasticsearch service.
* Amazon S3 store compressed logs that can be used for backups or for further analysis.
* Amazon Elasticsearch service store the logs that can be easily searched with Kibana, which is a part of the managed service from AWS.

### Fluentd

Fluentd should be deployed to the cluster as a deamon set in order to read all pods.

There is a CX Cloud provided [helm chart](https://github.com/cxcloud/helm-fluentd-kinesis-firehose) for installing Fluentd to the Kubernetes cluster.

To get started:

* Install the repository:

```bash
helm repo add cxcloud-fluentd https://raw.githubusercontent.com/cxcloud/helm-fluentd-kinesis-firehose/master
```

* Update repositories:

```bash
helm repo update
```

* Install the chart with version 0.1.0 and the release name `my-fluentd-release` into the namespace `kube-system`:

```bash
helm install cxcloud-fluentd/helm-fluentd-kinesis-firehose \
  --version 0.1.0 \
  --name my-release \
  --namespace kube-system
```

The helm chart is more in detail documented on the GitHub repository, [helm-fluentd-kinesis-firehose](https://github.com/cxcloud/helm-fluentd-kinesis-firehose).

### Kinesis Firehose, S3 and Elasticsearch

The Fluentd daemonset requires that an AWS account has already been provisioned with a Kinesis Firehose stream and with its data stores (eg. Amazon S3 bucket, Amazon Elasticsearch Service, etc).

Available is a CX Cloud provided Terraform module, [terraform-kinesis-firehose-elasticsearch](https://github.com/cxcloud/terraform-kinesis-firehose-elasticsearch) for helping with the installation of Kinesis Firehose, Amazon S3 bucket and Amazon Elasticsearch Service.

The following example show how the module can be used in Terraform.

```
module "kinesis-firehose-elasticsearch" {
  source                       = "github.com/cxcloud/terraform-kinesis-firehose-elasticsearch?ref=v1.1.0"
  region                       = "eu-west-1"
  es_name                      = "cxcloud"
  es_ver                       = 6.5
  es_instance_type             = "t2.small.elasticsearch"
  es_instance_count            = 1
  es_dedicated_master_enabled  = false
  es_ebs_size                  = 35
  es_snapshot_start_hour       = 23
  es_name_tag                  = "CX Cloud"
  es_whitelisted_ips           = ["34.240.135.89/32"]
  stream_name                  = "cxcloud"
  s3_bucket                    = "sandbox-logging"
  s3_buffer_size               = 10
  s3_buffer_interval           = 60
  s3_compression_format        = "GZIP"
  es_index_name                = "cxcloud"
  es_type_name                 = "logs"
  es_buffering_size            = 10
  es_buffering_interval        = 60
  s3_backup_mode               = "AllDocuments"
  whitelisted_aws_account_arns = ["arn:aws:iam::012345678901:root"]
}
```

The Terraform module is more in detail documented on the GitHub repository, [terraform-kinesis-firehose-elasticsearch](https://github.com/cxcloud/terraform-kinesis-firehose-elasticsearch).


# Infrastructure Operations


# Getting Access To An Already Running Infrastructure

In most real world cases, you haven't probably generated and set up the Kubernetes infrastructure yourself. But you might need to access it to deploy services or change settings.

To do that, you should have access to the AWS account that the infrastructure is running in, and export the `AWS_PROFILE` variable like before:

```bash
$ export AWS_PROFILE=my-cxcloud-dev
```

Now run the following script to download the configuration from the running cluster to your system:

```bash
$ kops export kubecfg somecluster.k8s.local --state=s3://somecluster.k8s.local
```

This command will download the configuration to your system and sets up a Kubernetes `context` for easier use. Now you can run any `kubectl` or `cxcloud` command to interact with your cluster.

## Kubernetes Contexts

In case you have access to more than one Kubernetes cluster, you can use `kubectl config` tool and switch between different contexts.

To see a list of your available contexts run:

```bash
$ kubectl config get-contexts
```

This command will print a list of configured contexts on your system and the currently in use context.

To switch between contexts, run:

```bash
$ kubectl config use-context mycluster.k8s.local
```


# Deleting Deployments

After you have deployed a service using `cxcloud deploy` you can run the following command to delete the deployment and all of it's configurations:

```bash
$ cxcloud deploy --purge
```

This will undo anything that is done by running `cxcloud deploy` except it's namespace. If you wish to destroy the namespace as well, run:

```bash
$ cxcloud deploy --purge-all
```

{% hint style="warning" %}
If multiple services are using the same namespace that you have defined in your manifest, this command will purge all other services as well because it will destroy the namespace.
{% endhint %}


# Managing Secrets

Kubernetes has the capability to store secrets and make them available to your services. You can store different secrets and use them through your CX Cloud deployment environment variables. To store a secret in `applications` namespace (it's the default namespace for CX Cloud apps) run the following command:

```bash
$ kubectl create secret generic prod-db-secret \
    --from-literal=username=produser \
    --from-literal=password=Y4nys7f11 \
    --namespace=applications
```

{% hint style="info" %}
You have to specify which namespace you want your secret to be created in, because only deployments in that namespace will be able to access the secrets.
{% endhint %}

Now when you create a deployment, you can reference your secret:

{% code title=".cxcloud.yaml" %}

```yaml
deployment:
  name: my-backend
  image:
    name: my-backend-image
    repository: YOUR_AWS_ECR_REPOSITORY_URL_HERE
    version: 1.1.1
  containerPort: 8080
  replicas: 2
  env:
    - name: NODE_ENV
      value: production
    - name: PROD_DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: prod-db-secret
          key: username
    - name: PROD_DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: prod-db-secret
          key: password
```

{% endcode %}

When the service is deployed, your secrets will become available under the environment variables you have specified. For example in NodeJS, you can access them like so:

```javascript
console.log(process.env.PROD_DB_USERNAME); // produser
console.log(process.env.PROD_DB_PASSWORD); // Y4nys7f11
```

For more information about Kubernetes secrets, [visit this article](https://kubernetes.io/docs/concepts/configuration/secret/).

## Example: Storing and using secrets in NodeJS and \`node-config\` module

One of the core services that you can generate using the CX Cloud CLI is Commerce service. It is communicating with **commercetools** platform and thus requires some configurations (like API key, etc). We can take advantage of Kubernetes secrets explained above to store these information without having to publish them to GitHub.

First, generate a service using the CLI and choose Commercetools, [as explained in its section](/guides/managing-infrastructure/storing-secrets-in-kubernetes).

Then, according to `node-config`'s [documentation](https://github.com/lorenwest/node-config/wiki/Environment-Variables#custom-environment-variables), create a file named `custom-environment-variables.json` in the `config` folder of the generated service with the following content:

```javascript
{
  "commerceTools": {
    "projectKey": "COMMERCETOOLS_PROJECT_KEY",
    "admin": {
      "clientId": "COMMERCETOOLS_ADMIN_CLIENT_ID",
      "clientSecret": "COMMERCETOOLS_ADMIN_CLIENT_SECRET"
    },
    "user": {
      "clientId": "COMMERCETOOLS_USER_CLIENT_ID",
      "clientSecret": "COMMERCETOOLS_USER_CLIENT_SECRET"
    }
  }
}
```

This file will tell `node-config` to look for those environment variables and map them to certain keys. For example the configuration key `commerceTools.admin.clientId` will map to `COMMRCETOOLS_ADMIN_CLIENT_ID` and so on.

Now we have to store those data in Kubernetes and make them available to our service using the specified environment variables. To do that, first let's create a secret in `applications` namespace:

```bash
$ kubectl create secret generic prod-commercetools \
    --from-literal=projectKey=xxxxxxx \
    --from-literal=adminClientId=xxxxxxx \
    --from-literal=adminClientSecret=xxxxxxx \
    --from-literal=userClientId=xxxxxxx \
    --from-literal=userClientSecret=xxxxxxx \
    --namespace=applications
```

Replace the `xxxxxxx` above with your actual data. A secret will be created.

Now modify your `.cxcloud.yaml` file and add the proper environment variables referencing the secret you just created (new values are added from line 12):

{% code title=".cxcloud.yaml" %}

```yaml
deployment:
  name: $APP_NAME
  image:
    name: $APP_NAME
    repository: xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/newsite.example.com
    version: $APP_VERSION
  containerPort: 4003
  replicas: 2
  env:
    - name: NODE_ENV
      value: production
    - name: COMMERCETOOLS_PROJECT_KEY
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: projectKey
    - name: COMMERCETOOLS_ADMIN_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: adminClientId
    - name: COMMERCETOOLS_ADMIN_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: adminClientSecret
    - name: COMMERCETOOLS_USER_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: userClientId
    - name: COMMERCETOOLS_USER_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: prod-commercetools
          key: userClientSecret
```

{% endcode %}

When done, increase the version in your `package.json` file and run:

```bash
$ cxcloud deploy
```


# Application Development


# Git Repository Conventions

The development strategy required to use a "monorepo" in order to deploy all services using CI/CD pipeline. The repository contains multiple packages (microservices) that can but do not have to be related.

## Project structure

The file structure for the monorepo should look like:

```
my-monorepo
├── packages
|   ├── package 1
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   ├── package 2
|   |   ├── .cxcloud.yaml
|   |   └── ...
|   └── package n
|       ├── .cxcloud.yaml
|       └── ...
├── .cxcloud.yaml
└── ...
```

Every microservice folder should include `.cxcloud.yaml` file with the configuration for Kubernetes.

There is no naming convention for the packages, but it is recommended to add a prefix to the microservice such as `"package-"` or `"service-"`.

In order to simplify and optimize the workflow around development with the multi-package repository, the managing tool such as [Lerna](https://lerna.js.org/) can be in use.


# Process Engine

A simple helper for using multiple AWS SQS queues at the same time. This tool provides an easy way to map incoming events to actions.

## Installation and Usage

```bash
npm install @cxcloud/process-engine
```

```typescript
import {
  createQueueProcessor,
  createQueuePool
} from '@cxcloud/process-engine-core';

const pool = createQueuePool([
  createQueueProcessor(
    {
      name: 'my-sqs-queue',
      concurrency: 2
    },
    [
      {
        conditions: [
          {
            path: 'myEvent.name',
            value: 'someValue'
          },
          {
            path: 'customer.type',
            value: 'gold'
          }
        ],
        action: (message, sendMessage) => {
          console.log('Received Message:', message.data);
          message.deleteMessage().then(() => {
            // Next should be called after each message is processed
            message.next();
          });
        }
      }
    ],
    // CatchAll function (for events that don't match any of the processors)
    message => {
      console.error('No processor found');
      message.next();
    }
  ),
  createQueueProcessor(/* ... */),
  createQueueProcessor(/* ... */)
]);

pool.start();
```

## Documentation

### createQueuePool(processors: QueueProcessor\[]) ⇒ QueuePool

This function creates a queue pool that can be started at the same time and queried to access each item.

The resulting `QueuePool` instance has the following methods:

* `start()` — Start all the queue processor instances
* `findByName(name: String) ⇒ QueueProcessor` — Find a queue processor instance

  by it's name

### createQueueProcessor(options, actionMap, fallbackFn) ⇒ QueueProcessor

* `options` (Object)
  * **name** (String) — ***Required***: name of the remote queue to be watched
  * **region** (String) — the region to send/read service requests. Default is

    `process.env.AWS_REGION`
  * **accessKeyId** (String) — your AWS access key ID. Default is

    `process.env.AWS_ACCESS_KEY`
  * **secretAccessKey** (String) — your AWS secret access key. Default is

    `process.env.AWS_SECRET_KEY`
  * **visibilityTimeout** (Integer) — duration (in seconds) that the received

    messages are hidden from subsequent retrieve requests after being retrieved

    by a ReceiveMessage request.
  * **waitTimeSeconds** (Integer) — duration (in seconds) for which the call

    will wait for a message to arrive in the queue before returning. If a

    message is available, the call will return sooner than WaitTimeSeconds.

    Default is 20
  * **maxNumberOfMessages** (Integer) — maximum number of messages to return.

    Amazon SQS never returns more messages than this value but may return fewer.

    Default is 1
  * **concurrency** (Integer) — number of concurrency fetcher to start. Default

    is 1
  * **debug** (Boolean) — enable debug mode. Default is false
* `actionMap` (Array of `ActionMap`). Each `ActionMap` (Object):
  * **conditions** (Array of Objects) — An array of conditions to meet. All of

    the conditions must be met for the function to be triggered.

    * **path** (String) — Object path (of the received event body)
    * **value** (String) — Value of the object path
  * **action** (Function) — The action function to be called when the conditions are met for a received event. Params:
    * **event** (Object) — The received evemt
      * type (String): default is "Message"
      * data (Unknown): JSON.parsed message.Body or a string (if could not be

        parsed)
      * message (Object): reference to the received message
      * name (String): name of the remote queue
      * url (String): url of the connected queue
      * **deleteMessage() ⇒ Promise** (Function):

        Helper to deleteMessage (or `SQS.deleteMessage()`) when the job is completed.
      * **changeMessageVisibility(timeout) ⇒ Promise** (Function):

        Helper to changeMessageVisibility (or `SQS.changeMessageVisibility()`) when the job is completed.
      * **delay(timeout) ⇒ Promise** (Function):

        Helper to changeMessageVisibility (or `SQS.changeMessageVisibility()`) without completing the job.
      * **sendMessage(params = {}) ⇒ Promise** (Function): send a new message in the queue
      * **next()** (Function): call this method when you've completed your jobs

        in the event callback.
    * **sendMessage(params = {}) ⇒ Promise** (Function) — A shortcut to send a message to the same queue processor that the event came from
* `fallbackFn` — A fallback action in case a message doesn't match any conditions. Signature is same as the `action` mentioned above.