Installing Agents

Kubernetes / Helm

Flowmill agents can easily be installed in Kubernetes environments using Helm. Note that you should already have a flowmill.yaml file customized with an agent key after completing the Prerequisites.

By default, the Helm chart will automatically launch the kernel agents on each Kubernetes node and the Kubernetes collector and AWS collector.

Before starting, make sure you have a valid agent key and flowmill.yaml. This file can be automatically generated by the Flowmill UI in the agent tab.

Open the flowmill.yaml in a text editor. If you did not autogenerate the keyId and secret, enter values for a valid agent key and note the tenantName will be different. Additionally, edit the 'clusterName' to be a value unique in your tenant. This will be used as the "Environment" label in the Flowmill application.

# TODO: Update these values based on your deployment
flowmill:
clusterName: YOUR_CLUSTER_NAME
tenantName: YOUR_TENANT_NAME
# TODO: Log in to https://app.flowmill.com/agents?activeTab=agent-keys
# to generate an agent key, and fill in the values here
agent:
keyId: "TODO_AGENT_KEY"
secret: "TODO_AGENT_SECRET"
# You can also use a kubernetes secret to provide your key
# If this is not null, the chart will ignore the key id and secret settings above
# and use this instead
agentExistingSecret: null

Now its time to install the agents. First, add the chart repo to Helm:

helm repo add flowmill https://charts.flowmill.com \
&& helm repo update

Then to install using Helm 3 do the following using the flowmill.yaml file that you generated from the Agents tab:

kubectl create namespace flowmill \
&& helm install -n flowmill --values flowmill.yaml flowmill flowmill/flowmill-k8s

For Helm 2 do the following:

helm install flowmill/flowmill-k8s --name flowmill-k8s -n flowmill --values flowmill.yaml

To update an existing installation:

helm repo update \
&& helm upgrade flowmill-k8s flowmill/flowmill-k8s -f flowmill.yaml -n flowmill

Finally, confirm everything is working as expected. There should be one k8s-agent per node.

$ kubectl get pods -n flowmill
NAME READY STATUS RESTARTS AGE
flowmill-k8s-agent-4ndcj 1/1 Running 0 15h
flowmill-k8s-agent-kp6v6 1/1 Running 0 15h
flowmill-k8s-agent-mj5qk 1/1 Running 0 15h
flowmill-k8s-collector-577dd97f74-v2qhq 3/3 Running 0 15h

Amazon ECS (Elastic Container Service)

Flowmill also supports running agents in ECS environments. In this environment, the Flowmill kernel agent collects metadata directly from the local Docker daemon on each instance and does not create load on the ECS API. These instructions will create an ECS task for the kernel collector and the AWS collector, along with services for each.

Again it is recommended to download task definitions with autogenerated agent keys. However, it is also possible to edit the definitions with existing agent keys.

Next open the task definition for the AWS and kernel collectors.

  • Update the FLOWMILL_AUTH_KEY_ID and FLOWMILL_AUTH_SECRET if needed

  • Note that the FLOWMILL_INTAKE_NAME may be different based on your tenant.

  • Set FLOWMILL_AGENT_LABELS_ENVIRONMENT to a value unique to this ECS cluster. This will be used as the environment label in the Flowmill application.

Next, register the task definitions

aws ecs register-task-definition --cli-input-json file://flowmill-kernel-collector-task-definition.json
aws ecs register-task-definition --cli-input-json file://flowmill-aws-collector-task-definition.json

Create the associated services

aws ecs create-service --cli-input-json file://flowmill-kernel-collector-service.json
aws ecs create-service --cli-input-json file://flowmill-aws-collector-service.json

Docker

The Flowmill kernel collector and AWS collector can also be run directly through Docker.

To run the Flowmill kernel collector, customize the following commands below. Note that the flowmill-agent (ie. kernel collector) must be run on each node while one aws collector should be run per AWS account.

  • Update the FLOWMILL_AUTH_KEY_ID and FLOWMILL_AUTH_SECRET with values from your agent key

  • Update FLOWMILL_INTAKE_NAME based on your tenant name

sudo docker run \
--env FLOWMILL_AUTH_KEY_ID="TODO_KEY_ID_GOES_HERE" \
--env FLOWMILL_AUTH_SECRET="TODO_SECRET_GOES_HERE" \
--env FLOWMILL_INTAKE_NAME="flowmill.intake.flowmill.com" \
--env FLOWMILL_INTAKE_HOST="intake.flowmill.com" \
--env FLOWMILL_AUTHZ_SERVER="app.flowmill.com" \
--env FLOWMILL_INTAKE_PORT=443 \
--privileged \
--pid host \
--network host \
--volume /sys/fs/cgroup:/hostfs/sys/fs/cgroup \
--volume /usr/src:/var/run/flowmill/host/usr/src \
--volume /lib/modules:/var/run/flowmill/host/lib/modules \
--volume /etc:/var/run/flowmill/host/etc \
--volume /var/cache:/var/run/flowmill/host/cache \
--volume /var/run/docker.sock:/var/run/docker.sock \
--name flowmill-agent \
flowmill/agent \
--log-console
sudo docker run \
--env FLOWMILL_AUTH_KEY_ID="TODO_KEY_ID_GOES_HERE" \
--env FLOWMILL_AUTH_SECRET="TODO_SECRET_GOES_HERE" \
--env FLOWMILL_INTAKE_NAME="flowmill.intake.flowmill.com" \
--env FLOWMILL_INTAKE_HOST="intake.flowmill.com" \
--env FLOWMILL_AUTHZ_SERVER="app.flowmill.com" \
--env FLOWMILL_INTAKE_PORT=443 \
--name flowmill-aws-collector \
flowmill/aws-collector \
--log-console

Verifying Agents

The Flowmill agent tab provides real time information on agents that have connected.