Use Cases
Getting Started
Integrations
Appendix
Flow Logs
Installing Agents
Note: Customized deployment instructions with your agent keys already filled in can be found at the Deployment page in the Flowmill UI.
Kubernetes / Helm
Flowmill agents can easily be installed in Kubernetes environments using Helm.
By default, the Helm chart will automatically launch the kernel agents on each Kubernetes node and the Kubernetes collector.
Before starting, make sure you have a valid agent key. Agent keys can be created in the Flowmill UI.
First, add the chart repo to Helm:
1
helm repo add flowmill https://charts.flowmill.com \
2
&& helm repo update
Copied!
Then to install using Helm 3 do the following using the agent key/secret that you generated from the Agents tab:
1
kubectl create namespace flowmill \
2
&& helm install flowmill flowmill/flowmill \
3
--namespace flowmill \
4
--set "auth.keyId=<AGENT_KEY_ID_GOES_HERE>" \
5
--set "auth.keySecret=<AGENT_KEY_SECRET_GOES_HERE>" \
6
--set "flowmill.clusterName=default"
Copied!
For Helm 2 do the following:
1
kubectl create namespace flowmill \
2
&& helm install flowmill/flowmill \
3
--namespace flowmill \
4
--name flowmill \
5
--set "auth.keyId=<AGENT_KEY_ID_GOES_HERE>" \
6
--set "auth.keySecret=<AGENT_KEY_SECRET_GOES_HERE>" \
7
--set "flowmill.clusterName=default"
Copied!
To update an existing installation:
1
helm repo update \
2
&& helm upgrade flowmill flowmill/flowmill \
3
--namespace flowmill \
4
--set "auth.keyId=<AGENT_KEY_ID_GOES_HERE>" \
5
--set "auth.keySecret=<AGENT_KEY_SECRET_GOES_HERE>" \
6
--set "flowmill.clusterName=default"
Copied!
Finally, confirm everything is working as expected. There should be one k8s-agent per node.
1
$ kubectl get pods -n flowmill
2
NAME READY STATUS RESTARTS AGE
3
flowmill-k8s-agent-tmxct 1/1 Running 0 146m
4
flowmill-k8s-agent-tpt2p 1/1 Running 0 147m
5
flowmill-k8s-agent-wlfrc 1/1 Running 0 146m
6
flowmill-k8s-aws-collector-6958dbf488-44qgw 1/1 Running 0 148m
7
flowmill-k8s-k8s-collector-58f755b69f-xd6b4 2/2 Running 0 148m
Copied!
For more options to configure your Flowmill installation using Helm, please see the Helm Chart's README for a list of configurable values.
Amazon ECS (Elastic Container Service)
Flowmill also supports running agents in ECS environments. In this environment, the Flowmill kernel agent collects metadata directly from the local Docker daemon on each instance and does not create load on the ECS API. These instructions will create an ECS task for the kernel collector and the AWS collector, along with services for each.
Again it is recommended to download task definitions with autogenerated agent keys. However, it is also possible to edit the definitions with existing agent keys.
Next open the task definition for the AWS and kernel collectors.
- Update the FLOWMILL_AUTH_KEY_ID and FLOWMILL_AUTH_SECRET if needed
- Set FLOWMILL_AGENT_LABELS_ENVIRONMENT to a value unique to this ECS cluster. This will be used as the environment label in the Flowmill application.
Next, register the task definitions
1
aws ecs register-task-definition --cli-input-json file://flowmill-kernel-collector-task-definition.json
2
aws ecs register-task-definition --cli-input-json file://flowmill-aws-collector-task-definition.json
Copied!
Create the associated services
1
aws ecs create-service --cli-input-json file://flowmill-kernel-collector-service.json
2
aws ecs create-service --cli-input-json file://flowmill-aws-collector-service.json
Copied!
Docker
The Flowmill kernel collector and AWS collector can also be run directly through Docker.
To run the Flowmill kernel collector, customize the following commands below. Note that the flowmill-agent (ie. kernel collector) must be run on each node while one aws collector should be run per AWS account.
- Update the FLOWMILL_AUTH_KEY_ID and FLOWMILL_AUTH_SECRET with values from your agent key
1
sudo docker run \
2
--env FLOWMILL_AUTH_KEY_ID="TODO_KEY_ID_GOES_HERE" \
3
--env FLOWMILL_AUTH_SECRET="TODO_SECRET_GOES_HERE" \
4
--env FLOWMILL_INTAKE_HOST="intake.flowmill.com" \
5
--env FLOWMILL_AUTHZ_SERVER="app.flowmill.com" \
6
--env FLOWMILL_INTAKE_PORT=443 \
7
--privileged \
8
--pid host \
9
--network host \
10
--volume /sys/fs/cgroup:/hostfs/sys/fs/cgroup \
11
--volume /usr/src:/var/run/flowmill/host/usr/src \
12
--volume /lib/modules:/var/run/flowmill/host/lib/modules \
13
--volume /etc:/var/run/flowmill/host/etc \
14
--volume /var/cache:/var/run/flowmill/host/cache \
15
--volume /var/run/docker.sock:/var/run/docker.sock \
16
--name flowmill-agent \
17
flowmill/agent \
18
--log-console
Copied!
1
sudo docker run \
2
--env FLOWMILL_AUTH_KEY_ID="TODO_KEY_ID_GOES_HERE" \
3
--env FLOWMILL_AUTH_SECRET="TODO_SECRET_GOES_HERE" \
4
--env FLOWMILL_INTAKE_HOST="intake.flowmill.com" \
5
--env FLOWMILL_AUTHZ_SERVER="app.flowmill.com" \
6
--env FLOWMILL_INTAKE_PORT=443 \
7
--name flowmill-aws-collector \
8
flowmill/aws-collector \
9
--log-console
Copied!
Nomad
Flowmill agents can be run in the Nomad environment as well. The jobspec below can be used to get started. You will need to fill in the "TODO" items below.
1
# nomad jobspec for flowmill-kernel-collector
2
​
3
job "flowmill-kernel-collector" {
4
datacenters = [
5
"<TODO: DATA CENTERS WHERE THIS SHOULD BE DEPLOYED>",
6
]
7
group "kernel-collector" {
8
task "kernel-collector" {
9
driver = "docker"
10
env {
11
FLOWMILL_CLUSTER_NAME="<TODO: YOUR CLUSTER NAME GOES HERE>"
12
FLOWMILL_AGENT_CLUSTER="<TODO: YOUR CLUSTER NAME GOES HERE>"
13
FLOWMILL_AUTH_KEY_ID="<TODO: AGENT KEY ID>"
14
FLOWMILL_AUTH_SECRET="<TODO: AGENT SECRET>"
15
FLOWMILL_INTAKE_HOST="intake.flowmill.com"
16
FLOWMILL_INTAKE_PORT="443"
17
FLOWMILL_AUTHZ_SERVER="app.flowmill.com"
18
FLOWMILL_CRASH_COLLECTOR_HOST="app.flowmill.com"
19
FLOWMILL_KERNEL_HEADERS_AUTO_FETCH="true"
20
FLOWMILL_DISABLE_HTTP_METRICS="false"
21
}
22
config {
23
image = "flowmill/agent:latest-v0.8"
24
args = [
25
"--log-console",
26
"--force-docker-metadata",
27
]
28
volumes = [
29
"/etc:/var/run/flowmill/host/etc",
30
"/lib/modules:/var/run/flowmill/host/lib/modules",
31
"/sys/fs/cgroup:/hostfs/sys/fs/cgroup",
32
"/usr/src:/var/run/flowmill/host/usr/src",
33
"/var/cache:/var/run/flowmill/host/cache",
34
"/var/run/docker.sock:/var/run/docker.sock",
35
]
36
privileged = true
37
network_mode = "host"
38
pid_mode = "host"
39
}
40
resources {
41
cpu = 1500 # 1500 MHz
42
memory = 500 # 500 MB
43
}
44
}
45
count = 1
46
restart {
47
attempts = 2
48
interval = "30m"
49
delay = "15s"
50
mode = "delay"
51
}
52
ephemeral_disk {
53
size = 300
54
}
55
}
56
type = "system"
57
update {
58
max_parallel = 1
59
min_healthy_time = "10s"
60
healthy_deadline = "3m"
61
progress_deadline = "10m"
62
auto_revert = true
63
canary = 0
64
}
65
}
Copied!
Verifying Agents
Last modified 1yr ago