.png)
Runtime Enforcement
So far in the eBPF Journey applied to InterSystems Workloads, we've been pretty much read only when it comes to system calls, binary execution, and file monitoring. But much like the Network Security Policies that were in play with the last post that enforce connectivity, what if we can enforce system calls, file access, and processes in the same manner across an entire cluster ?
Enter, Tetragon, a flexible Kubernetes-aware security observability and runtime enforcement tool that applies policy and filtering directly with eBPF, allowing for reduced observation overhead, tracking of any process, and real-time enforcement of policies.
Enforcement when your application cant provide it.
Where it Runs
Observability and Enforcement Cluster Wide
.png)
Up and Running
The obligatory steps to get up and running if you chose to do so, performed in the style of an Isovalent Lab.
Cluster
Kind cluster, 3 worker nodes wide, without a default CNI.
cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
networking:
disableDefaultCNI: true
EOF
Cilium
Install Cilium, if for nother else, a CNI.
cilium install version 1.16.2
Tetragon
Here we install the star of our show, Tetragon, as a daemon set.
EXTRA_HELM_FLAGS=(--set tetragon.hostProcPath=/proc)
helm repo add cilium https://helm.cilium.io
helm repo update
helm install tetragon ${EXTRA_HELM_FLAGS[@]} cilium/tetragon -n kube-system
kubectl rollout status -n kube-system ds/tetragon -w
IRIS Workload
Quick IRIS pod, not privileged, but easily modified to be so. This is the pod we will be executing things on to explain some of the tracing policy behavior.
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: iris-nopriv
labels:
app: iris
spec:
imagePullSecrets:
- name: isc-pull
containers:
- name: iris-priv
image: containers.intersystems.com/intersystems/iris-community:2024.2
ports:
- containerPort: 80
EOF
Tracing Policies
TracingPolicies are custom resources that make it easy to setup real-time filters for kernel events. A TracingPolicy matches and filters system calls for observability and also triggers an action on these matches.
Right out of the box though, process_exec and process_exit without having to load any tracing policies.
In one terminal, execute your ZF, in the other, examine the Tetragon events:
kubectl exec -ti -n kube-system tetragon-sw9k4 -c tetragon -- tetra getevents -o compact --pods iris-nopriv
If we take a look at the process execution in Tetragon for the following call out that prints the current working directory.
This may be obvious to you, but running ZF with the "/SHELL" argument, invokes bash, and then calls the command, where as when it is ommitted, it calls out to the binary directly. Now we used the compact output in the above, but if you observe the events in json format, you can see how they are called differently, with the /shell option having a parent process.
ie:
"cwd": "/usr/irissys/bin",
"binary": "/usr/irissys/bin/irisdb",
"arguments": "-w /home/irisowner -s /usr/irissys/mgr",
"flags": "execve",
{
"process_exec": {
"process": {
"exec_id": "a2luZC13b3JrZXIyOjI3OTI5NDExNjY1NzMzMzozMDA4MzAw",
"pid": 3008300,
"uid": 51773,
"cwd": "/usr/irissys/mgr/user",
"binary": "/usr/bin/pwd",
"flags": "execve clone",
"start_time": "2024-10-01T02:39:42.761250745Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "iris-priv",
"container": {
"id": "containerd://5f5df2f9ff01fc737c88f83254ca37c0c214ff7a648c9eabd5f01edcc0804e56",
"name": "iris-priv",
"image": {
"id": "containers.intersystems.com/intersystems/iris-community@sha256:493c073cc968f1053511e6cf56301767ab304f21a60afdf65543e56ef1217cb4",
"name": "containers.intersystems.com/intersystems/iris-community:2024.2"
},
"start_time": "2024-10-01T02:12:16Z",
"pid": 68242
},
"pod_labels": {
"app": "iris"
},
"workload": "iris-priv",
"workload_kind": "Pod"
},
"docker": "5f5df2f9ff01fc737c88f83254ca37c",
"parent_exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNzQ1NzA1ODoyOTc4MzQ0",
"tid": 3008300
},
"parent": {
"exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNzQ1NzA1ODoyOTc4MzQ0",
"pid": 2978344,
"uid": 51773,
"cwd": "/usr/irissys/bin",
"binary": "/usr/irissys/bin/irisdb",
"arguments": "-w /home/irisowner -s /usr/irissys/mgr",
"flags": "execve",
"start_time": "2024-10-01T02:15:21.272075833Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "iris-priv",
"container": {
"id": "containerd://5f5df2f9ff01fc737c88f83254ca37c0c214ff7a648c9eabd5f01edcc0804e56",
"name": "iris-priv",
"image": {
"id": "containers.intersystems.com/intersystems/iris-community@sha256:493c073cc968f1053511e6cf56301767ab304f21a60afdf65543e56ef1217cb4",
"name": "containers.intersystems.com/intersystems/iris-community:2024.2"
},
"start_time": "2024-10-01T02:12:16Z",
"pid": 67506
},
"pod_labels": {
"app": "iris"
},
"workload": "iris-priv",
"workload_kind": "Pod"
},
"docker": "5f5df2f9ff01fc737c88f83254ca37c",
"parent_exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNjUzMzA5NzoyOTc4MzQ0",
"tid": 2978344
}
},
"node_name": "kind-worker2",
"time": "2024-10-01T02:39:42.761250372Z"
}
{
"process_exec": {
"process": {
"exec_id": "a2luZC13b3JrZXIyOjI3OTAxMjk3OTY2NTcxMTozMDA1Mzcy",
"pid": 3005372,
"uid": 51773,
"cwd": "/usr/irissys/mgr/user",
"binary": "/bin/bash",
"arguments": "-p -c pwd",
"flags": "execve clone",
"start_time": "2024-10-01T02:35:01.624258293Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "iris-priv",
"container": {
"id": "containerd://5f5df2f9ff01fc737c88f83254ca37c0c214ff7a648c9eabd5f01edcc0804e56",
"name": "iris-priv",
"image": {
"id": "containers.intersystems.com/intersystems/iris-community@sha256:493c073cc968f1053511e6cf56301767ab304f21a60afdf65543e56ef1217cb4",
"name": "containers.intersystems.com/intersystems/iris-community:2024.2"
},
"start_time": "2024-10-01T02:12:16Z",
"pid": 68097
},
"pod_labels": {
"app": "iris"
},
"workload": "iris-priv",
"workload_kind": "Pod"
},
"docker": "5f5df2f9ff01fc737c88f83254ca37c",
"parent_exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNzQ1NzA1ODoyOTc4MzQ0",
"tid": 3005372
},
"parent": {
"exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNzQ1NzA1ODoyOTc4MzQ0",
"pid": 2978344,
"uid": 51773,
"cwd": "/usr/irissys/bin",
"binary": "/usr/irissys/bin/irisdb",
"arguments": "-w /home/irisowner -s /usr/irissys/mgr",
"flags": "execve",
"start_time": "2024-10-01T02:15:21.272075833Z",
"auid": 4294967295,
"pod": {
"namespace": "default",
"name": "iris-priv",
"container": {
"id": "containerd://5f5df2f9ff01fc737c88f83254ca37c0c214ff7a648c9eabd5f01edcc0804e56",
"name": "iris-priv",
"image": {
"id": "containers.intersystems.com/intersystems/iris-community@sha256:493c073cc968f1053511e6cf56301767ab304f21a60afdf65543e56ef1217cb4",
"name": "containers.intersystems.com/intersystems/iris-community:2024.2"
},
"start_time": "2024-10-01T02:12:16Z",
"pid": 67506
},
"pod_labels": {
"app": "iris"
},
"workload": "iris-priv",
"workload_kind": "Pod"
},
"docker": "5f5df2f9ff01fc737c88f83254ca37c",
"parent_exec_id": "a2luZC13b3JrZXIyOjI3NzgzMjYyNjUzMzA5NzoyOTc4MzQ0",
"tid": 2978344
}
},
"node_name": "kind-worker2",
"time": "2024-10-01T02:35:01.624257883Z"
}
The JSON events get sent to the tetragon log and can be sent to a SIEM system or observability for actionable insights.
Runtime Enforcement
This may lack a little bit of imagination for a use case, but what if we wanted to forbid anybody from calling out and "catting" the license file?
For this, we need to apply a TracingPolicy, that enforces an matchAction These policies are a little involved, but this one is the long way of saying "Hey, if you run `cat /usr/irissys/mgr/iris.key`, I am going to kill you (SIGKILL you).
kubectl apply -f - <<EOF
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "iris-read-file-sigkill"
spec:
kprobes:
- call: "fd_install"
syscall: false
return: false
args:
- index: 0
type: int
- index: 1
type: "file"
selectors:
- matchPIDs:
- operator: NotIn
followForks: true
isNamespacePID: true
values:
- 1
matchArgs:
- index: 1
operator: "Prefix"
values:
- "/usr/irissys/secrets/"
- "/usr/irissys/mgr/iris.key"
matchActions:
- action: FollowFD
argFd: 0
argName: 1
- call: "__x64_sys_close"
syscall: true
args:
- index: 0
type: "int"
selectors:
- matchActions:
- action: UnfollowFD
argFd: 0
argName: 0
- call: "__x64_sys_read"
syscall: true
args:
- index: 0
type: "fd"
- index: 1
type: "char_buf"
returnCopy: true
- index: 2
type: "size_t"
selectors:
- matchActions:
- action: Sigkill
EOF
Once deployed, you should see it loaded as a TracingPolicy resource:
So lets see it enforce the policy:
.png)
The -1 tells us something is awry, and the command was unsuccessful.
But not known to the fellow brogrammer, we administratively blocked it and sent a SIGKILL to the process!
.png)
That is going to be a long call to the WRC for the unsuspecting end user (or wrc specialist).
Experiments
I found a couple that were interesting in the hundreds I stole, applied, and played around with, notable was one that gave up the system calls per binary. If you really wanted to nerd out, you could literally block by syscall.
Another one that was mesmerizing was the file access TracingPolicy, which showed all processes accessing all the files.
These and other polices can be found in the examples repo @ tetragon:
- System calls
- Process attributes
- Command-line arguments
- Network activity
- File system operations
