parsing kubernetes audit log

[jvrahav]

Hi Folks,

I'm trying to parse kubernetes audit logs and extract few fields from the logs such as User, Verb, Request Message, Response Message, Object URI. I created Nest filters to lift the nested json in the event, however i'm unsure how i can only fetch specific fields from the lifted json.

• One method that works is to use Remove_key but that becomes difficult as i need to specify every field that i dont want.
• I tried setting wildcard but that did not have any effect

So the current approach that works is to lift the json and then have a record modifier to remove the keys that are not required.

I would like to know if a simpler config is possible to parse the audit logs

A sample config

filters-custom.conf: |
   [FILTER]
     Name           record_modifier
     Alias          audit_apiserver_modifier
     Match          audit.apiserver.*
     Record         hostname ${NODE_NAME}
     Record         appname audit-kube-apiserver
     Record         filename kube-apiserver.log


   [FILTER]
     Name           modify
     Match          *
     Rename         log text
   
   [FILTER]
     Name               nest
     Match              audit.apiserver.*
     Operation          lift
     Nest_under         user
     Add_prefix         log_
     Wildcard           username

   [FILTER]
     Name               record_modifier
     Match              audit.apiserver.*
     Remove_key         log_groups
     Remove_key         log_uid
     Remove_key         log_extra


   [FILTER]
     Name               nest
     Match              audit.apiserver.*
     Operation          lift
     Nest_under         objectRef
     Add_prefix         object_
     Wildcard           resource
     Wildcard           name
     Wildcard           namespace

   [FILTER]
     Name               record_modifier
     Match              audit.apiserver.*
     Remove_key         object_uid
     Remove_key         object_resourceversion
     Remove_key         sourceips
     Remove_key         useragent

  
   [FILTER]
     Name               nest
     Match              audit.apiserver.*
     Operation          lift
     Nest_under         requestObject
     Add_prefix         request_
     Wildcard           kind
     Wildcard           reason
     Wildcard           message

   [FILTER]
     Name               record_modifier
     Match              audit.apiserver.*
     Remove_key         request_metadata
     Remove_key         request_spec

   [FILTER]
     Name               nest
     Match              audit.apiserver.*
     Operation          lift
     Nest_under         responseObject
     Add_prefix         response_
     Wildcard           reason
     Wildcard           message

   [FILTER]
     Name               record_modifier
     Match              audit.apiserver.*
     Remove_key         response_metadata
     Remove_key         response_spec
     Remove_key         response_items


   [FILTER]
     Name           record_modifier
     Alias          systemd_modifier
     Match          system-bundle.systemd.*
     Record         hostname ${NODE_NAME}
     Record         appname systemd
     Record         filename systemd-jvr

An example event

{
	"kind": "Event",
	"apiVersion": "audit.k8s.io/v1",
	"level": "RequestResponse",
	"auditID": "a6029022-4ff0-4c54-97ed-4099d0ca1923",
	"stage": "ResponseComplete",
	"requestURI": "/api/v1/namespaces/default/serviceaccounts?fieldManager=kubectl-create",
	"verb": "create",
	"user": {
		"username": "kubernetes-admin",
		"groups": ["system:masters", "system:authenticated"]
	},
	"sourceIPs": ["172.31.22.88"],
	"userAgent": "kubectl/v1.23.6 (linux/amd64) kubernetes/ad33385",
	"objectRef": {
		"resource": "serviceaccounts",
		"namespace": "default",
		"name": "test",
		"apiVersion": "v1"
	},
	"responseStatus": {
		"metadata": {},
		"code": 201
	},
	"requestObject": {
		"kind": "ServiceAccount",
		"apiVersion": "v1",
		"metadata": {
			"name": "test",
			"creationTimestamp": null
		}
	},
	"responseObject": {
		"kind": "ServiceAccount",
		"apiVersion": "v1",
		"metadata": {
			"name": "test",
			"namespace": "default",
			"uid": "d6ea858a-206d-4b4a-aca0-499e22f00729",
			"resourceVersion": "1676",
			"creationTimestamp": "2022-07-31T08:36:48Z"
		}
	},
	"requestReceivedTimestamp": "2022-07-31T08:36:48.679291Z",
	"stageTimestamp": "2022-07-31T08:36:48.684377Z",
	"annotations": {
		"authorization.k8s.io/decision": "allow",
		"authorization.k8s.io/reason": ""
	}
}

[patrick-stephens]

I mean, I'd just do a LUA filter to do it all in one pass with any custom behaviour you need.

I think Falco also did some demos and blog posts around integrating audit logs with Fluent Bit.