Once collector has been restarted, the netobserve ebpf-agent won't reconnect again.

[WENPIN1]

Hi all,
Once collector has been restarted, the netobserv ebpf-agent won't reconnect again.
Do we have any reconnect or retry mechanism on netobserv-ebpf-agent

ebpf-agent-1 | time="2025-06-18T03:57:36Z" level=error msg="Failed in send IPFIX data record" collector="1.xxx.xxx.xxx:9995" component=exporter/IPFIXProto error="error when sending message on the connection: write udp 1.xxx.xxx.xxx:44136->1.xxx.xxx.xxxx:9995: use of closed network connection"

Thank you very much!

[jotak]

Hi @WENPIN1 ,
Unfortunately there are known issues indeed with the IPFIX export from the ebpf-agent, and it's currently unmaintained (see this mention in the README:

We don't recommend using the agent's IPFIX exporter mode as it is not actively maintained (if you're interested in maintaining it, let us know!). Note that flowlogs-pipeline can also generate IPFIX exports, so a valid way to get IPFIX data is to export to flowlogs-pipeline (via GRPC, Kafka or direct-flp) and then configure IPFIX within flowlogs-pipeline.

Unless we find someone willing to maintain it, we'll probably remove it sooner or later.

So, the proposed alternative for IPFIX is to use the "direct-flp" mode, and use IPFIX export mode from the flowlogs-pipeline configuration.
A config like that should work:

flp-config.json

{
	"pipeline":[
		{"name": "writer","follows": "preset-ingester"}
	],
	"parameters":[
		{"name": "writer","write": {"type": "ipfix", "ipfix": {
                        "targetHost": "...",
                        "targetPort": "..."
                }}}
	]
}

See the freshly updated doc: https://github.com/netobserv/flowlogs-pipeline/blob/main/docs/api.md#write-ipfix (I just realized it was missing)

[WENPIN1]

After switching the export mode to flp-direct and using FLP_CONFIG as above, the issue happens as the same once I restart the collector. It seems that flowlog-pipeline won't reconnect to the collector once the connection has been closed for a while.

EXPORT="direct-flp"
FLP_CONFIG='{"pipeline":[{"name":"writer","follows":"preset-ingester"}],"parameters":[{"name":"writer","write":{"type":"ipfix","ipfix":{"targetHost":"127.0.0.1","targetPort":2055,"transport":"udp"}}}]}'

ebpf-agent-1 | time="2025-06-19T03:33:59Z" level=error msg="Failed in send v4 IPFIX record" component=write.Ipfix error="error when sending message on the connection: write udp 127.0.0.1:44936->127.0.0.1:2055: use of closed network connection"
ebpf-agent-1 | time="2025-06-19T03:33:59Z" level=debug msg="1 flows evicted" component=flow.MapTracer

The ipfix collecor is Elasticsearch filbeat netflow module which listen on UDP port 2055.
My configuration as follows;

docker-compose.yaml

services:
ebpf-agent:
cap_add:
# - SYS_ADMIN
- BPF
- PERFMON
- NET_ADMIN
environment:
- LOG_LEVEL=debug
env_file:
- env_flp_ipfix
image: netobserv-ebf-agent
ulimits:
memlock:
soft: -1
hard: -1
privileged: true
user: 0:0
network_mode: "host"
volumes:
- /sys/:/sys/
restart: always

---

content of 'env_flp_ipfix'
EXPORT="direct-flp"
#FLP_CONFIG='{"pipeline":[{"name": "writer","follows": "preset-ingester"}],"parameters":[{"name": "writer","write": {"type": "stdout","stdout": {"format":"json"}}}]}'
ENABLE_RTT=true
ENABLE_PKT_DROPS=true
ENABLE_DNS_TRACKING=true
METRICS_ENABLED=false
SAMPLING=10
FLP_CONFIG='{"pipeline":[{"name":"writer","follows":"preset-ingester"}],"parameters":[{"name":"writer","write":{"type":"ipfix","ipfix":{"targetHost":"127.0.0.1","targetPort":2055,"transport":"udp"}}}]}'

Thank you!

[WENPIN1]

One significant issue is that 'direct-flp' mode exports less information than "ipfix+udp" mode.
For examples; icmp_type_ipv4, icmp_code_ipv4, tcp_control_bits during the export processing.

[jotak]

FYI I'm working on a fix with IPFIX in flowlogs-pipeline: the two that you mentioned, and also an error wrongly reported when ports are missing (e.g. icmp)

[jotak]

netobserv/flowlogs-pipeline#1019 and netobserv/netobserv-ebpf-agent#749

@WENPIN1 I haven't tested yet, will do so probably tomorrow - you can test yourself in the meantime, with this image: quay.io/netobserv/netobserv-ebpf-agent:a5ffcd9

[WENPIN1]

Hi jotak,

It may reconnect to the IPFIX data collector but this version (netobserv-ebpf-agent:a5ffcd9) doesn't send any measurement values just only the templates.

It seems like a transition issue between v1.8.2-community and above.

Thank you very much!
-WENPIN

[jotak]

I've fixed a couple more things in my last commits. New image is: quay.io/netobserv/netobserv-ebpf-agent:46a1795

That said, I fear there's a new problem here, because your logs show "0 flows evicted" from MapTracer, which is something unrelated to the exporting process (it happens before exporting). So I wonder if 1.9.0-community (that we released this week) broke something for you?

Are there any other errors visible in the agent logs?

[jotak]

fwiw, these PRs worked for me, I could test by deploying the agent+direct-flp writing IPFIX, and on the other hand, a standalone flowlogs-pipeline configured as an IPFIX receiver. Restarting the receiver works / connection is restored after a delay.

[WENPIN1]

Hi jotak,

I've tested that quay.io/netobserv/netobserv-ebpf-agent:46a1795 is working as fixed and correctly on IPFIX/UDP.
It also works on gRPC correctly.

Thank you very much!

-WENPIN

[jotak]

The issue comes from the go-ipfix dependency which never re-creates the connection to the collector. A fix is proposed upstream: vmware/go-ipfix#434, and in the meantime, the dependency was forked via this pull-request: netobserv/flowlogs-pipeline#1019 (which also fixes other issues with IPFIX).

It is planned to land in v1.9.1-community in a few days.