Keepalived DNS failure

[ehrie]

Hoping someone here has heard of this issue and can point me in the right direction on how to solve it. I have two MiniPCs each loaded with proxmox and in LXC containers each contain PiHole, Unbound and Keepalived. I set up the keepalived install using this guide: https://www.wundertech.net/ultimate-pi-hole-setup/
A couple of notes:

Both of my Piholes work perfectly when their native IP is set as the DNS server for the given network. When keepalived is started, both lxc containers show the outputs one would expect when systemctl status keepalived.service is called.

Now here's my problem. The first lxc that is booted up is the only one able to access DNS on the virtual IP. If I shutdown keepalived on that lxc, the other container will take over. I can see the state change to MASTER. If I go to the vip/admin I am now routed to the web admin panel of that pihole, but any DNS requests to the VIP fail with a Connection Refused on port 53 of the VIP. If I reenable the downed keepalived it will switch back and the vip will work again. I want to emphasize it's not the master that always has control. It's whichever lxc I start first. I am at a total loss on how to troubleshoot this as google has yeilded nobody else I can find having this kind of problem. I would welcome any ideas. I can provide any logs, settings, whatever is needed. I am not sure what would help so I will not spam this initial post.

[pqarmitage]

I think you will need to trace the DNS packets to see which miniPC the packets are being sent to; e.g. run tcpdump on both of the devices and see which one is receiving the DNS requests.

I note that keepalived configuration is very outdated. I would delete the authentication, stop using unicast, and use VMACs. You might find that using VMACs resolves your problem.

I'm not sure I can see the purpose of this setup. Why not just run the DNS servers (and Pi-hole) on both miniPCs simultaneously, and configure clients with the IP addresses of both DNS servers. That way you remove a lot of the complexity and the DNS resolver in the client will try one DNS server, and if that doesn't respond will try the other.

[ehrie]

Thanks, I tried the VMAC suggestion and trimming the fat in the conf files to no avail. The VMAC addition caused this error in my Proxmox console upon starting the LXC container:

TASK ERROR: command '/usr/bin/termproxy 5901 --path /vms/101 --perm VM.Console -- /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=proxmox2' -o 'UserKnownHostsFile=/etc/pve/nodes/proxmox2/ssh_known_hosts' -o 'GlobalKnownHostsFile=none' -t [email protected] /usr/bin/dtach -A /var/run/dtach/vzctlconsole101 -r winch -z lxc-console -n 101 -e -1' failed: exit code 1

It did look like the vmac was functioning though. I ended up binning this as I just can't get it to work. No idea where the source of this Connection Refused on just port 53 is coming from. DNS packets are getting to the 2nd pihole when I shut the Master down, but the only thing tcpdump was telling me was it was unreachable.

00:01:49.005105 ARP, Request who-has 10.15.1.8 tell 10.15.1.110, length 46
00:01:49.005109 ARP, Reply 10.15.1.8 is-at bc:24:11:6c:da:61 (oui Unknown), length 28
00:01:49.005232 IP 10.15.1.110.41082 > 10.15.1.8.domain: 53430+ [1au] A? espn.com. (49)
00:01:49.005248 IP 10.15.1.8 > 10.15.1.110: ICMP 10.15.1.8 udp port domain unreachable, length 85
00:01:49.005485 IP 10.15.1.110.55970 > 10.15.1.8.domain: 53430+ [1au] A? espn.com. (49)
00:01:49.005491 IP 10.15.1.8 > 10.15.1.110: ICMP 10.15.1.8 udp port domain unreachable, length 85
00:01:49.005634 IP 10.15.1.110.47855 > 10.15.1.8.domain: 53430+ [1au] A? espn.com. (49)
00:01:49.005640 IP 10.15.1.8 > 10.15.1.110: ICMP 10.15.1.8 udp port domain unreachable, length 85

To answer your last question there's two reasons I wanted to do this with keepalived rather than your suggestion which is what I have now. Firstly, I wanted all logs and things in a single place and the second and more important one is I have some older smart home equipment that seems to only store one DNS server and won't utilize any others. Whenever my primary would be taken down for whatever reason I need to manually restart them to get things working again. it's not often, but it's a minor irritation I was hoping to solve.

[pqarmitage]

I think the cause of this is that the DNS server is listening on specific IP addresses, i.e. the ones that are configured on the system when the DNS server starts. When the VIP is added, the DNS server does not add the new IP address to the addresses it is listening on. netstat -anp | grep ":53 " should show you what addresses your DNS server is listening on.

It might be possible to reconfigure the DNS server so that it listens on 0.0.0.0:53 and :::53. Alternatively you might need to run a notify_master script to restart the DNS server when the backup becomes master.

[ehrie]

Thanks! I took a fresh look at this and your first reply, plus noticing the apt install keepalived was installing a really old version of keepalived combined to finally solve this for me. When I did a manually installation of 2.3.3, plus use vmac and vmac xmit base to the config and now it all works. Trying this on 2.2.7, which is what the above command was installing I think illuminated what the problem was. When failover happened for some reason the lxc container was also claiming the ip address of the vip on its vlan and not with the same MAC address. I don't know enough about the weeds of networking to explain that, but the same settings on version 2.3.3 fixed it and now everything works. Thanks again for taking the time to look at this for me.