A homelab detective story about intermittent services, IP conflicts, and too many smart devices
The Setup
Living in a house with five people means we have a lot of devices. Between phones, tablets, laptops, and gaming systems, our network stays pretty busy. As someone who enjoys tinkering with technology, I’ve also built a small homelab setup that includes a 5-node mini PC Kubernetes cluster and a Proxmox server. It’s my playground for testing new services and learning about infrastructure.
One of the services I deployed months ago was Kavita, a self-hosted ebook server that runs on my Kubernetes cluster. I set it up with MetalLB as a load balancer, which automatically assigns external IP addresses to services in the cluster, making them accessible from anywhere on my home network.
The Mystery
Here’s where things got weird. Kavita had a frustrating habit of being completely unreliable. Some days I’d navigate to its IP address and everything would work perfectly – I could browse my ebook collection, read comics, no problems. Other days, I’d get connection errors like the service didn’t exist at all.
At first, I chalked it up to my own inexperience. Maybe I’d misconfigured something in Kubernetes. Maybe there was an issue with my MetalLB setup. But honestly, I got busy with other things and just lived with the inconsistency for months.
Enter Pangolin
Fast forward to this past weekend. I decided to experiment with Pangolin, a self-hosted solution that lets you securely access your home services from outside your network. It’s like having your own personal VPN tunnel to specific applications.
Naturally, I chose Kavita as my test service to expose through Pangolin. Saturday evening was spent getting everything configured:
- Pangolin running in a cloud VPS
- A Newt client running on my Proxmox box to bridge the connection
- Kavita still running in my Kubernetes cluster
Everything worked beautifully. I could access my ebooks from anywhere with an internet connection.
The Plot Thickens
Sunday morning arrived, and like any proud homelab enthusiast, the first thing I did was test my new setup. I refreshed the Pangolin-exposed Kavita app and… success! Feeling like a conquering hero, I went to the living room to relax and watch the Browns lose yet another game.
During halftime, I decided to check on my setup again. Back to the office, refresh the page, and… bad gateway error
SON OF A…
Now I was frustrated. This intermittent behavior had been bugging me for months, and now it was affecting my shiny new remote access setup. Time to get to the bottom of this once and for all.
The Investigation
To troubleshoot this properly, I needed to check each component in my somewhat convoluted setup:
Step 1: Check Pangolin (VPS)
docker ps
Pangolin container was running fine.
Step 2: Check Newt (Proxmox)
systemctl status newt
Newt showed as running, but I noticed connection refused errors when it tried to reach Kavita’s MetalLB IP address. This meant packets were reaching the destination, but nothing was listening on that port.
Step 3: Check Kavita (Kubernetes)
kubectl describe svc kavita -n default
According to Kubernetes, Kavita was running perfectly. I even logged into the worker node and could curl the service locally without issues.
The Breakthrough
Back on my Proxmox machine, I had a hunch. What if something else on the network was using Kavita’s IP address? On a whim, I ran:
nmap -sn 192.xxx.x.xxx # Kavita's MetalLB IP
This showed a MAC address that belonged to my worker node. But when I checked the ARP table:
ip neigh show 192.xxx.x.xxx
There it was! A completely different MAC address! Something else was claiming Kavita’s IP address.
The Solution
Armed with this knowledge, I knew what to do:
- I looked up my MetalLB address pool configuration
- Updated it to exclude the conflicting IP address
- Restarted the Kavita service
- MetalLB assigned a fresh IP address
- Everything started working again
But I still needed to identify the culprit. I installed NetAlertX on my Proxmox machine and ran a network scan.
The villain revealed: My smart TV.
The “Aha!” Moment
Suddenly, everything made perfect sense. All those months of Kavita working sometimes and failing other times corresponded exactly to whether my smart TV was powered on or off. When the TV was on, it would grab Kavita’s IP address (probably through some quirky DHCP behavior or static IP configuration I’d forgotten about). When the TV was off, Kavita worked fine.
Lessons Learned
This experience taught me several valuable lessons about homelab management:
- Document Your Network
I clearly had forgotten about configuring that IP address on my TV months ago. A simple network documentation spreadsheet would have saved me hours of troubleshooting. - IP Address Management Matters
In a house full of smart devices, proper IP address planning is crucial. I now use separate VLANs and more carefully managed DHCP ranges for different device types. - MetalLB Configuration
When configuring MetalLB address pools, always exclude IP addresses that might be used by other devices. Consider using a dedicated range that’s completely separate from your DHCP pool. - Network Monitoring Tools
Tools like NetAlertX are invaluable for keeping track of what’s actually on your network. I now run regular scans to catch conflicts before they cause problems. - Trust Your Troubleshooting Process
The systematic approach – checking each component in the chain – eventually led to the answer. Even when the root cause seems unlikely (who suspects their TV?), methodical debugging wins.
Prevention for the Future
To prevent this from happening again, I’ve made a few changes:
- Dedicated IP ranges: MetalLB now uses a completely separate subnet from any static IPs I might assign to TVs or other devices
- Network documentation: I maintain a simple spreadsheet tracking static IP assignments
- Regular monitoring: NetAlertX runs weekly scans and alerts me to new devices or conflicts
- Better labeling: All my smart home devices now have descriptive hostnames instead of default names
Conclusion
Sometimes the most frustrating technical problems have the simplest explanations. In my case, months of intermittent service issues were caused by a smart TV that was essentially playing musical chairs with IP addresses.
The next time you’re dealing with mysterious network issues in your homelab, remember: it might not be your carefully crafted Kubernetes configuration or your intricate Docker networking. It could just be too many damn devices fighting over the same IP address.
Have you run into similar network conflicts in your homelab? Share your stories in the comments . I’d love to hear about other unexpected culprits you’ve discovered!
Technical Details:
- Kubernetes: 5-node mini PC cluster
- Load Balancer: MetalLB
- Remote Access: Pangolin + Newt client
- Monitoring: NetAlertX
- Virtualization: Proxmox VE
This post is part of my homelab series where I document the lessons learned from tinkering with self-hosted services and infrastructure.
