VPN Problems and Ubuntu: killing off the dnsmasq zombie

I’ve been having problems with VPN, DNS and Ubuntu for a year. But, I’m also pretty lazy when it comes to spending time on configuration. And configuring VPNs is like last on my list of ways I’d like to spend my time.

In short, I’d rather reboot than figure out exactly why my networking just stopped working.

REBOOT.

Fortunately, I had an easy (for me) work-around for most of my VPN needs: use SSH and a jump-host for getting to servers. I found it annoying when I wanted to look at a website on protected network space, or had a service on an unusual port that I wanted to test things against. I would work around with SSH tunnels, or I would fire up my Mac, whose VPN settings worked flawlessly.

That all said, I thought today, a sunny, lovely fall day in Portland, I would fix my VPN.

And so, my buddy @uberj_ helped me get things sorted.

The root cause of all my VPN heartache was the dnsmasq daemon controlling my DNS. And, related, network-manager. There are a few places that document exactly how to disable dnsmasq

  • DNS in Ubuntu 12.04 http://www.stgraber.org/2012/02/24/dns-in-ubuntu-12-04/
  • Disabling dnsmasq as your local DNS server in Ubuntu http://mark.orbum.net/2012/05/14/disabling-dnsmasq-as-your-local-dns-server-in-ubuntu/

However, they leave out one important step: killing off the existing dnsmasq process. For the unlucky, restarting network-manager does not kill off dnsmasq.

So, to find and kill dnsmasq, do the following:

 sudo service network-manager stop
 kill `ps -C dnsmasq -o pid=`
 sudo service network-manager start

Then, start your VPN and check out the contents of the /etc/resolv.conf. If all went well, you’ve got nameserver addresses other than 127.0.0.1 in the file.

Yay!

Sadly, this was not the end of my story.

After a few minutes, NetworkManager started dnsmasq up again!

Zombie dnsmasq

So, like any reasonable sysadmin, I opened up the /etc/NetworkManager/NetworkManager.conf file, uncommented the dns=dnsmasq line, and replaced it with dns=/dev/null. My guess was that you can probably put just about anything other than dnsmasq into that line to permanently disable the plugin.

I ran sudo service network-manager restart, checked /etc/resolv.conf and felt pretty smug.

I tried also uninstalling dnsmasq-base package, but unfortunately that takes out a number of other packages I appear to need. So, I left /dev/null in my NetworkManager.conf, and updated this blog post.

But wait...

While editing this blog post, dnsmasq took over my DNS settings again.

A clue as to what was happening was in /var/log/syslog:

Oct 18 10:20:10 localhost dnsmasq[30535]: started, version 2.59 cache disabled
Oct 18 10:20:10 localhost dnsmasq[30535]: compile time options: IPv6 GNU-getopt DBus i18n DHCP TFTP conntrack IDN
Oct 18 10:20:10 localhost dnsmasq[30535]: DBus support enabled: connected to system bus
Oct 18 10:20:10 localhost dnsmasq[30535]: warning: no upstream servers configured

It turns out that dnsmasq was still getting revived by NetworkManager. Why NetworkManager doesn’t seem to care about configuration settings was beyond my willingness to investigate today. So, I did some more searching about truly killing of dnsmasq for good.

And I found this thread, and this sample configuration file. In the output for the dnsmasq process from ps:

nobody   30777 30759  0 10:21 ?        00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/var/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.0.1 --conf-file=/var/run/nm-dns-dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus --conf-dir=/etc/NetworkManager/dnsmasq.d

I dug into the thread, and the suggestion was to set port=0 in the config. I created a file called custom in /etc/NetworkManager/dnsmasq.d. And ran sudo service network-manager restart.

And then I got this in my syslog:

Oct 18 10:21:10 localhost dnsmasq[30777]: started, version 2.59 DNS disabled

FINALLY.

FINALLY!

An experiment in attention

I’ve had reoccurring thoughts about attention and who I give mine to. In the last week, I’ve been mentioned in a couple “women in tech” twitter lists. This seems to happen about quarterly and someone will create a list of 50 or so, or maybe 100+ women on a list.

I spent a couple days looking through my 5k followers and assembled a list of the women and women’s groups who are following me. I probably missed a few, and I know I followed a few people who don’t consider themselves women. Sorry! Just let me know and I’ll add/drop as needed.

So, why bother with a list like this?

Before I created this list, I was following about 920 people. Which, in itself is a sort of ridiculous number. How could I possibly pay attention to that many people?

I really don’t, right? I just check into twitter, sample the firehose, and then step away for minutes, hours or days.

When I do sample the tweet stream, whose voices do I listen to? There are certainly a few close friends whose feeds I look at directly, and a few other people I’m interested in who I will catch up on a backlog. Otherwise, it’s whoever is the most vocal.

What I noticed is: most of the voices I hear from when I sample the feed are men. It wasn’t anywhere near balanced. That’s on social activity, technical rants, technical praise and blogging.

I’d like to be skewed toward women’s voices for a while. Particularly on tech issues. So, I just added about 450 women to my feed who were already following me.

My next step may be replacing my primary feed with this list I’ve made. I wrote a tool a while back to extract URLs and RSS feeds from my friend’s twitter profiles and feeds. The code isn’t awesome, just rough and practical. But you could do something similar using it. You need a set of read/write API keys (create an app, then make it read/write), but I did the “hard” work for you:

./opml.py --consumer_key [key here] --consumer_secret [secret here] --access_token [token here] --access_token_secret [token secret here]  --to_follow [file of twitter handles]

I left some crud in there that links the script directly to my account for the list. Sorry! If anyone actually wants to use this, I’ll clean it up. (just ping me on github)

Anyway, doing this kind of attention hacking for yourself isn’t hard. It is drudgery to go through all your followers and guess who is what gender. But it is interesting to spend a few hours contemplating what the people you’re giving your attention to have in common, and how you might hack it a bit to hear from different perspectives from time to time.

I’m turning comments off because I don’t care to hear from anyone who thinks I’m somehow being sexist by changing who I pay attention to. Cheer up, haters! I’m sure plenty of people are already paying attention to you.