Thursday, 15 January 2015

Basic readiness check before developing with micro services - DNS

So, you have your application nicely sliced up into purposeful, self-contained units which can call upon each-other as required - great.

Presuming each of these components is communicating over http, how are you going to deploy them and make them accessible to each other?  Why not use a beautiful Platform As A Service environment - developers can deploy new apps, and manage existing apps to their hearts' content without any assistance from other teams.  Lovely.

Ok, let's presume that your setup has big ambitions and limited IP addresses so you're going to need these applications to have hostnames.  No big deal, and easier to quickly verify what is what.  Who would use IP addresses directly - this is the 21st century!

What if DNS hasn't been set up to cope with some additional load.  It only craps out every other week, but when it does your applications can't reach each other.  Deployments fail, developers get stalled tracking down problems that they haven't caused.  Live demonstrations are now considered risky.

Welcome to my world.  It's the third or fourth time that I've bugged a team that should be able to trace this problem and arrange for it to be fixed (even temporarily).  Their response is that the impact seems to be broader than the single host that the app is currently complained about.  Their recommendation is to update the app's config with IP addresses for a while.

We have a central configuration system, but I don't think it is intended to act as a replacement for an /etc/hosts file.

After publicly lecturing me about how the configuration management system should make dirty hacks like this (I'm paraphrasing) possible, they contacted someone who could flush a cache or restart the misbehaving service to solve the problem temporarily.

No comments:

Post a comment