The postman always rings twice

Wednesday, April 8, 2020

Intro

In the microservice world, very often, a service depends on another service. Which in turn might depend on others and so on. A common practice is to let a service die if its dependencies are not ready when it starts then leave the responsibility to restart it to the orchestration platform. There are cases where a restart is not possible or wanted.

File:The Postman came twice.jpg

What if instead of dying, we make the service make sure the dependency is ready before actually starting its feature presentation?

So let’s

I created a bash script that waits until the service becomes available.

until ping -c1 $HOST
do
	sleep 10s
done

This implementation exits the loop as soon as the host becomes reachable by ping. The ping can be replaced by any command that would exit with code 0 when the dependency is avaliable and not 0 otherwise.

But just twice

Off course, you might not wait forever. Let’s say after a minute or so, exit with an error. Another thing is perhaps not waiting a constant time but increase the time between waits.

WAIT_TIME_SECONDS=10
until ping -c1 $HOST
do
	sleep ${WAIT_TIME_SECONDS}s
	
	WAIT_TIME_SECONDS=$(( 2  *  $WAIT_TIME_SECONDS ))

	if  (( $WAIT_TIME_SECONDS  >  $MAX_WAIT ))
	then
		exit 255
	fi
done

Feed back

You might want to discard output from each of the failing command execution.

until ping -c1 $HOST &>/dev/null
do
	# ...
done

You might also want to know each time an iteration started.

	echo Waiting ${WAIT_TIME_SECONDS}s ...

Mongo Bonus

I needed this for a mongo server, the thing is ping responds way before mongo is available and therefore not good. Finding the right command per each case is important.

until mongo --host $HOST --eval db
do
	# ... 
done

Outro

Dealing with dependencies is becoming more and more complicated as we break problems into smaller ones. Finding ways to make your life easier will make the difference.