Early the other morning I was upgrading a good number of instances in AWS.
Merrily going along with the updates from older to newer tech, (m1 to m3 etc).
Simple enough I thought.
We have multiple machines behind elastic load balancers and the status checks are reasonably frequent.
I began to update machines … keeping an eye on the load balancers and the number of out of service machines, I began to notice something a little odd.
A quick check to the monitoring URL to make sure everything was as expected
Machines that had been upgraded were not coming back into service.
10 minutes pass, then 20 then 30, still the machines were marked Out of service. Hmmmm something seemed amiss.
Time for a quick test. What still not added back in, lets remove one and add it back in and see what happens.
What do you know, bam, 5 minutes after a re-add and there it was back in service.
I’m am not saying this is always the case as on another account it took up to 15 mins to add the machines back in service. Keep it in mind if casually updating AWS EC2 instances that are load balanced, take a look and make sure they are back in service and if not remove the instance and re-add it.
That should do the trick.
Have fun, AWS can be loads of fun.