Specially when you have no way to control TTL on intermediate devices and clients itself specifically. Everything which comes in between client ( let’s say a laptop ) and route53 may have their own caching policies, does that make DNS based routing a weak link ?
All DNS’s around the internet and browsers are supposed to respect the TTL for the origin DNS entry, but as you said, sometimes they don’t. This is at the detriment of the service or application that doesn’t respect the TTL because it really breaks the layer of abstraction that DNS is supposed to provide between names and IP.
For the vast majority of cases, TTL is respected and works for routing. If you’re concerned about this, you can deploy using an EIP assuming you have a service that you can assign an EIP. Even services that do override the origin TTL won’t cache forever, or else they would slowly become nonfunctional.