Doesn’t using the HPA make the Deployment YML out of date?

In one of the previous videos, Nigel said that you should NEVER do a rollback using kubectl as it invalidates the declarative nature of the YML. However, the instance that some of your pods are scaled due to high demand (e.g. have 5 replicas) and your YML stored in Git has 3, and you do a kubectl apply to update the version, will it not bring it down to 3 replicas despite the fact the deployment is scaled out? This is because your YML file replicas line no longer matches the observed state?