what happens when an instance has 2 security groups with conflicting rule, like one allow and another denies HTTP?
The answer to your question is the most permissive rule will be applied. You can check the link below for more details
When Amazon EC2 decides whether to allow traffic to reach an instance, it evaluates all of the rules from all of the security groups that are associated with the instance.
From AWS "If there is more than one rule for a specific port, Amazon EC2 applies the most permissive rule. For example, if you have a rule that allows access to TCP port 22 (SSH) from IP address 203.0.113.1, and another rule that allows access to TCP port 22 from everyone, everyone has access to TCP port 22."
More specific to your question however, Security Groups are always permissive. It’s not possible to create a deny rule as you asked, so your specific conflicting rules question is actually not possible.
To expand on Ankit’s answer then, there actually is no such thing as an Explicit Deny (making this incorrect). As Mourad mentioned however, there is an "implicit" deny. Any possible security group rule is an explicit allow only, and therefore overrules the implicit deny.
You can read more specifically about it here
Hope this helps!