Ingress Firewall Rule

Seems like firewall is not configured properly.

I tried to mimic your use-case and with correct firewall rules and it worked. Here are the steps that I followed:

1. Created a VPC with 2 subnets – each for frontend and backend

2. Created firewall rules to allow SSH

3. Create firewall rule to allow IMCP from frontend to backend

4. Created frontend vm (using frontend subnet) and backend vm (using backend subnet)

5. Connect to frontend VM via SSH and then tried to ping backend VM — it worked

Here are the commands that I used to create the test environment:

#setting project for this test  
gcloud config set project playground-s-11-682e485f  
#creating a vpc  
gcloud compute networks create vpc-ingress-test --subnet-mode=custom --mtu=1460 --bgp-routing-mode=regional  
#adding two subnets  
# frontent-subnet | 10.1.0.0/20 | us-east1  
# backend-subnet | 10.2.0.0/20 | us-east4  
gcloud compute networks subnets create frontend-subnet --range=10.1.0.0/20 --network=vpc-ingress-test --region=us-east1  
gcloud compute networks subnets create backend-subnet --range=10.2.0.0/20 --network=vpc-ingress-test --region=us-east4  
#create firewall rule to allow ingress from frontend to backend  
gcloud compute firewall-rules create allow-icmp-from-frontend --network vpc-ingress-test --allow icmp --source-ranges 10.1.0.0/20  
# allow SSH on both backend and frontend VM  
gcloud compute firewall-rules create allow-ssh-fronent-backend --direction=INGRESS --priority=900 --network=vpc-ingress-test --action=ALLOW --rules=tcp:22 --source-ranges=0.0.0.0/0  
#creating frontend vm  
gcloud beta compute instances create frontend-vm   
--zone=us-east1-b   
--machine-type=n1-standard-1   
--subnet=frontend-subnet  
#creating backend vm  
gcloud beta compute instances create backend-vm   
--zone=us-east4-c   
--machine-type=n1-standard-1   
--subnet=backend-subnet

And here a quick test

#-------------------------------------------  
# Testing  
#-------------------------------------------  
#setting project for this test  
gcloud config set project playground-s-11-682e485f  
#getting all the VMs created  
gcloud compute instances list  
NAME         ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP   STATUS  
frontend-vm  us-east1-b  n1-standard-1               10.1.0.3     35.227.47.15  RUNNING  
backend-vm   us-east4-c  n1-standard-1               10.2.0.4     34.86.115.14  RUNNING  
#ssh to frontend-vm and ping IP of backend-vm  
gcloud compute ssh --zone us-east1-b frontend-vm  
ping -c 3 10.2.0.4  
Linux frontend-vm 4.19.0-13-cloud-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64  
The programs included with the Debian GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Mon Dec 28 04:09:36 2020 from 35.229.119.44  
ping -c 3 10.2.0.4cloud_user_p_2f3f022d@frontend-vm:~$ ping -c 3 10.2.0.4  
PING 10.2.0.4 (10.2.0.4) 56(84) bytes of data.  
64 bytes from 10.2.0.4: icmp_seq=1 ttl=64 time=12.4 ms  
64 bytes from 10.2.0.4: icmp_seq=2 ttl=64 time=11.5 ms  
64 bytes from 10.2.0.4: icmp_seq=3 ttl=64 time=11.4 ms  
--- 10.2.0.4 ping statistics ---  
3 packets transmitted, 3 received, 0% packet loss, time 5ms  
rtt min/avg/max/mdev = 11.363/11.739/12.390/0.470 ms

Hope this helps!