When we are setting up the Billing Account Export, we create a new Dataset and specify the project name. So each big query export for billing is for a single project and do we need to do this for every project ?
No, you do not need to create a billing export for each project.
On the last slide at the end of the "Set Up Billing Export (Lab)" video, Mattias says "Export must be set up per billing account". The billing account covers all of projects that you created up to this point in the course. Do a query on your BigQuery table (or just click the "Preview" button) and you’ll notice "project.id" and "project.name" columns.
Log in with your Admin account and take a look at billing to see how your projects are associated with a billing account. https://console.cloud.google.com/billing