How to create a connection between cloudsql and bigquery in the data project?

[mohangmk]

@moschetti
We would like to know how to connect CloudSQL from BigQuery.

We have tried below methods but none of them are helping completely

1. External connection - This is not possible because external IP is not enabled for CloudSQL in the current environment
2. Export the data to a storage bucket in CSV file and create a dataset in BigQuery - This is creating multiple conflicts.

Please let us know if we have any other method to connect CloudSQL without any issues from BigQuery.

[moschetti]

To query CloudSQL from BigQuery, you will need a public IP as federated queries do not support querying CloudSQL with private IP, there are notes here on how to do this without providing access, but that would still require a public IP, so we'd need to weigh the risk/benefit.

What are the issues you're facing with exporting to CSV?

[mohangmk]

There is no issue with CSV export but we have issue with creating a dataset in big query by using the same CSV file due to null conflicts and it is also not supporting nested or repeated data

[scotthardin]

Cloud SQL solution coming: for https://issuetracker.google.com/issues/64579566

aj...@google.com added comment #65:
Cloud SQL users: we are soon going to be rolling out new custom formatting controls that will allow users to choose custom quote, field delimiter, and escape characters when exporting and importing CSVs from and to Cloud SQL. We understand that this issue has been open for a very, very long time. We appreciate your patience for a little while longer as we make this available. Until then, please continue to use the workaround recommended earlier in this bug. https://b.corp.google.com/issues/64579566#comment39

Best, Akhil Cloud SQL PM