I need to run a bigquery script in python, which needs to output as a CSV in google cloud storage. Currently, my script triggers the big query code and saves to my PC directly.
However, I need to get this running in Airflow so I can't have any local dependencies.
My current script saves the output to my local machine and then I have to move it into GCS. Looked online and I can't figure it out. (ps im very new to python so im sorry in advance if this has been asked before!)
import pandas as pd
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
def run_script():
df = pd.read_gbq('SELECT * FROM `table/veiw` LIMIT 15000',
project_id='PROJECT',
dialect='standard'
)
df.to_csv('XXX.csv', index=False)
def copy_to_gcs(filename, bucket, destination_filename):
credentials = GoogleCredentials.get_application_default()
service = discovery.build('storage', 'v1', credentials=credentials)
body = {'name': destination_filename}
req = service.objects().insert(bucket=bucket,body=body, media_body=filename)
resp = req.execute()
current_date = datetime.date.today()
filename = (r"C:\Users\LOCALDRIVE\ETC\ETC\ETC.csv")
bucket = 'My GCS BUCKET'
str_prefix_datetime = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
destfile = 'XXX' + str_prefix_datetime + '.csv'
print('')
```