Write File in Azure Blob

[akash-goel]

Hi Team ,

We are using Azure blob Storage and want to write a file in Blob using Vaex . we have tried export_hdf5 and export Functions it did not worked for us , Can you please suggest how we can write a hdf5 file in Azure Blob usign Vaex.

Thanks

[akash-goel]

Do Vaex supports Azure Blob Storage or any way to write file in blob storage using Vaex ? Please guide

[JovanVeljanoski]

Take a look at:

• [BUG-REPORT] Azure Data Lake Storage Gen2 using fsspec #1272
• [FEATURE-REQUEST] Read files from ADLS Gen 2 (Cloud support for Azure Data Lake) #1248

Should probably help you out

[JovanVeljanoski]

Try export_arrow or export_parquet where other file systems are supported.

[akash-goel]

Thanks Jovan

we have tried with export_arrow ,export_parquet and export_many option as well , for 20 million records Vaex is taking 80 seconds to read the data but during write operation into the Azure blob for both (Parquet and Arrow ) it is generating the files but taking more than 30 min. If we are using Dask Library then it is taking 5 min only . Is this expected time from Vaex or anything we are missing , please find the below code your reference.
Request you to guide us in case we are missing anything .

Parquet Generation Code

%%time

import vaex
import adlfs
from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions, __version__
engagement_code = 'E-T123456789'
product_type = 'Type'
account_name = 'accountName'        
account_key = 'accountKey'                
container_name = 'container'
fs = adlfs.AzureBlobFileSystem(account_name='accountName', account_key='accountKey')
mergedDf.export_many('containerName/output_changed_chunk_arrow/output_chunk-{i:03}.parquet', chunk_size=1000000,fs=fs)

Export Arrow

%%time
import vaex
import adlfs
from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions, __version__
engagement_code = 'E-T123456789'
product_type = 'Type'
account_name = 'accountName'        
account_key = 'accountKey'                
container_name = 'container'
fs = adlfs.AzureBlobFileSystem(account_name='accountName', account_key='accountKey')
mergedDf.export_many('containerName/output_changed_chunk_arrow/output_chunk-{i:03}.parquet', chunk_size=1000000,fs=fs)

[JovanVeljanoski]

Can you format your code nice please.. hard to read like this

[akash-goel]

Done

[maartenbreddels]

When we are using export_hdf5 function and passing filesystem then getting this error "TypeError: export_hdf5(url,fs) got an unexpected keyword argument 'fs'"

hdf5 cannot be written to the cloud, only read (see https://vaex.io/docs/example_io.html#Cloud-Support )

How are you using dask here, is this multi node or multi processing?