Investigate globus file size storage

[peetucket]

Three questions to look at:

• Do all items deposited through Globus have no file sizes in the byte_size column of the attached files table? Or is it just more recent Globus deposits?
• What happens with "mixed deposits" (i.e. start with Globus, then add files via upload in V2)? Do some files have file sizes but not others (assume yes)?
• Verify the reason we do not store file sizes for Globus files (assumption is performance was slow, but do some archaeology to verify this).

The answers to this question will determine if are able to actually add file sizes to Globus objects for not. They will also help us with #3230

[peetucket]

From the first commit, app/jobs/fetch_globus_job.rb has created attached files with byte_size of 0:

4aff429#diff-c757f9bcc510ecd347cc838b2a41e331e84e68b98642b04d20350f9769bd4ffeR29-R39

Not clear from the code why though.

[peetucket]

Check Globus API to see if we can get file sizes along with the file names?

[lwrubel]

We are using the API to get the filesizes along with the filenames when using the list_files public method.

https://github.com/sul-dlss/globus_client/blob/5de26db0e52cb7b089a3aec48b2403409fdf82af/lib/globus_client/endpoint.rb#L125

[lwrubel]

1. Do all items deposited through Globus have no file sizes in the byte_size column of the attached files table? Or is it just more recent Globus deposits?

@edsu looked at this and noted that the globus_client get_filenames method is used for creating AttachedFiles. It only returns filenames: https://stanfordlib.slack.com/archives/C69UZCJ8M/p1689082395205739?thread_ts=1689028505.499099&cid=C69UZCJ8M It has an underlying method that does get the filesizes from the API. So, there is another method in the client we could be using (see previous comment).

2. What happens with "mixed deposits" (i.e. start with Globus, then add files via upload in V2)? Do some files have file sizes but not others (assume yes)?

@edsu and I looked at a "mixed deposit" and saw that only files uploaded via the browser have filesizes. https://stanfordlib.slack.com/archives/C69UZCJ8M/p1689100415282479?thread_ts=1689098275.906289&cid=C69UZCJ8M

[edsu]

Thanks for leaving those notes about what we found @lwrubel. I think we can close this and move #3230 into ready?

[peetucket]

Answered in other tickets. We have access to file sizes for all gobus files and will make use of them now.