Repo-level and pattern-level configuration in .thetaconfig

[nkandpa2]

This PR factors a bunch of the environment variable-based configuration out into a .thetaconfig file at the root of the repository. This file gets committed and distributed as part of the repo.

.thetaconfig is a json file with two top-level keys: repo and patterns.

The repo key maps to a dictionary of repository-level config fields like the atol and rtol parameters for equality checks, the signature size, pool size and threshold for LSH, and the maximum number of concurrent async jobs.

The patterns key maps to a list of dictionaries. Each dictionary contains the configuration for a particular glob pattern in the same way that gitattributes are specified for glob patterns. Right now, the only pattern-level configuration is the checkpoint format (PyTorch, TF, etc.).

The .thetaconfig file is created whenever a user runs git theta track for the first time. Each subsequent git theta track command adds another pattern-level dictionary to the file.

Here is an example of what .thetaconfig looks like:

{
    "repo": {
        "parameter_atol": 1e-08,
        "parameter_rtol": 1e-05,
        "lsh_signature_size": 16,
        "lsh_threshold": 1e-06,
        "lsh_pool_size": 10000,
        "max_concurrency": -1
    },
    "patterns": [
        {
            "pattern": "model.pt",
            "checkpoint_format": "pytorch"
        }
    ]
}

In addition to the code change for creating the .thetaconfig file, I also factored the .gitattributes reading/writing/parsing out into analogous classes.

Note 1:
In cases where we want to run a git command on a file that matches multiple patterns in .thetaconfig, the behavior is that patterns that appear later in .thetaconfig shadow previous patterns. This matches the behavior of .gitattributes.

For instance say we have the following .thetaconfig:

{
    "repo": {
        "parameter_atol": 1e-08,
        "parameter_rtol": 1e-05,
        "lsh_signature_size": 16,
        "lsh_threshold": 1e-06,
        "lsh_pool_size": 10000,
        "max_concurrency": -1
    },
    "patterns": [
        {
            "pattern": "dir1/dir2/model.bin",
            "checkpoint_format": "tf"
        },
        {
            "pattern": "dir1/dir2/**",
            "checkpoint_format": "pytorch"
        }
    ]
}

If we run a git command on dir1/dir2/model.bin, this would match both patterns, but the second pattern would shadow the first pattern. Thus, the file would be treated as a PyTorch checkpoint. It seems a bit weird since the first pattern exactly matches the file, but this behavior matches Git's parsing behavior for the .gitattributes file. Since both .gitattributes and .thetaconfig are written using the git theta track command, we should ensure that the parsing logic for these files is the same.

Note 2:
The way we were handling adding gitattributes in the past was slightly wrong. For instance, if we had attributes for the pattern dir/*.pt and we run git theta track dir/model.pt, then the code would add the theta filter to the dir/*.pt entry. This is incorrect behavior since we end up tracking more files than what we intend to. This is fixed in the PR.

[blester125]

I think this function is doing too much now, based on SoC, this function should really be about translating a checkpoint name into a class. We should probably have the path -> checkpoint type be done externally and then the result is passed in.

[blester125]

This is going to result in reading the config file a bunch of time yeah? Should we refactor to something like the config file is read once and then the config object is used?

[blester125]

I think we want to make sure that we decouple a lot of these functions from needing there to be an actual config, otherwise they get really hard to reuse or use in tests.

We probably want things like the signature size, atol, etc to be parameters where None values result in looking it up in the config?

[blester125]

pool size should probably be a parameter where a None results in a config look up, the same way signature size is

[blester125]

The base class never needs the path right? Can we move this to IncrementalUpdate init like the update_data parameter?

[blester125]

Same question as below wrt to holding onto the file name.

[blester125]

Can we add a comment about how we don't have a break here because gitattribute shadow behavior means we want to update every line where the pattern appears?

[blester125]

Is it possible to move the reading of sideloaded information out of the IncrementalUpdate class and have it expect an Optional[Dict[str, np.ndarray]] instead? That would remove the need to pass path into the updates everywhere right? Updaters created during the recursive part get their parameter data from git so they don't need to get passed update_data

[blester125]

I think we should go full one way or the other for configuration? Right now somethings are in the file and some things are env variables. My original though was that the config file would just populate env-vars with default values so all lookups are via envvars.

Do we want some new collection of values on the config object that are like runtime config values? That is where things like update type or update data path would live? They could have defaults in the config but get overwritten by EnvVars, kinda like the jax config object?

[blester125]

It looks like with this approach we are loosing the ability to tweak values from and EnvVar at runtime? We definitely want to be able to do that for things like max concurrency and I could see wanting to do that with other things too,