Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce Lambda asset size #74

Merged
merged 8 commits into from
Jan 13, 2025
Merged

reduce Lambda asset size #74

merged 8 commits into from
Jan 13, 2025

Conversation

hrodmn
Copy link
Contributor

@hrodmn hrodmn commented Jan 10, 2025

After #72 the Docker build created a Lambda asset that was greater than 262 MB unzipped 🙀

In this PR I am trying a few things to fix it:

  • switch to uv for dependency management
  • move requirements-cdk.txt into the deployment dependency group
  • clean up the dependencies in pyproject.toml, making sure ones we only need for tests are not listed in the default dependencies (I think this actually fixed it)
  • run cdk synth in CI and check the size of the assets in cdk.out

With these changes we squeak under the package size limit by 2 MB 😬

I compared the asset package size to titiler-cmr (which has a 230 MB asset size) and the main difference in titiler-multidim is the addition of zarr as a dependency which adds numcodecs which demands 33MB in the asset directory. This package is full of hefty dependencies!

Resolves #56 (for now)

Here is a summary of the largest dependencies and the size they occupy in the cdk Lambda asset:

henry@quercus:~/workspace/devseed/titiler-xarray-application/infrastructure/aws/cdk.out/asset.bcc78660745bbaa8bbfcec1da064c2564455348ac653ec35f5bf2eb9ebdbb979$ du -h --max-depth=1 | sort -hr
261M    .
47M     ./rasterio.libs
33M     ./numcodecs
30M     ./pandas
25M     ./numpy.libs
24M     ./numpy
18M     ./pyproj
16M     ./rasterio
13M     ./pyproj.libs
9.3M    ./h5py
5.6M    ./aiohttp
5.1M    ./h5py.libs
4.6M    ./pydantic_core
4.5M    ./cftime
3.1M    ./xarray
2.7M    ./tzdata
2.7M    ./pytz
1.5M    ./pydantic
1.3M    ./numexpr
1.1M    ./rio_tiler
1.1M    ./redis
...

@hrodmn hrodmn requested a review from vincentsarago January 13, 2025 18:04
@hrodmn
Copy link
Contributor Author

hrodmn commented Jan 13, 2025

With the C/C++ strip operation in the Lambda Dockerfile I have the /asset size down to ~215 MB which gives us plenty of headroom. If anyone would prefer I add that change in a separate PR I think we could get away with the pyproject.toml cleanup alone, but we would need to add more specific version constraints to avoid package updates causing the asset size to increase!

@hrodmn hrodmn merged commit 0f31041 into dev Jan 13, 2025
5 checks passed
@hrodmn hrodmn deleted the fix/lambda-docker-size branch January 13, 2025 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Package size exceeding lambda limits
2 participants