v4.4.1

What's Changed

• Update file tutorial with missing line by @rxu17 in #1123
• [SYNPY-1436] avoid duplicate user profile call during login by @danlu1 in #1124
• [SYNPY-1511] Correct missing synapse client propogation and caching by @BryanFauble in #1125

New Contributors

• @rxu17 made their first contribution in #1123

Full Changelog: v4.4.0...v4.4.1-rc

v4.4.1-rc

What's Changed

• Update file tutorial with missing line by @rxu17 in #1123
• [SYNPY-1436] avoid duplicate user profile call during login by @danlu1 in #1124
• [SYNPY-1511] Correct missing synapse client propogation and caching by @BryanFauble in #1125

New Contributors

• @rxu17 made their first contribution in #1123

Full Changelog: v4.4.0...v4.4.1-rc

v4.4.0

Highlights

• Downloading files stability improvements:
  • The download algorithm has been re-written to focus on stability, reliability,
    and performance.
  • Checkout the results of our benchmarking.
    Compared to v4.3.0 we are seeing a 10-50% decrease in overall transfer time for
    most test cases.
• New Docker containers every release:
  • Every production, release candidate, and develop release will create a new docker
    image available here
• Credentials passed by command line argument will now be evaluated before credentials
  stored in the ~/.synapseConfig file.

What's Changed

• [SYNPY-1341 & SYNPY-1479] Change syntax for conditional of ghcr release job by @jaymedina in #1102
• [SYNPY-1483] Update default order for credential provider by @BryanFauble in #1105
• [SYNPY-1482] Updating to remove loop executor during annotation store by @BryanFauble in #1104
• [SYNPY-1476] Splitting logic for downloading content from client.py by @BryanFauble in #1103
• [SYNPY-1485] feat: added isort to pre-commit hook by @linglp in #1108
• [SYNPY-1485] Running pre-commit on all files to run isort by @BryanFauble in #1109
• [SYNPY-1447] Update fillna method to work directly off original df by @jaymedina in #1113
• [SYNPY-1457] Fix deprecated Node.js build warnings by @jaymedina in #1112
• [SYNPY-1487] Fix missing coverage report by @jaymedina in #1114
• [SYNPY-1476] Refactoring multithreaded downloads to be more fault tolerant by @BryanFauble in #1107
• [SYNPY-1490] Merge develop into master by @BryanFauble in #1116
• [SYNPY-1494] Remove process pool executor by @BryanFauble in #1117
• Update os matrix for check-deploy by @BryanFauble in #1118
• [SYNPY-1476] Skip reading response content on error by @BryanFauble in #1119
• [SYNPY-1504] Remove download_location from File and rely only on Path by @BryanFauble in #1120

Full Changelog: v4.3.1...v4.4.0

v4.3.1

Bug Fixes

• [SYNPY-1494] - loky crashed uwsgi process in schematic manifest submission

Full Changelog: v4.3.0...v4.3.1-rc

v4.3.0

Highlights

• New tutorial:
  • Uploading data in bulk
    is our newest tutorial. It covers the basics of working with manifest files to manage synapse projects.
• Updates to syncToSynapse:
  • The syncToSynapse function has been refactored to improve performance and
    reliability.
  • Minor behavior change: File entities will no longer have it's version
    incremented during no-op changes. Only when file content, or fields on the file
    has been updated will a version number be incremented.
  • Optional booleans merge_existing_annotations and
    associate_activity_to_new_version have been added. Both are used to give more
    fine tuned control when working with this interface.
  • Check out the changes in the reference docs.

What's Changed

• [SYNPY-1474] Correct order of credentials in comment by @BryanFauble in #1091
• [SYNPY-1456] Flaky integration tests due to connection issues by @BryanFauble in #1085
• [SYNPY-1456] Revert to old ubuntu by @BryanFauble in #1092
• [SYNPY-1466] update user agent for command line by @danlu1 in #1090
• Update README.md by @allaway in #1093
• [SYNPY-1470] Remove SonarCloud's dependency on test job by @danlu1 in #1095
• [SYNPY-1470] Always run SonarCloud as long as previous jobs are not cancelled by @danlu1 in #1096
• [SYNPY-1356] Refactor sync to synapse by @BryanFauble in #1083
• [SYNPY-1356] Logic around disassociating Activity from File Entity by @BryanFauble in #1098
• [SYNPY-1341] Add CI step to containerize and upload synpy to the GHCR by @jaymedina in #1100
• [SYNPY-1384] Create uploading data in bulk tutorial by @BryanFauble in #1101

Full Changelog: v4.2.0...v4.3.0-rc

v4.3.0-rc

Highlights

• New tutorial:
  • Uploading data in bulk
    is our newest tutorial. It covers the basics of working with manifest files to manage synapse projects.
• Updates to syncToSynapse:
  • The syncToSynapse function has been refactored to improve performance and
    reliability.
  • Minor behavior change: File entities will no longer have it's version
    incremented during no-op changes. Only when file content, or fields on the file
    has been updated will a version number be incremented.
  • Optional booleans merge_existing_annotations and
    associate_activity_to_new_version have been added. Both are used to give more
    fine tuned control when working with this interface.
  • Check out the changes in the reference docs.

What's Changed

• [SYNPY-1474] Correct order of credentials in comment by @BryanFauble in #1091
• [SYNPY-1456] Flaky integration tests due to connection issues by @BryanFauble in #1085
• [SYNPY-1456] Revert to old ubuntu by @BryanFauble in #1092
• [SYNPY-1466] update user agent for command line by @danlu1 in #1090
• Update README.md by @allaway in #1093
• [SYNPY-1470] Remove SonarCloud's dependency on test job by @danlu1 in #1095
• [SYNPY-1470] Always run SonarCloud as long as previous jobs are not cancelled by @danlu1 in #1096
• [SYNPY-1356] Refactor sync to synapse by @BryanFauble in #1083
• [SYNPY-1356] Logic around disassociating Activity from File Entity by @BryanFauble in #1098
• [SYNPY-1341] Add CI step to containerize and upload synpy to the GHCR by @jaymedina in #1100
• [SYNPY-1384] Create uploading data in bulk tutorial by @BryanFauble in #1101

Full Changelog: v4.2.0...v4.3.0-rc

v4.2.0

Highlights

• Continued Async Support:
  • Multi-threaded Uploads Are Now Async: This change optimizes the upload process to handle multiple tasks simultaneously: uploading file parts concurrently, performing MD5 checksum calculations in separate processes, and making HTTP calls asynchronously. Further, memory management techniques were implemented to prevent crashes on low-resource EC2 instances. Benchmark results can be found here.
  • HTTPX Support & Async Client Instances: In this update, support for the HTTPX library has been introduced, enhancing the functionality of Synapse with two new async client instances for seamless interaction with Synapse and Storage Providers. Additionally, the retry and back-off mechanisms have been revamped to improve reliability and performance, ensuring smoother operation even under challenging network conditions.

What's Changed

• [SYNPY-1419] Httpx async client by @BryanFauble in #1076
• [SYNPY-1453] Guard around modified time item in cache by @BryanFauble in #1080
• [SYNPY-1420] Re-write uploads to mix async, multi-threading, and multi-processing by @BryanFauble in #1078
• [SYNPY-1417] Updates for the annotation model by @BryanFauble in #1081
• [SYNPY-1456] Flaky integration tests by @BryanFauble in #1082
• [SYNPY-1465] Adding Annotations to sync and async docs by @jaymedina in #1087

Full Changelog: v4.1.1...v4.2.0

v4.1.1

Highlights

• New Downloading Data in Bulk Tutorial
  • See Downloading data in bulk tutorial for more details on downloading data in bulk from Synapse.
• Downloading Files Troubleshooting Improvement
  • Improved error logging for when users are downloading files using commands like get-download-list from Synapse.

What's Changed

• Regen pipfile.lock for dependabot alert by @BryanFauble in #1069
• [SYNPY-1383] Downloading data in bulk tutorial by @BryanFauble in #1070
• [SYNPY-1443] Correct which fields are merged if there is a change by @BryanFauble in #1072
• [SYNPY-1442] Log update to capture exception when download-list fails by @BryanFauble in #1071
• [SYNPY-1445] Raise exception on HTTP error when retrieving file from storage location by @BryanFauble in #1074

Full Changelog: v4.1.0...v4.1.1

v4.1.0

Highlights

• New Interfaces:

  • Combines data and behavior into a single class, simplifying the understanding and usage of the system's models.
  • New Interface: Introduced a revamped interface in the Synapse Python Client, shifting from a functional programming approach to an object-oriented one.
  • Enhanced Developer Experience: This change highlights much needed quality of life updates for developers. Improved autocomplete, hoverdocs, and examples in docstrings provide a significantly better coding experience.
  • Asyncio Support: Introduced support for asyncio, enabling more efficient use of system resources and enhancing performance for IO-bound tasks.
  • Extensibility: Laying the foundation for an extensible platform, facilitating easier addition of new features, and improvements to the Synapse Python Client.

• synapseutils.walk Improvement:

  • Improved performance for synapseutils.walk.

• Pandas Range Expansion:

  • Expanded pandas range to >=1.5, <3.0.

• Version Notation Support:

  • Using syn123.version notation is now supported with syn.get, synapseutils.syncFromSynapse, and syn.setProvenance. This enhances consistency in version management across various activities.

What's Changed

• [SYNPY-1349] Detect annotation type by @BryanFauble in #1022
• [SYNPY-1345] Migrate to mkdocstrings by @BryanFauble in #1023
• [SYNPY-1412] Fixing typing issue in copy_functions by @jaymedina in #1048
• [SYNPY-1322] Object Orientated Programming Interfaces by @BryanFauble in #1013
• [SYNPY-1344] Creating a decorator to be used in async class methods by @BryanFauble in #1049
• [SYNPY-1344] Remove return type hinting on decorator by @BryanFauble in #1050
• [SYNPY-1344] Moving to pytest-socket for blocking socket connections in unit tests by @BryanFauble in #1051
• [SYNPY-1344] Set client in conftest since login is not used in unit tests by @BryanFauble in #1052
• [SYNPY-1344] Allows Socket Connections for Windows Systems in Unit Tests by @BWMac in #1054
• [SYNPY-1398] Support .[version] syntax for input SynIDs by @jaymedina in #1047
• [SYNPY-1344] Adding activity model for OOP by @BryanFauble in #1055
• [SYNPY-1398] Correct broken docstring by @BryanFauble in #1057
• [SYNPY-1344] Remove need to manually propogate otel context by @BryanFauble in #1056
• [SYNPY-1348] Add User profile and Permission models by @BryanFauble in #1058
• [SYNPY-1411] Prevent Repeated syn.get calls in _help_walk by @BWMac in #1059
• [SYNPY-1347] Adds OOP Team Model by @BWMac in #1053
• [SYNPY-1416] File model finishing touches for OOP by @BryanFauble in #1060
• [SYNPY-1416] Adding to the file.store docstring by @BryanFauble in #1062
• [SYNPY-1253] Check MD5 before upload to verify change in content by @BryanFauble in #1063
• [SYNPY-1415] Publish sequence diagram for walk function by @BryanFauble in #1064
• [SYNPY-448] allow changing Synapse name in changeFileMetaData by @danlu1 in #1065
• [SYNPY-1415] Folder model finishing touches by @BryanFauble in #1061
• [SYNPY-1326] Expand range for pandas dependency to <3.0 by @BryanFauble in #1066
• [SYNPY-1414] Project model finishing touches by @BryanFauble in #1067

Full Changelog: v4.0.0...v4.1.0-rc

v4.0.0

Highlights

• Only authentication through Personal Access Token
  (aka: Authentication bearer token) is supported. Review the Authentication document for information on setting up your usage of a Personal Access Token to authenticate with Synapse.
• Date type Annotations on Synapse entities are now timezone aware. Review our reference documentation for Annotations. The pytz package is reccomended if you regularly work with data across time zones.
  • If you do not set the tzinfo field on a date or datetime instance we will use the timezone of the machine where the code is executing.
  • If you are using the Manifest TSV for bulk actions on your projects you'll now see that [synapseutils.sync.syncFromSynapse][] will store dates as YYYY-MM-DDTHH:MM:SSZ. Review our documentation for an example manifest file. Additionally, if you'd like to upload an annotation in a specific timezone please make sure that it is in ISO 8601 format. If you do not specify a timezone it is assumed to use the timezone of the machine where the code is executing.
• Support for annotations with multiple values through the Manifest TSV with the usage of a comma delimited bracket wrapped list. Any manifest files wishing to take advantage of multi-value annotations need to match this format. Examples:
  • ["Annotation, with a comma", another annotation]
  • [1,2,3]
  • [2023-12-04T07:00:00Z,2000-01-01T07:00:00Z]
• Migration and expansion of the docs site! You'll see that the look, feel, and flow of all of the information on this site has been touched. As we move forward we hope that you'll provide the Data Processing and Engineering team feedback on areas we can improve.
• Expansion of the available Python Tutorials can be found starting here.

What's Changed

• Adding a label to the dockerfile to automatically label it for this repo by @BryanFauble in #1018
• Updates Dockerfile to Correctly Install Dependencies by @BWMac in #1019
• [SYNPY-1358] Correction of timestamp in annotations from manifest file by @BryanFauble in #1020
• [SYNPY-1336] Benchmarking upload with annotations by @BryanFauble in #1021
• [SYNPY-1321] Download benchmark results by @BryanFauble in #1024
• [SYNPY-1360] Migrating to mkdocstrings by @BryanFauble in #1025
• [SYNPY-1366] Add code coverage by @BryanFauble in #1029
• [SYNPY-1362] High level best practices for project structure by @thomasyu888 in #1028
• [SYNPY-1371] Migrate to Google Style by @BWMac in #1033
• [SYNPY-1302] Replace getPermission with get_acl and add new get_permissions by @danlu1 in #1037
• [SYNPY-1334] Revamp getting started docs by @BryanFauble in #1036
• [SYNPY-1332] Pypi deployment strategy by @BryanFauble in #1038
• [SYNPY-1370] Documentation Upgrade by @jaymedina in #1032
• [SYNPY-1370] Minor formatting fixes by @BryanFauble in #1039
• [SYNPY-1371] Doc fixes by @BryanFauble in #1040
• [SYNPY-1225] Support authToken only by @BryanFauble in #1041
• [SYNPY-1392] Remove some deprecated pieces by @BryanFauble in #1043
• [Synpy 1369] Migrate to Google style by @danlu1 in #1042
• [SYNPY-1387] Update Structure Project doc by @danlu1 in #1044
• [SYNPY-1357] Allow multiple values in manifest TSV by @BryanFauble in #1030

New Contributors

• @jaymedina made their first contribution in #1032

Full Changelog: v3.2.0...v4.0.0-rc