boto3 client with smart_open memory leak

[jia-zhengwei]

Describe the bug

When use boto3 to read remote oss parquet file get memory leak.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No memory leak

Current Behavior

As the method be called many times, the memory leak happend.

Reproduction Steps

Run this script many times

self.client = boto3.client(
    "s3",
    aws_access_key_id=access_key_id,
    aws_secret_access_key=secret_access_key,
    endpoint_url=endpoint,
    config=Config(
        s3={"addressing_style": "virtual", "signature_version": "s3v4"}
        ),
)
with smart_open.open(
    f"s3://{bucket_name}/{file_path}",
    mode="rb",
    transport_params={"client": self.client},
) as f:
    parquet_file = pq.ParquetFile(f)
    ...

self.client.close()

Possible Solution

I guess may related to smart_open object not be closed

Additional Information/Context

no

SDK version used

boto3==1.34.54 botocore==1.34.54

Environment details (OS name and version, etc.)

centos

[tim-finnigan]

Thanks for reaching out. Can you please share your debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script to help with further investigation? And can you share any more details regarding the memory leak? If you're able to provide a memory profiling analysis that could be helpful as well. Otherwise if the issue here is with smart_open then you may want to consider opening an issue in that repository.

[jia-zhengwei]

After added log(boto3.set_stream_logger("boto3", level=logging.DEBUG)), it shows nothing new log.

It's not smart_open bug.

I have a class which will init new instance of boto3 client, and each call will raise the memory up.

def print_memory_used():
    import threading

    import psutil

    def monitor_memory():
        pid = os.getpid()
        process = psutil.Process(pid)

        while True:
            memory_info = process.memory_info()
            memory_usage_mb = memory_info.rss / (1024**2)  # MB
            logger.info(
                f"[Memory Monitor] Process ID: {pid}, Memory Usage: {memory_usage_mb:.2f} MB"  # noqa 
            )
            time.sleep(1)

    memory_thread = threading.Thread(target=monitor_memory, daemon=True)
    memory_thread.start()

[tim-finnigan]

For us to investigate a memory issue with Boto3, please share a full Boto3 snippet without third party code and complete logs (with any sensitive info redacted).

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.