Unable to locate executable file: git-lfs. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

The self-hosted runner: tt-metal-ci-vm-152 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The self-hosted runner: tt-metal-ci-vm-149 lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

Process completed with exit code 2.

Unable to download artifact(s): Artifact not found for name: eager-dist-ubuntu-20.04-any-profiler Please ensure that your artifact is not expired and the artifact was uploaded using a compatible version of toolkit/upload-artifact. For more information, visit the GitHub Artifacts FAQ: https://github.com/actions/toolkit/blob/main/packages/artifact/docs/faq.md

The action 'Run demo regression tests' has timed out after 70 minutes.

This can happen if a test hung and is normally an issue with the test, rather than infra. Tenstorrent cards seem to be in use. Killing PIDs and exiting unsuccessfully.

Process completed with exit code 1.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5876305103302002, inference_time=0.0023811068385839462), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.5876305103302002, inference_time=0.0023811068385839462), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

The hookimpl pytest_addhooks uses old-style configuration options (marks or attributes). Please use the pytest.hookimpl(tryfirst=True) decorator instead to configure the hooks. See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers

The hookimpl pytest_addhooks uses old-style configuration options (marks or attributes). Please use the pytest.hookimpl(tryfirst=True) decorator instead to configure the hooks. See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-True] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=2, inference_and_compile_time=0.36928701400756836, inference_time=0.0026929322630167007), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-False] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=2, inference_and_compile_time=0.5124471187591553, inference_time=0.002397526055574417), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.30364370346069336, inference_time=0.0023804474622011185), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-True] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=2, inference_and_compile_time=0.36928701400756836, inference_time=0.0026929322630167007), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io_multi_device[wormhole_b0-True-1-2-128-device_params0-False] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=2, inference_and_compile_time=0.5124471187591553, inference_time=0.002397526055574417), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

Expected None, but tests/nightly/single_card/functional_unet/experimental/functional_unet/tests/test_unet_trace.py::test_unet_trace_2cq_same_io[1-2-128-device_params0] returned UNetPerformanceStatistics(groups=2, batch=1, num_devices=1, inference_and_compile_time=0.30364370346069336, inference_time=0.0023804474622011185), which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`?

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The hookimpl pytest_addhooks uses old-style configuration options (marks or attributes). Please use the pytest.hookimpl(tryfirst=True) decorator instead to configure the hooks. See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers

The hookimpl pytest_addhooks uses old-style configuration options (marks or attributes). Please use the pytest.hookimpl(tryfirst=True) decorator instead to configure the hooks. See https://docs.pytest.org/en/latest/deprecations.html#configuring-hook-specs-impls-using-markers

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

`top_k_top_p_filtering` is scheduled for deletion in v4.39. Use `TopKLogitsWarper` and `TopPLogitsWarper` instead.

Hugepages service not found. Using old rc.local method

Hugepages service not found. Using old rc.local method

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

The use of `x.T` on tensors of dimension other than 2 to reverse their shape is deprecated and it will throw an error in a future release. Consider `x.mT` to transpose batches of matrices or `x.permute(*torch.arange(x.ndim - 1, -1, -1))` to reverse the dimensions of a tensor. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3637.)

Hugepages service not found. Using old rc.local method

Hugepages service not found. Using old rc.local method

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Hugepages service not found. Using old rc.local method

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Hugepages service not found. Using old rc.local method

No files were found with the provided path: /work/generated/test_reports/. No artifacts will be uploaded.

Hugepages service not found. Using old rc.local method

To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Expected None, but models/demos/wormhole/mamba/demo/demo.py::test_demo[user_input0-state-spaces/mamba-2.8b-slimpj-50-device_params0] returned DemoResult(generated_text=['The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was fi

`resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.

Expected None, but models/demos/wormhole/mamba/demo/demo.py::test_demo[user_input0-state-spaces/mamba-2.8b-slimpj-50-device_params0] returned DemoResult(generated_text=['The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was first settled by the Iroquois, who were followed by the French, and then the English, who were followed by the Americans, who were followed by the Canadians, who were followed by the Indians, who were followed by', 'The city of Sarnia is located on the eastern shore of Lake Huron at its extreme southern point where it flows into the St. Clair River . Most of the surrounding area is flat , and the elevation ranges from 169 metres ( 554 ft ) and 281 metres ( 922 ft ) above sea level . The soil mostly comprises clay . Despite this high percentage of clay , the soil is remarkably rich for cultivation . Prior to the Ice Age , glaciers covered most of the area , as can be seen not only by the existence of the Great Lakes themselves but also of alluvial sand deposits, terminal moraines, and rich oil reserves.\nThe area was fi

Support for class-based `config` is deprecated, use ConfigDict instead.

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Pydantic V1 style `@validator` validators are deprecated. You should migrate to Pydantic V2 style `@field_validator` validators, see the migration guide for more details

Disk usage is 64 %

Disk usage is 77 %

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 57 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 38 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 74 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 67 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 86 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 77 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 71 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 62 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 70 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 88 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 81 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 61 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 59 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 61 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 88 %

Touching and printing out SMI info

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 89 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 53 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 57 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 60 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 64 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 66 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 39 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 66 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Hugepages is now setup.

Disk usage is 59 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages is now setup.

Disk usage is 58 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages is now setup.

Disk usage is 58 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages is now setup.

Disk usage is 59 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages is now setup.

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available

Disk usage is 79 %

Touching and printing out SMI info

tt-smi reset was successful

Hugepages service found. Command returned with exit code 3. Restarting it so we can ensure hugepages are available